Skip to contentSkip to conversation

Multiple mount protection

RFC PATCH In review Kalpak Shah 7 participants 21 messages May 21, 2007 – Jun 1, 2007 ext4 file system

lore
Share & tools
lore

Kalpak Shah

author
#0 lore raw

RFC PATCH 1/1 Multiple mount protection

Hi,

There have been reported instances of a filesystem having been mounted at 2 places at the same time causing a lot of damage to the filesystem. This patch reserves superblock fields and an INCOMPAT flag for adding multiple mount protection(MMP) support within the ext4 filesystem itself. The superblock will have a block number (s_mmp_block) which will hold a MMP structure which has a sequence number which will be periodically updated every 5 seconds by a mounted filesystem. Whenever a filesystem will be mounted it will wait for s_mmp_interval seconds to make sure that the MMP sequence does not change. To further make sure, we write a random sequence number into the MMP block and wait for another s_mmp_interval secs. If the sequence no. doesn't change then the mount will succeed. In case of failure, the nodename, bdevname and the time at which the MMP block was last updated will be displaye
d. tune2fs can be used to set s_mmp_interval as desired.

It will also protect against running e2fsck on a mounted filesystem by adding similar logic to ext2fs_open().

Any comments or views are welcome!

Signed-off-by Andreas Dilger <[email protected]>
Signed-off-by Kalpak Shah <[email protected]>
lib/ext2fs/ext2_fs.h +6 -3
Index: e2fsprogs-1.40/lib/ext2fs/ext2_fs.h
===================================================================
@@ -568,8 +568,9 @@ struct ext2_super_block {
        __u16   s_want_extra_isize;     /* New inodes should reserve # bytes */
        __u32   s_flags;                /* Miscellaneous flags */
        __u16   s_raid_stride;          /* RAID stride */
-       __u16   s_pad;                  /* Padding */
-       __u32   s_reserved[166];        /* Padding to the end of the block */
+       __u16   s_mmp_interval;         /* Wait for # seconds in MMP checking */
+       __u64   s_mmp_block;            /* Block for multi-mount protection */
+       __u32   s_reserved[164];        /* Padding to the end of the block */
 };
 
 /*

@@ -631,10 +632,12 @@ struct ext2_super_block {
 #define EXT2_FEATURE_INCOMPAT_META_BG          0x0010
 #define EXT3_FEATURE_INCOMPAT_EXTENTS          0x0040
 #define EXT4_FEATURE_INCOMPAT_64BIT            0x0080
+#define EXT4_FEATURE_INCOMPAT_MMP              0x0100
 
 
 #define EXT2_FEATURE_COMPAT_SUPP       0
-#define EXT2_FEATURE_INCOMPAT_SUPP     (EXT2_FEATURE_INCOMPAT_FILETYPE)
+#define EXT2_FEATURE_INCOMPAT_SUPP     (EXT2_FEATURE_INCOMPAT_FILETYPE| \
+                                        EXT4_FEATURE_INCOMPAT_MMP)
 #define EXT2_FEATURE_RO_COMPAT_SUPP    (EXT2_FEATURE_RO_COMPAT_SPARSE_SUPER| \
                                         EXT2_FEATURE_RO_COMPAT_LARGE_FILE| \
                                         EXT2_FEATURE_RO_COMPAT_BTREE_DIR)

Thanks,
Kalpak.

#1 lore raw
Quoted 17 lines · Kalpak Shah

Hi,

There have been reported instances of a filesystem having been mounted at 2 places at the same time causing a lot of damage to the filesystem. This patch reserves superblock fields and an INCOMPAT flag for adding multiple mount protection(MMP) support within the ext4 filesystem itself. The superblock will have a block number (s_mmp_block) which will hold a MMP structure which has a sequence number which will be periodically updated every 5 seconds by a mounted filesystem. Whenever a filesystem will be mounted it will wait for s_mmp_interval seconds to make sure that the MMP sequence does not change. To further make sure, we write a random sequence number into the MMP block and wait for another s_mmp_interval secs. If the sequence no. doesn't change then the mount will succeed. In case of failure, the nodename, bdevname and the time at which the MMP block was last updated will be displayed. tune2fs can be used to set s_mmp_interval as desired.

What would the default value of s_mmp_interval be? 5 seconds? more?

If I am not reading this wrong a mount will take more than 's_mmp_interval' seconds to complete. Wouldn't this be too much of a penalty during boot up if the system has many 'mount at boot' filesystems?

Also, I am curious about this. Is there a test case for mounting the same filesystem multiple times? Does this use different paths to reach the device? Or is there a race? Or does it happen on a device shared by multiple hosts?

-Manoj

author
#2 lore raw
Quoted 19 lines · Manoj Joseph
Quoted 17 lines · Kalpak Shah

Hi,

There have been reported instances of a filesystem having been mounted at 2 places at the same time causing a lot of damage to the filesystem. This patch reserves superblock fields and an INCOMPAT flag for adding multiple mount protection(MMP) support within the ext4 filesystem itself. The superblock will have a block number (s_mmp_block) which will hold a MMP structure which has a sequence number which will be periodically updated every 5 seconds by a mounted filesystem. Whenever a filesystem will be mounted it will wait for s_mmp_interval seconds to make sure that the MMP sequence does not change. To further make sure, we write a random sequence number into the MMP block and wait for another s_mmp_interval secs. If the sequence no. doesn't change then the mount will succeed. In case of failure, the nodename, bdevname and the time at which the MMP block was last updated will be displayed. tune2fs can be used to set s_mmp_interval as desired.

Quoted 1 line

What would the default value of s_mmp_interval be? 5 seconds? more?

I have set the default value to 6 seconds. Depending on specific conditions (hardware, etc.) it can be increased using tunefs.

Quoted 4 lines

If I am not reading this wrong a mount will take more than 's_mmp_interval' seconds to complete. Wouldn't this be too much of a penalty during boot up if the system has many 'mount at boot' filesystems?

Yes it may take a maximum of s_mmp_interval*2 seconds to mount a filesystem which has INCOMPAT_MMP feature set. Its up to the user to use this feature, if he finds the penalty is too large, he can do away with this feature. This feature will mostly be used for filesystems used in failover scenarios.

Quoted 6 lines

Also, I am curious about this. Is there a test case for mounting the same filesystem multiple times? Does this use different paths to reach the device? Or is there a race? Or does it happen on a device shared by multiple hosts?

If you are using some HA software, there is the possibility of a race. Yes it can happen on a device shared by multiple hosts.

A simple test case for this will be:

$ dd if=/dev/zero of=img0 bs=1M count=256
$ mke2fs -F -j img0
$ ln img0 img1
$ losetup /dev/loop0 img0
$ losetup /dev/loop1 img1
$ mount /dev/loop0 /mnt/loop0
$ mount /dev/loop1 /mnt/loop1

This succeeds currently causing a multiple mount.

Thanks,
Kalpak.

Quoted 6 lines
(quoted diff) +0 -2
-Manoj
 
-

To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 5 lines · Kalpak Shah
Quoted 2 lines

Also, I am curious about this. Is there a test case for mounting the same filesystem multiple times? Does this use different paths to reach the device? Or is there a race? Or does it happen on a device shared by multiple hosts?

If you are using some HA software, there is the possibility of a race. Yes it can happen on a device shared by multiple hosts.

Ah, if the HA-software doesn't deal with multiple mounts for filesystems it is managing, then I would claim that the software is flawed. :)

But yes, turning on MMP would help.

It might also help to make the frequency at which sequence number gets updated (currently 5 sec) tunable. Would making that also a field in the super block be a bad idea (set only by mkfs/tunefs)?

It might also be worthwhile to write the dev_t, the path of the device and the hostname to the s_mmp_block, along with the random sequence. (I assume there is enough space.) If the mount is being failed because of a multiple mount scenario, these fields could be used to provide useful diagnostics.

My $ 0.02. :)

Regards,
Manoj

author
#4 lore raw
Quoted 8 lines · Manoj Joseph
Quoted 6 lines · Kalpak Shah
Quoted 5 lines

Also, I am curious about this. Is there a test case for mounting the same filesystem multiple times? Does this use different paths to reach the device? Or is there a race? Or does it happen on a device shared by multiple hosts?

Quoted 5 lines
Quoted 2 lines

If you are using some HA software, there is the possibility of a race. Yes it can happen on a device shared by multiple hosts.

Ah, if the HA-software doesn't deal with multiple mounts for filesystems it is managing, then I would claim that the software is flawed. :)

Well, it is known to happen so it wouldn't be bad to make sure.

Quoted 6 lines

But yes, turning on MMP would help.

It might also help to make the frequency at which sequence number gets updated (currently 5 sec) tunable. Would making that also a field in the super block be a bad idea (set only by mkfs/tunefs)?

Updating the MMP sequence too often would hurt the filesystem performance.

Quoted 6 lines

It might also be worthwhile to write the dev_t, the path of the device and the hostname to the s_mmp_block, along with the random sequence. (I assume there is enough space.) If the mount is being failed because of a multiple mount scenario, these fields could be used to provide useful diagnostics.

Yes, the dev_t, host name, the sequence and the time last updated would all be printed. There is lots of space since we have an entire block.

Quoted 1 line

My $ 0.02. :)

Thanks. :)

Quoted 3 lines

Regards,
Manoj

#5 lore raw
In reply to Kalpak Shah
Quoted 4 lines · Kalpak Shah
Quoted 3 lines · Manoj Joseph
Quoted 2 lines · Kalpak Shah

Hi,

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines

There have been reported instances of a filesystem having been mounted at 2 places at the same time causing a lot of damage to the filesystem. This patch reserves superblock fields and an INCOMPAT flag for adding multiple mount protection(MMP) support within the ext4 filesystem itself. The superblock will have a block number (s_mmp_block) which will hold a MMP structure which has a sequence number which will be periodically updated every 5 seconds by a

Quoted 8 lines
Quoted 8 lines

mounted filesystem. Whenever a filesystem will be mounted it will wait for s_mmp_interval seconds to make sure that the MMP sequence does not change. To further make sure, we write a random sequence number into the MMP block and wait for another s_mmp_interval secs. If the sequence no. doesn't change then the mount will succeed. In case of failure, the nodename, bdevname and the time at which the MMP block was last updated will be displayed. tune2fs can be used to set s_mmp_interval as desired.

Frankly, I don't understand why we need this feature. The filesystem limitations (=not ready for clusters) should be described in docs. That's enough from my POV...

Quoted 10 lines
Quoted 2 lines

What would the default value of s_mmp_interval be? 5 seconds? more?

I have set the default value to 6 seconds. Depending on specific conditions (hardware, etc.) it can be increased using tunefs.

Quoted 4 lines

If I am not reading this wrong a mount will take more than 's_mmp_interval' seconds to complete. Wouldn't this be too much of a penalty during boot up if the system has many 'mount at boot' filesystems?

Quoted 5 lines

Yes it may take a maximum of s_mmp_interval*2 seconds to mount a filesystem which has INCOMPAT_MMP feature set. Its up to the user to use this feature, if he finds the penalty is too large, he can do away with this feature. This feature will mostly be used for filesystems used in failover scenarios.

I hope the feature will be disabled by default. It sounds strange that I have to way 6 secs to mount a FS if I (and 99% of Linux users) needn't to share same FS between two mountpoint.

I have 5 filesystems on my workstation = 30 secs penality during boot?!

Quoted 7 lines
Quoted 4 lines

Also, I am curious about this. Is there a test case for mounting the same filesystem multiple times? Does this use different paths to reach the device? Or is there a race? Or does it happen on a device shared by multiple hosts?

If you are using some HA software, there is the possibility of a race. Yes it can happen on a device shared by multiple hosts.

That's reason why people use OCFS or GFS.

Quoted 10 lines

A simple test case for this will be:

$ dd if=/dev/zero of=img0 bs=1M count=256
$ mke2fs -F -j img0
$ ln img0 img1
$ losetup /dev/loop0 img0
$ losetup /dev/loop1 img1
$ mount /dev/loop0 /mnt/loop0
$ mount /dev/loop1 /mnt/loop1

This succeeds currently causing a multiple mount.

And what? That's wrong FS usage.

Karel

Signature · 1 line
 Karel Zak  <[email protected]>
author
#6 lore raw
Quoted 3 lines · Karel Zak

Frankly, I don't understand why we need this feature. The filesystem limitations (=not ready for clusters) should be described in docs. That's enough from my POV...

It is highly advocated that ext3/4 filesystem should not be multiply mounted. This just makes doubly sure of this only if the user desires.

Quoted 16 lines
Quoted 10 lines
Quoted 2 lines

What would the default value of s_mmp_interval be? 5 seconds? more?

I have set the default value to 6 seconds. Depending on specific conditions (hardware, etc.) it can be increased using tunefs.

Quoted 4 lines

If I am not reading this wrong a mount will take more than 's_mmp_interval' seconds to complete. Wouldn't this be too much of a penalty during boot up if the system has many 'mount at boot' filesystems?

Quoted 5 lines

Yes it may take a maximum of s_mmp_interval*2 seconds to mount a filesystem which has INCOMPAT_MMP feature set. Its up to the user to use this feature, if he finds the penalty is too large, he can do away with this feature. This feature will mostly be used for filesystems used in failover scenarios.

Quoted 5 lines

I hope the feature will be disabled by default. It sounds strange that I have to way 6 secs to mount a FS if I (and 99% of Linux users) needn't to share same FS between two mountpoint.

I have 5 filesystems on my workstation = 30 secs penality during boot?!

This feature won't be enabled by default. Its absolutely the users discretion if he wants to enable this feature. It can be set by tune2fs and can be disabled without unmounting the filesystem. So you won't have to waste time during mounting unless you choose to.

Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

Also, I am curious about this. Is there a test case for mounting the same filesystem multiple times? Does this use different paths to reach the device? Or is there a race? Or does it happen on a device shared by multiple hosts?

If you are using some HA software, there is the possibility of a race. Yes it can happen on a device shared by multiple hosts.

That's reason why people use OCFS or GFS.

OCFS and GFS are clustered file systems and hence provide read-write support at multiple mount points.

Note that the MMP feature will make that you can't run e2fsck on a mounted filesystem. So in short the filesystem cannot be opened in read-write mode by more than 1 entity.

Quoted 10 lines
Quoted 10 lines

A simple test case for this will be:

$ dd if=/dev/zero of=img0 bs=1M count=256
$ mke2fs -F -j img0
$ ln img0 img1
$ losetup /dev/loop0 img0
$ losetup /dev/loop1 img1
$ mount /dev/loop0 /mnt/loop0
$ mount /dev/loop1 /mnt/loop1
Quoted 3 lines
Quoted 1 line

This succeeds currently causing a multiple mount.

And what? That's wrong FS usage.

Here I had just described a test case for reproducing multiple mounts.

Thanks,
Kalpak.

Quoted 3 lines

Karel

maintainer
#7 lore raw
In reply to Kalpak Shah

Hi Kalpak,

Quoted 2 lines · Kalpak Shah

It will also protect against running e2fsck on a mounted filesystem by adding similar logic to ext2fs_open().

Your patch didn't add this logic to ext2fs_open(); it just reserved the space in the superblock.

I don't mind reserving the space so we don't have to worry about conflicting superblock uses, but I'm still on the fence about actually adding this functionality (a) into e2fsprogs, and (b) into the ext4 kernel code. I guess it depends on how complicated/icky the implementation code is, I guess. The question as before is whether the complexity is worth it, given that someone who is actually going to be subject to accidentally mounting an ext3/4 filesystem on multiple systems needs to be using an HA system anyway. So basically this is just to protect against (a) a bug/failure in the HA subsystem, and (b) the idiotic user that failed to realized he/she needed to set up an HA subsystem in the first place. Granted, the universe is going to create idiots at a faster rate that we can deal with it, but that's why I'm still not 100% convinced the complexity is worth it.

To be fair, if I was on a L3 support team having to deal with these idiots, I'd probably feel differently. :-)

- Ted

#8 lore raw

Hi Ted,

For what it's worth, we have several petabytes of data residing in
ext3 file systems, a large staff of mainly non-idiots, and HA s/w,
and I still feel strongly that multi-mount protection is a good idea.
People, software, and hardware all malfunction in myriad ways, and the more you have, the greater the odds (or so it seems to us). This relatively simple safeguard at the fs level has high value IMHO.

Regards,

Jim

Quoted 32 lines · Theodore Tso

Hi Kalpak,

Quoted 2 lines · Kalpak Shah

It will also protect against running e2fsck on a mounted filesystem by adding similar logic to ext2fs_open().

Your patch didn't add this logic to ext2fs_open(); it just reserved the space in the superblock.

I don't mind reserving the space so we don't have to worry about conflicting superblock uses, but I'm still on the fence about actually adding this functionality (a) into e2fsprogs, and (b) into the ext4 kernel code. I guess it depends on how complicated/icky the implementation code is, I guess. The question as before is whether the complexity is worth it, given that someone who is actually going to be subject to accidentally mounting an ext3/4 filesystem on multiple systems needs to be using an HA system anyway. So basically this is just to protect against (a) a bug/failure in the HA subsystem, and (b) the idiotic user that failed to realized he/she needed to set up an HA subsystem in the first place. Granted, the universe is going to create idiots at a faster rate that we can deal with it, but that's why I'm still not 100% convinced the complexity is worth it.

To be fair, if I was on a L3 support team having to deal with these idiots, I'd probably feel differently. :-)

- Ted
-
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#9 lore raw
In reply to Theodore Tso

Hi Ted,

Quoted 8 lines · Theodore Tso

Hi Kalpak,

Quoted 2 lines · Kalpak Shah

It will also protect against running e2fsck on a mounted filesystem by adding similar logic to ext2fs_open().

Your patch didn't add this logic to ext2fs_open(); it just reserved the space in the superblock.

Yeah the earlier patch for just reserving the fields.

Quoted 6 lines

I don't mind reserving the space so we don't have to worry about conflicting superblock uses, but I'm still on the fence about actually adding this functionality (a) into e2fsprogs, and (b) into the ext4 kernel code. I guess it depends on how complicated/icky the implementation code is, I guess.

I am attaching the kernel and e2fsrogs patches so that you can suggest any short-comings in the implementation. These patches are still a WIP.

Quoted 9 lines

The question as before is whether
the complexity is worth it, given that someone who is actually going to be subject to accidentally mounting an ext3/4 filesystem on multiple systems needs to be using an HA system anyway. So basically this is just to protect against (a) a bug/failure in the HA subsystem, and (b) the idiotic user that failed to realized he/she needed to set up an HA subsystem in the first place. Granted, the universe is going to create idiots at a faster rate that we can deal with it, but that's why I'm still not 100% convinced the complexity is worth it.

Given the amount of damage that multiple mounts can cause to the filesystem, it would be desirable to make doubly sure. Also the MMP feature is quite uncomplicated and absolutely tunable.

Thanks for your views.

- Kalpak.

Quoted 9 lines

To be fair, if I was on a L3 support team having to deal with these idiots, I'd probably feel differently. :-)

- Ted
-
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

fs/ext4/super.c +226 -0
Index: linux-2.6.19/fs/ext4/super.c
===================================================================
@@ -35,6 +35,8 @@
 #include <linux/namei.h>
 #include <linux/quotaops.h>
 #include <linux/seq_file.h>
+#include <linux/kthread.h>
+#include <linux/utsname.h>
 
 #include <asm/uaccess.h>
 

@@ -481,6 +483,9 @@ static void ext4_put_super (struct super
 		invalidate_bdev(sbi->journal_bdev, 0);
 		ext4_blkdev_remove(sbi);
 	}
+	if (sbi->s_mmp_tsk)
+		kthread_stop(sbi->s_mmp_tsk);
+
 	sb->s_fs_info = NULL;
 	kfree(sbi);
 	return;

@@ -1441,6 +1446,223 @@ static ext4_fsblk_t descriptor_loc(struc
 	return (has_super + ext4_group_first_block_no(sb, bg));
 }
 
+static inline
+int write_mmp_block(struct super_block *sb, struct buffer_head *bh,
+		    const char *bdev_name)
+{
+	int retval;
+
+	mark_buffer_dirty(bh);
+	retval = sync_dirty_buffer(bh);
+	if (retval)
+		ext4_error(sb, "write_mmp_block",
+			   "Error writing to MMP block.");
+
+	return retval;
+}
+
+static inline
+int read_mmp_block(struct super_block *sb, struct buffer_head **bh,
+		   ext4_fsblk_t mmp_block)
+{
+	if (*bh)
+		clear_buffer_uptodate(*bh);
+
+	*bh = sb_bread(sb, mmp_block);
+	if (!*bh) {
+		ext4_warning(sb, "read_mmp_block",
+			     "Error while reading MMP block %llu", mmp_block);
+		return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * kmmpd will update the MMP sequence every s_mmp_interval seconds
+ */
+static int kmmpd(void *data)
+{
+	struct super_block *sb = (struct super_block *) data;
+	struct ext4_super_block *es = EXT4_SB(sb)->s_es;
+	struct buffer_head *bh = NULL;
+	struct mmp_struct *mmp;
+	ext4_fsblk_t mmp_block;
+	u32 seq = 0;
+	unsigned long failed_writes = 0;
+	int retval;
+	int mmp_interval = cpu_to_le16(es->s_mmp_interval);
+
+	mmp_block = le32_to_cpu(es->s_mmp_block);
+	retval = read_mmp_block(sb, &bh, mmp_block);
+	if (retval)
+		goto failed;
+
+	mmp = (struct mmp_struct *)(bh->b_data);
+	mmp->mmp_magic = cpu_to_le32(EXT4_MMP_MAGIC);
+	mmp->mmp_time = cpu_to_le64(get_seconds());
+	mmp->mmp_interval = mmp_interval;
+	bdevname(bh->b_bdev, mmp->mmp_bdevname);
+
+	down_read(&uts_sem);
+	memcpy(mmp->mmp_nodename, init_uts_ns.name.nodename, 64);
+	up_read(&uts_sem);
+
+	while (!kthread_should_stop()) {
+		if (++seq >= EXT4_MMP_FSCK_ON)
+			seq = 1;
+
+		mmp->mmp_seq = cpu_to_le32(seq);
+		mmp->mmp_time = cpu_to_le64(get_seconds());
+
+		retval = write_mmp_block(sb, bh, mmp->mmp_bdevname);
+		/*
+		 * Don't spew too many error messages. Print one every
+		 * (s_mmp_interval * 60) seconds.
+		 */
+		if (retval && (failed_writes % 60) == 0) {
+				ext4_warning(sb, "kmmpd",
+					     "Error writing to MMP block");
+				failed_writes++;
+		}
+
+		if (!(le32_to_cpu(es->s_feature_incompat) &
+		    EXT4_FEATURE_INCOMPAT_MMP)) {
+			ext4_warning(sb, "kmmpd", "kmmpd being stopped "
+				     "since MMP feature has been "
+				     "disabled.");
+			goto failed;
+		}
+
+		if (sb->s_flags & MS_RDONLY) {
+			ext4_warning(sb, "kmmpd", "kmmpd being stopped since "
+				     "filesystem has been remounted as readonly.");
+			goto failed;
+		}
+
+		schedule_timeout_interruptible(mmp_interval * HZ);
+	}
+
+	/* Unmount seems to be clean */
+	mmp->mmp_seq = cpu_to_le32(EXT4_MMP_CLEAN);
+	mmp->mmp_time = cpu_to_le64(get_seconds());
+
+	retval = write_mmp_block(sb, bh, mmp->mmp_bdevname);
+
+failed:
+	brelse(bh);
+	return 0;
+}
+
+void dump_mmp_msg(struct super_block *sb, struct mmp_struct *mmp,
+		  const char *function, const char *msg)
+{
+	ext4_warning(sb, function, msg);
+	ext4_warning(sb, function, "Dumping MMP information:\n"
+		     "Time last updated: %llu\n"
+		     "Last node which updated MMP: %s\n"
+		     "Last block device which updated MMP: %s\n",
+		     le64_to_cpu(mmp->mmp_time), mmp->mmp_nodename,
+		     mmp->mmp_bdevname);
+}
+
+/*
+ * Protect the filesystem from being mounted more than once.
+ */
+static int ext4_multi_mount_protect(struct super_block *sb,
+				    ext4_fsblk_t mmp_block)
+{
+	struct ext4_super_block *es = EXT4_SB(sb)->s_es;
+	struct buffer_head *bh = NULL;
+	struct mmp_struct *mmp = NULL;
+	u32 seq;
+	unsigned int wait_interval = 2 * le32_to_cpu(es->s_mmp_interval);
+	int retval;
+
+	if (mmp_block < le32_to_cpu(es->s_first_data_block) ||
+	    mmp_block > ext4_blocks_count(EXT4_SB(sb)->s_es)) {
+		ext4_warning(sb, "ext4_multi_mount_protect",
+			     "Invalid MMP block in superblock");
+		goto failed;
+	}
+
+	retval = read_mmp_block(sb, &bh, mmp_block);
+	if (retval)
+		goto failed;
+
+	mmp = (struct mmp_struct *)(bh->b_data);
+	if (le32_to_cpu(mmp->mmp_magic) != EXT4_MMP_MAGIC) {
+		ext4_error(sb, "ext4_multi_mount_protect",
+			   "Invalid magic number in MMP block");
+		goto failed;
+	}
+
+	if (le32_to_cpu(es->s_mmp_interval) == 0)
+		es->s_mmp_interval = cpu_to_le32(EXT4_MMP_DEF_INTERVAL);
+
+	seq = le32_to_cpu(mmp->mmp_seq);
+	if (seq == EXT4_MMP_CLEAN)
+		goto skip;
+
+	if (seq == EXT4_MMP_FSCK_ON) {
+		dump_mmp_msg(sb, mmp, "ext4_multi_mount_protect",
+			     "fsck is running on the filesystem");
+		goto failed;
+	}
+
+	/* wait for MMP interval and check seq again */
+	schedule_timeout_uninterruptible(HZ * wait_interval);
+
+	retval = read_mmp_block(sb, &bh, mmp_block);
+	if (retval)
+		goto failed;
+	mmp = (struct mmp_struct *)(bh->b_data);
+	if (seq != le32_to_cpu(mmp->mmp_seq)) {
+		dump_mmp_msg(sb, mmp, "ext4_multi_mount_protect",
+			     "Device is already active on another node.");
+		goto failed;
+	}
+
+skip:
+	/* write a new random sequence number */
+	get_random_bytes(&seq, sizeof(u32));
+	mmp->mmp_seq = cpu_to_le32(seq);
+	retval = write_mmp_block(sb, bh, sb->s_id);
+	if (retval)
+		goto failed;
+
+	/* wait for MMP interval and check seq again */
+	schedule_timeout_uninterruptible(HZ * wait_interval);
+
+	retval = read_mmp_block(sb, &bh, mmp_block);
+	if (retval)
+		goto failed;
+	mmp = (struct mmp_struct *)(bh->b_data);
+	if (seq != le32_to_cpu(mmp->mmp_seq)) {
+		dump_mmp_msg(sb, mmp, "ext4_multi_mount_protect",
+			     "Device is already active on another node.");
+		goto failed;
+	}
+
+	/* Start a kernel thread to update the MMP block periodically */
+	EXT4_SB(sb)->s_mmp_tsk = kthread_run(kmmpd, sb, "kmmpd-%02x:%02x",
+					     MAJOR(sb->s_dev), MINOR(sb->s_dev));
+	if (IS_ERR(EXT4_SB(sb)->s_mmp_tsk)) {
+		EXT4_SB(sb)->s_mmp_tsk = 0;
+		ext4_warning(sb, "ext4_multi_mount_protect",
+		       "Unable to create kmmpd thread for %s.", sb->s_id);
+		goto failed;
+	}
+
+	brelse(bh);
+	return 0;
+
+failed:
+	brelse(bh);
+
+	return 1;
+}
+
 
 static int ext4_fill_super (struct super_block *sb, void *data, int silent)
 {

@@ -1770,6 +1992,10 @@ static int ext4_fill_super (struct super
 			  EXT4_HAS_INCOMPAT_FEATURE(sb,
 				    EXT4_FEATURE_INCOMPAT_RECOVER));
 
+	if (EXT4_HAS_INCOMPAT_FEATURE(sb, EXT4_FEATURE_INCOMPAT_MMP))
+		if (ext4_multi_mount_protect(sb, le64_to_cpu(es->s_mmp_block)))
+			goto failed_mount2;
+
 	/*
 	 * The first inode we look at is the journal inode.  Don't try
 	 * root first: it may be modified in the journal!
include/linux/ext4_fs_sb.h +2 -0
Index: linux-2.6.19/include/linux/ext4_fs_sb.h
===================================================================
@@ -90,6 +90,8 @@ struct ext4_sb_info {
 	unsigned long s_ext_extents;
 #endif
 	unsigned int  s_want_extra_isize;	       /* New inodes should reserve # bytes */
+
+	struct task_struct * s_mmp_tsk;		/* Kernel thread for multiple mount protection */
 };
 
 #endif	/* _LINUX_EXT4_FS_SB */
include/linux/ext4_fs.h +31 -4
Index: linux-2.6.19/include/linux/ext4_fs.h
===================================================================
@@ -578,10 +578,11 @@ struct ext4_super_block {
 	__le32	s_free_blocks_count_hi;	/* Free blocks count */
 	__le16  s_min_extra_isize;	/* All inodes have at least # bytes */
 	__le16  s_want_extra_isize;	/* New inodes should reserve # bytes */
-	__le32	s_flags;		/* Miscellaneous flags */
+/*160*/ __le32	s_flags;		/* Miscellaneous flags */
 	__le16	s_raid_stride;	  	/* RAID stride */
-	__le16	s_pad;		  	/* Padding */
-	__le32	s_reserved[166];	/* Padding to the end of the block */
+	__le16	s_mmp_interval;		/* Wait for # seconds in MMP checking */
+	__le64	s_mmp_block;		/* Block for multi-mount protection */
+	__u32	s_reserved[164];	/* Padding to the end of the block */
 };
 
 #ifdef __KERNEL__

@@ -680,13 +681,15 @@ static inline int ext4_valid_inum(struct
 #define EXT4_FEATURE_INCOMPAT_META_BG		0x0010
 #define EXT4_FEATURE_INCOMPAT_EXTENTS		0x0040 /* extents support */
 #define EXT4_FEATURE_INCOMPAT_64BIT		0x0080
+#define EXT4_FEATURE_INCOMPAT_MMP		0x0100
 
 #define EXT4_FEATURE_COMPAT_SUPP	EXT2_FEATURE_COMPAT_EXT_ATTR
 #define EXT4_FEATURE_INCOMPAT_SUPP	(EXT4_FEATURE_INCOMPAT_FILETYPE| \
 					 EXT4_FEATURE_INCOMPAT_RECOVER| \
 					 EXT4_FEATURE_INCOMPAT_META_BG| \
 					 EXT4_FEATURE_INCOMPAT_EXTENTS| \
-					 EXT4_FEATURE_INCOMPAT_64BIT)
+					 EXT4_FEATURE_INCOMPAT_64BIT| \
+					 EXT4_FEATURE_INCOMPAT_MMP)
 #define EXT4_FEATURE_RO_COMPAT_SUPP	(EXT4_FEATURE_RO_COMPAT_SPARSE_SUPER| \
 					 EXT4_FEATURE_RO_COMPAT_LARGE_FILE| \
 					 EXT4_FEATURE_RO_COMPAT_EXTRA_ISIZE| \

@@ -850,6 +853,30 @@ void ext4_get_group_no_and_offset(struct
 			unsigned long *blockgrpp, ext4_grpblk_t *offsetp);
 
 /*
+ * This structure will be used for multiple mount protection. It will be
+ * written into the block number saved in the s_mmp_block field in the
+ * superblock.
+ */
+#define EXT4_MMP_MAGIC	   0x004D4D50 /* ASCII of MMP */
+#define EXT4_MMP_CLEAN	   0xFF4D4D50 /* Value of mmp_seq for clean unmount */
+#define EXT4_MMP_FSCK_ON   0xE24D4D50 /* Value of mmp_seq when being fscked */
+struct mmp_struct {
+	__le32	mmp_magic;
+	__le32	mmp_seq;
+	__le64	mmp_time;
+	char	mmp_nodename[64];
+	char	mmp_bdevname[BDEVNAME_SIZE];
+	__le16	mmp_interval;
+	__le16	mmp_pad1;
+	__le32	mmp_pad2;
+};
+
+/*
+ * Interval in number of seconds to update the MMP sequence number.
+ */
+#define EXT4_MMP_DEF_INTERVAL   5
+
+/*
  * Function prototypes
  */
lib/e2p/feature.c +2 -0
Index: e2fsprogs-1.39/lib/e2p/feature.c
===================================================================
@@ -67,6 +67,8 @@ static struct feature feature_list[] = {
 			"extent" },
 	{	E2P_FEATURE_INCOMPAT, EXT4_FEATURE_INCOMPAT_64BIT,
 			"64bit" },
+	{       E2P_FEATURE_INCOMPAT, EXT4_FEATURE_INCOMPAT_MMP,
+			"mmp" },
 	{	0, 0, 0 },
 };
lib/ext2fs/ext2_fs.h +30 -3
Index: e2fsprogs-1.39/lib/ext2fs/ext2_fs.h
===================================================================
@@ -570,8 +570,9 @@ struct ext2_super_block {
 	__u16	s_want_extra_isize; 	/* New inodes should reserve # bytes */
 	__u32	s_flags;		/* Miscellaneous flags */
 	__u16   s_raid_stride;		/* RAID stride */
-	__u16   s_pad;			/* Padding */
-	__u32   s_reserved[166];	/* Padding to the end of the block */
+	__u16   s_mmp_interval;		/* Wait for # seconds in MMP checking */
+	__u64   s_mmp_block;		/* Block for multi-mount protection */
+	__u32   s_reserved[164];	/* Padding to the end of the block */
 };
 
 /*

@@ -633,10 +634,12 @@ struct ext2_super_block {
 #define EXT2_FEATURE_INCOMPAT_META_BG		0x0010
 #define EXT3_FEATURE_INCOMPAT_EXTENTS		0x0040
 #define EXT4_FEATURE_INCOMPAT_64BIT		0x0080
+#define EXT4_FEATURE_INCOMPAT_MMP		0x0100
 
 
 #define EXT2_FEATURE_COMPAT_SUPP	0
-#define EXT2_FEATURE_INCOMPAT_SUPP	(EXT2_FEATURE_INCOMPAT_FILETYPE)
+#define EXT2_FEATURE_INCOMPAT_SUPP	(EXT2_FEATURE_INCOMPAT_FILETYPE| \
+					 EXT4_FEATURE_INCOMPAT_MMP)
 #define EXT2_FEATURE_RO_COMPAT_SUPP	(EXT2_FEATURE_RO_COMPAT_SPARSE_SUPER| \
 					 EXT2_FEATURE_RO_COMPAT_LARGE_FILE| \
 					 EXT4_FEATURE_RO_COMPAT_DIR_NLINK| \

@@ -712,4 +715,28 @@ struct ext2_dir_entry_2 {
 #define EXT2_DIR_REC_LEN(name_len)	(((name_len) + 8 + EXT2_DIR_ROUND) & \
 					 ~EXT2_DIR_ROUND)
 
+/*
+ * This structure will be used for multiple mount protection. It will be
+ * written into the block number saved in the s_mmp_block field in the
+ * superblock.
+ */
+#define	EXT2_MMP_MAGIC    0x004D4D50 /* ASCII for MMP */
+#define	EXT2_MMP_CLEAN    0xFF4D4D50 /* Value of mmp_seq for clean unmount */
+#define	EXT2_MMP_FSCK_ON  0xE24D4D50 /* Value of mmp_seq when being fscked */
+struct mmp_struct {
+	__u32	mmp_magic;
+	__u32	mmp_seq;
+	__u64	mmp_time;
+	char	mmp_nodename[64];
+	char	mmp_bdevname[32];
+	__u16	mmp_interval;
+	__u16	mmp_pad1;
+	__u32	mmp_pad2;
+};
+
+/*
+ * Interval in number of seconds to update the MMP sequence number.
+ */
+#define EXT2_MMP_DEF_INTERVAL	5
+
 #endif	/* _LINUX_EXT2_FS_H */
lib/ext2fs/ext2fs.h +4 -1
Index: e2fsprogs-1.39/lib/ext2fs/ext2fs.h
===================================================================
@@ -190,6 +190,7 @@ typedef struct ext2_file *ext2_file_t;
 #define EXT2_FLAG_IMAGE_FILE		0x2000
 #define EXT2_FLAG_EXCLUSIVE		0x4000
 #define EXT2_FLAG_SOFTSUPP_FEATURES	0x8000
+#define EXT2_FLAG_SKIP_MMP		0x18000
 
 /*
  * Special flag in the ext2 inode i_flag field that means that this is

@@ -462,7 +463,8 @@ typedef struct ext2_icount *ext2_icount_
 					 EXT3_FEATURE_INCOMPAT_JOURNAL_DEV|\
 					 EXT2_FEATURE_INCOMPAT_META_BG|\
 					 EXT3_FEATURE_INCOMPAT_RECOVER|\
-					 EXT3_FEATURE_INCOMPAT_EXTENTS)
+					 EXT3_FEATURE_INCOMPAT_EXTENTS|\
+					 EXT4_FEATURE_INCOMPAT_MMP)
 #endif
 #define EXT2_LIB_FEATURE_RO_COMPAT_SUPP	(EXT2_FEATURE_RO_COMPAT_SPARSE_SUPER|\
 					 EXT2_FEATURE_RO_COMPAT_LARGE_FILE|\

@@ -991,6 +993,7 @@ extern void ext2fs_swap_inode(ext2_filsy
 extern void ext2fs_swap_extent_header(struct ext3_extent_header *eh);
 extern void ext2fs_swap_extent_index(struct ext3_extent_idx *ix);
 extern void ext2fs_swap_extent(struct ext3_extent *ex);
+extern void ext2fs_swap_mmp(struct mmp_struct *mmp);
 
 /* valid_blk.c */
 extern int ext2fs_inode_has_valid_blocks(struct ext2_inode *inode);
misc/tune2fs.c +154 -4

232 lines not on this page · raw message

e2fsck/pass1.c +36 -33

87 lines not on this page · raw message

e2fsck/unix.c +49 -0

61 lines not on this page · raw message

e2fsck/problem.c +5 -0
Index: e2fsprogs-1.39/e2fsck/problem.c
===================================================================
@@ -376,6 +376,11 @@ static struct e2fsck_problem problem_tab
 	  N_("last @g @b @B uninitialized.  "),
 	     PROMPT_FIX, PR_PREEN_OK },
 
+	/* Resize_inode not enabled, but resize inode is non-zero */
+	{ PR_0_MMP_INVALID_BLK,
+	  N_("@S has invalid MMP block.  "),
+	  PROMPT_CLEAR, PR_PREEN_OK },
+
 	/* Pass 1 errors */
 
 	/* Pass 1: Checking inodes, blocks, and sizes */
e2fsck/problem.h +3 -0
Index: e2fsprogs-1.39/e2fsck/problem.h
===================================================================
@@ -212,6 +212,9 @@ struct problem_context {
 /* Last group block bitmap is uninitialized. */
 #define PR_0_BB_UNINIT_LAST			0x000039
 
+/* The MMP block in the superblock is invalid. */
+#define PR_0_MMP_INVALID_BLK			0x00003A
+
 /*
  * Pass 1 errors
  */
lib/ext2fs/swapfs.c +10 -0
Index: e2fsprogs-1.39/lib/ext2fs/swapfs.c
===================================================================
@@ -70,6 +70,8 @@ void ext2fs_swap_super(struct ext2_super
 	sb->s_min_extra_isize = ext2fs_swab16(sb->s_min_extra_isize);
 	sb->s_want_extra_isize = ext2fs_swab16(sb->s_want_extra_isize);
 	sb->s_flags = ext2fs_swab32(sb->s_flags);
+	sb->s_mmp_interval = ext2fs_swab16(sb->s_mmp_interval);
+	sb->s_mmp_block = ext2fs_swab64(sb->s_mmp_block);
 	for (i=0; i < 4; i++)
 		sb->s_hash_seed[i] = ext2fs_swab32(sb->s_hash_seed[i]);
 	for (i=0; i < 17; i++)

@@ -274,4 +276,12 @@ void ext2fs_swap_inode(ext2_filsys fs, s
 				sizeof(struct ext2_inode));
 }
 
+void ext2fs_swap_mmp(struct mmp_struct *mmp)
+{
+	mmp->mmp_magic = ext2fs_swab32(mmp->mmp_magic);
+	mmp->mmp_seq = ext2fs_swab32(mmp->mmp_seq);
+	mmp->mmp_time = ext2fs_swab64(mmp->mmp_time);
+	mmp->mmp_interval = ext2fs_swab16(mmp->mmp_interval);
+}
+
 #endif
lib/ext2fs/openfs.c +112 -0

136 lines not on this page · raw message

lib/ext2fs/ext2_err.et.in +9 -0
Index: e2fsprogs-1.39/lib/ext2fs/ext2_err.et.in
===================================================================
@@ -338,5 +338,14 @@ ec	EXT2_ET_EXTENT_LEAF_BAD,
 ec	EXT2_ET_EXTENT_NO_SPACE,
 	"No free space in extent map"
 
+ec	EXT2_ET_MMP_MAGIC_INVALID,
+	"MMP: Invalid magic number in MMP block"
+
+ec	EXT2_ET_MMP_FAILED,
+	"MMP: Device already active on another node"
+
+ec	EXT2_ET_MMP_FSCK_ON,
+	"MMP: Seems as if fsck is already being run on the filesystem."
+
 	end
lib/ext2fs/closefs.c +49 -0

61 lines not on this page · raw message

author
#10 lore raw
Quoted 5 lines · Kalpak Shah

Hi Ted,

Quoted 2 lines · Theodore Tso

Hi Kalpak,

Quoted 16 lines
Quoted 6 lines · Kalpak Shah
Quoted 2 lines

It will also protect against running e2fsck on a mounted filesystem by adding similar logic to ext2fs_open().

Your patch didn't add this logic to ext2fs_open(); it just reserved the space in the superblock.

Yeah the earlier patch for just reserving the fields.

Quoted 6 lines

I don't mind reserving the space so we don't have to worry about conflicting superblock uses, but I'm still on the fence about actually adding this functionality (a) into e2fsprogs, and (b) into the ext4 kernel code. I guess it depends on how complicated/icky the implementation code is, I guess.

Hi Ted,

So can I assume that the INCOMPAT_MMP flag and the s_mmp_interval and s_mmp_block superblock fields will be reserved regardless of whether the patches go into ext4? I had attached the patches in the last mail so you can share your views on them.

Thanks,
Kalpak.

maintainer
#11 lore raw
Quoted 5 lines · Kalpak Shah

So can I assume that the INCOMPAT_MMP flag and the s_mmp_interval and s_mmp_block superblock fields will be reserved regardless of whether the patches go into ext4? I had attached the patches in the last mail so you can share your views on them.

Yes, i've reserved the code point and superblock fields. I'm not going to add INCOMPAT_MMP flag to the supported file until I get and integrate the patch ext2fs_open() that actually tests for the flag, though, since that would be a bit silly.

I assume the patch will add a flag to ext2fs_open which skips the MMP checking. After all, tune2fs is allowed to make changes to the superblock while the filesystem is mounted. So it needs to be able to open the filesystem read/only even if it is mounted.

Regards,

- Ted

author
#12 lore raw
Quoted 8 lines · Theodore Tso
Quoted 5 lines · Kalpak Shah

So can I assume that the INCOMPAT_MMP flag and the s_mmp_interval and s_mmp_block superblock fields will be reserved regardless of whether the patches go into ext4? I had attached the patches in the last mail so you can share your views on them.

Yes, i've reserved the code point and superblock fields.

Thanks.

Quoted 6 lines

I'm not going to add INCOMPAT_MMP flag to the supported file until I get and
integrate the patch ext2fs_open() that actually tests for the flag, though, since that would be a bit silly.

I assume the patch will add a flag to ext2fs_open which skips the MMP checking.

Yes I have added a EXT2_FLAG_SKIP_MMP flag to ext2fs_open() to bypass MMP which will be set if tunefs is used with -f option. Also MMP check will not be run if the filesystem is being opened readonly.

Thanks,
Kalpak.

Quoted 7 lines

After all, tune2fs is allowed to make changes to the
superblock while the filesystem is mounted. So it needs to be able to open the filesystem read/only even if it is mounted.

Regards,

- Ted

#13 lore raw
In reply to Kalpak Shah
Quoted 3 lines · Kalpak Shah

Hi,

There have been reported instances of a filesystem having been mounted at 2 places at the same time causing a lot of damage to the filesystem. This patch reserves superblock fields and an INCOMPAT flag for adding multiple mount protection(MMP) support within the ext4 filesystem itself. The superblock will have a block number (s_mmp_block) which will hold a MMP structure which has a sequence number which will be periodically updated every 5 seconds by a mounted filesystem. Whenever a filesystem will be mounted it will wait for s_mmp_interval seconds to make sure that the MMP sequence does not change. To further make sure, we write a random sequence number into the MMP block and wait for another s_mmp_interval secs. If the sequence no. doesn't change then the mount will succeed. In case of failure, the nodename, bdevname and the time at which the MMP block was last updated will be displa

ye

Quoted 1 line

d. tune2fs can be used to set s_mmp_interval as desired.

That will make laptop users very unhappy if you spin up their disks every 5 seconds. And even on other systems it might reduce the MTBF if you write the super block much more often than before. It might be better to set it up in some way to only increase that number when the super block is written for some other reason anyways.

-Andi

author
#14 lore raw
Quoted 5 lines · Andi Kleen
Quoted 3 lines · Kalpak Shah

Hi,

There have been reported instances of a filesystem having been mounted at 2 places at the same time causing a lot of damage to the filesystem. This patch reserves superblock fields and an INCOMPAT flag for adding multiple mount protection(MMP) support within the ext4 filesystem itself. The superblock will have a block number (s_mmp_block) which will hold a MMP structure which has a sequence number which will be periodically updated every 5 seconds by a mounted filesystem. Whenever a filesystem will be mounted it will wait for s_mmp_interval seconds to make sure that the MMP sequence does not change. To further make sure, we write a random sequence number into the MMP block and wait for another s_mmp_interval secs. If the sequence no. doesn't change then the mount will succeed. In case of failure, the nodename, bdevname and the time at which the MMP block was last updated will be disp

laye

Quoted 6 lines
Quoted 1 line

d. tune2fs can be used to set s_mmp_interval as desired.

That will make laptop users very unhappy if you spin up their disks every 5 seconds. And even on other systems it might reduce the MTBF if you write the super block much more often than before. It might be better to set it up in some way to only increase that number when the super block is written for some other reason anyways.

The super block only saves the block number of the MMP block. So the super block is not updated but the contents of the MMP block are updated every 5 seconds.

If any user is unhappy with the 5 seconds interval, he can modify the interval to be greater, with the caveat that it will take 2*mmp_interval seconds during mounting the filesystem.

Thanks,
Kalpak.

Quoted 2 lines

-Andi

#15 lore raw
In reply to Andi Kleen
Quoted 7 lines · Andi Kleen
Quoted 1 line · Kalpak Shah

There have been reported instances of a filesystem having been

mounted at 2 places at the same time causing a lot of damage to the filesystem.... The superblock will have a block number (s_mmp_block) which will hold a MMP structure which has a sequence number which will be periodically updated every 5 seconds by a mounted filesystem.

Quoted 5 lines

That will make laptop users very unhappy if you spin up their disks every 5 seconds. And even on other systems it might reduce the MTBF if you write the super block much more often than before. It might be better to set it up in some way to only increase that number when the super block is written for some other reason anyways.

It was mentioned before but deserves mentioning again that this will be an optional feature, mostly for use on SANs, iSCSI, etc where a disk might be accessed by multiple nodes at the same time. That means there will not be any impact for desktop users waiting 10s for each of their filesystems to mount.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

#16 lore raw
Quoted 5 lines

It was mentioned before but deserves mentioning again that this will be an optional feature, mostly for use on SANs, iSCSI, etc where a disk might be accessed by multiple nodes at the same time. That means there will not be any impact for desktop users waiting 10s for each of their filesystems to mount.

A safety feature that is optional? Doesn't sound very safe to me. If the safety is needed it should be probably default, otherwise it isn't needed.

-Andi

maintainer
#17 lore raw
In reply to Andi Kleen
Quoted 6 lines · Andi Kleen

That will make laptop users very unhappy if you spin up their disks every 5 seconds. And even on other systems it might reduce the MTBF if you write the super block much more often than before. It might be better to set it up in some way to only increase that number when the super block is written for some other reason anyways.

You would never want to use this feature on a laptop; it would buy no benefit for its costs, since with (all common) laptops, their hard drives can't be shared with other machines in a cluster.

Unfortunately, it's not possible to do what you suggest, since one of the whole points of increasing the sequence number every 5 seconds is to act as a keep-alive, so another machine trying to access the shared hard drive can tell whether or not the machine which currently had the hard drive mounted is still alive or not.

This is why I and others have been a little worried about implementing this feature, since it adds complexity which has to be in a proper HA system anyway, and what is there isn't really an optimal HA solution (since it lacks STONITH) and so you have to implement the functionality again _anyway_ using a proper HA solution.

The argument on the other side is that it protects against failed HA solutions, and against users who are too stupid to know that they need an HA solution. It does do the first; the second would only apply if the users who were too stupid to realize they needed an HA solution, were smart enough to enable it the MMP feature --- and because of its many costs, including keeping the disk spun up on laptops, and delaying the time required to mount the disk by 10 seconds, I don't think it will ever be enabled by default. Hence, I don't really think it helps the idiotic user problem.

But apparently a belt-and-suspenders approach to HA is comforting to some users, and so I don't mind reserving the space. The code to implement it still seems like more complexity than what should be in the kernel. My suggestion would be to put it in a separate file, and make it be something which has to be explicitly configured to enable it, possibly as a module (but that may add too much extra hair). I really don't think the save-the-stupid-user argument holds water, but the belt-and-suspenders argument IFF you are using a shared-disk setup is a valid, although probably not a common setup.

Regards,

- Ted

#18 lore raw
Quoted 3 lines

Unfortunately, it's not possible to do what you suggest, since one of the whole points of increasing the sequence number every 5 seconds is to act as a keep-alive, so another machine trying to access the shared

Clusters usually have other ways to do this, haven't they? Typically they have STONITH too. It's probably too simple minded to just replace a real cluster setup which also handles split brain and other conditions. So it's purely against mistakes.

Besides relying on it would seem dangerous because it is not synchronous and you could do a lot of damage in 5 seconds.

The rationale of the lazy writing would be that at least in usual operation super block should be written regularly and with 5 seconds delay just being a not fully reliable heuristic it probably doesn't matter if the possible delay is a little longer even.

-Andi

maintainer
#19 lore raw
Quoted 8 lines · Andi Kleen
Quoted 3 lines

Unfortunately, it's not possible to do what you suggest, since one of the whole points of increasing the sequence number every 5 seconds is to act as a keep-alive, so another machine trying to access the shared

Clusters usually have other ways to do this, haven't they? Typically they have STONITH too. It's probably too simple minded to just replace a real cluster setup which also handles split brain and other conditions. So it's purely against mistakes.

Yes, it's only real value is to protect against Cluster-HA malfunctions or misconfiguration.

Quoted 2 lines

Besides relying on it would seem dangerous because it is not synchronous and you could do a lot of damage in 5 seconds.

Well, the MMP feature is assigned an incompatible feature bit, so a kernel who doesn't know about MMP will refuse to touch it; and a kernel which does follow the MMP protocol will check the MMP block (delaying the mount by 10 seconds) to make sure no other system is using the block.

So aside from being !@#!@ annoying (which is why it will never be the default), it does work, modulo the problem that without STONITH or any kind of I/O fencing, we do risk the other system coming back to life and then modifying the filesystem in parallel. So as everyone has said, this is not solution that works in isolation, but is really only a backup.

The question of whether the complexity and then 10 second mount delay for what is only a backup solution is worth it is obviously going to be a very subjective one --- and as I've said previously, I'm on the fence on this.

- Ted

#20 lore raw
Quoted 8 lines · Theodore Tso
Quoted 4 lines · Andi Kleen

Clusters usually have other ways to do this, haven't they? Typically they have STONITH too. It's probably too simple minded to just replace a real cluster setup which also handles split brain and other conditions. So it's purely against mistakes.

Yes, it's only real value is to protect against Cluster-HA malfunctions or misconfiguration.

While I agree that HA systems _should_ be enough for this, in our experience even with an HA system some people get it wrong (e.g. manually mounting and bypassing HA, HA itself is broken, comms failure, STONITH failure, etc).

I agree it is not intended to be a replacement for an HA/STONITH solution, just belt & suspenders that would have saved hundreds of TB of user data in several cases if it were available. We will enable it by default on all of our filesystems, and of course I'd advise anyone in a SAN environment (whether they _intend_ to have shared disk access or not) to enable it also.

Quoted 8 lines
Quoted 2 lines

Besides relying on it would seem dangerous because it is not synchronous and you could do a lot of damage in 5 seconds.

Well, the MMP feature is assigned an incompatible feature bit, so a kernel who doesn't know about MMP will refuse to touch it; and a kernel which does follow the MMP protocol will check the MMP block (delaying the mount by 10 seconds) to make sure no other system is using the block.

Correct. There is a "fast path" where it will wait a shorter time during mount if the fs is reported cleanly unmounted. We can't skip the check entirely, because 2 systems might be mounting at the same time.

Quoted 6 lines

So aside from being !@#!@ annoying (which is why it will never be the default), it does work, modulo the problem that without STONITH or any kind of I/O fencing, we do risk the other system coming back to life and then modifying the filesystem in parallel. So as everyone has said, this is not solution that works in isolation, but is really only a backup.

If the kmmpd is not scheduled in more than 10s then it will re-read the block to ensure that the local system is still the one in control. If not, it will ext3_error() and (in our case at least) this will make the client fs read-only. Even if there is some IO leakage from the local client, this is far better than to continue running with 2 systems writing to the same disk.

Ideally there would also be a block-layer functionality to fence the IO on the local system (e.g. plug the elevator output, I don't think that there is anything that could be done about IO already submitted to the device), but the function I thought did this (set_device_rdonly()) is only checked at mount time and is useless.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.