Skip to contentSkip to conversation

re-enable non-clustered mount & add MMP support

PATCH In review Heming Zhao via Ocfs2-devel 1 participant 23 messages Jul 30, 2022 – Aug 10, 2022 OCFS2

lore
Share & tools
lore
#0 lore raw

cover re-enable non-clustered mount & add MMP support

This serial patches re-enable ocfs2 non-clustered mount feature.

the previous patch c80af0c250c8 (Revert "ocfs2: mount shared volume without ha stack") revert Gang's non-clustered mount patch. This serial patches re-enable ocfs2 non-clustered mount.

the key different between local mount and non-clustered mount:
local mount feature (tunefs.ocfs2 --fs-features=[no]local) can't do convert job without ha stack. non-clustered mount feature can run totally without ha stack.

For avoiding data corruption when non-clustered & clustered mount are happening at same time, this serial patches also introduces MMP feature. MMP (Multiple Mount Protection) idea got from ext4 MMP (fs/ext4/mmp.c) which protects fs from being mounted more than once. For ocfs2 is a clustered fs and also for compatible with existing slotmap feature, I did some optimization and modification when porting from ext4 MMP to ocfs2.

The related userspace code for supporting MMP had been sent to github for reviewing:
- https://github.com/markfasheh/ocfs2-tools/pull/58

ocfs2-tools enable MMP and check status:

```
# enable MMP
tunefs.ocfs2 --fs-feature=mmp /dev/vdb

# check the command result
tunefs.ocfs2 -Q "%H\n" /dev/vdb | grep MMP

# active MMP on nocluster mount
mount -t ocfs2 -o nocluster /dev/vdb /mnt

# check slotmap info
# echo slotmap | PAGER=cat debugfs.ocfs2 /dev/vdb
```

=== below are test cases for patches ====

<1> non-clustered mount vs local mount

1.1 tunefs.ocfs2 can't convert local/nolocal mount without ha stack.

```
(on ha stack env)
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
tunefs.ocfs2 --fs-features=local /dev/vdb  (<== success)
tunefs.ocfs2 --fs-features=nolocal /dev/vdb  (<== success)
(on another node without ha stack)
tunefs.ocfs2 --fs-features=local /dev/vdb  (<== failure)
```

1.2 non-cluster feature can run without ha stack.

```
(on ha stack env)
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
(on another node without ha stack)
mount -t ocfs2 -o nocluster /dev/vdb /mnt  (<== success)
```

<2> do clustered & non-clustered mount on same node

2.1 non-clustered mount => clustered mount

```
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
mount -t ocfs2 -o nocluster /dev/vdb /mnt
mount -t ocfs2 /dev/vdb /mnt               (<=== failure)
```

2.2 clustered mount => non-clustered mount

```
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
mount -t ocfs2 /dev/vdb /mnt
mount -t ocfs2 -o nocluster /dev/vdb /mnt  (<=== failure)
```

<3> one node does clustered mount, another does non-clustered mount

test rule: clustered mount and non-clustered mount can not exist at same time.

3.1 clustered mount @node1 => [no]clustered mount @node2

```
node1:
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
mount -t ocfs2 /dev/vdb /mnt

node2:
mount -t ocfs2 /dev/vdb /mnt              (<== success)
umount /mnt
mount -t ocfs2 -o nocluster /dev/vdb /mnt (<== failure)
```

3.2 enable mmp, repeate 3.1 case

```
node1:
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
tunefs.ocfs2 --fs-features=mmp /dev/vdb   (<== enable mmp)
mount -t ocfs2 /dev/vdb /mnt

node2:
mount -t ocfs2 /dev/vdb /mnt              (<== wait ~22s [*] for mmp,
then success)
umount /mnt
mount -t ocfs2 -o nocluster /dev/vdb /mnt (<== failure)
```

[*] 22s:
(OCFS2_MMP_MIN_CHECK_INTERVAL * 2 + 1) * 2 times (calling schedule_timeout_interruptible)

3.3 noclustered mount @node1 => [no]clustered mount @node2

```
node1:
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
mount -t ocfs2 -o nocluster /dev/vdb /mnt

node2:
mount -t ocfs2 /dev/vdb /mnt              (<== failure)
mount -t ocfs2 -o nocluster /dev/vdb /mnt (<== success, without mmp
enable)
umount /mnt               (<== will ZERO out slotmap area while node1
still mounting)
```

3.4 enable mmp, repeate 3.3 case.

```
node1:
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
tunefs.ocfs2 --fs-features=mmp /dev/vdb   (<== enable mmp)
mount -t ocfs2 -o nocluster /dev/vdb /mnt

node2:
mount -t ocfs2 /dev/vdb /mnt              (<== failure)
mount -t ocfs2 -o nocluster /dev/vdb /mnt (<== failure, denied by mmp)
```

<4> simulate mounting after machine crash

info:
- below all steps do on one node
- address 287387648 is the '//slot_map' extent address.
- test the rule: If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type.

4.0 how to calculate '//slot_map' extent address

```
# PAGER=cat debugfs.ocfs2 -R "stats" /dev/vdb | grep "Block Size Bits"
        Block Size Bits: 12   Cluster Size Bits: 12

# PAGER=cat debugfs.ocfs2 -R "stat //slot_map" /dev/vdb | grep -A1
# "Block#"
        ## Offset        Clusters       Block#          Flags
        0  0             1              70163           0x0
```

70163 * (1<<12) = 70163 * 4096 = 287387648

4.1 clustered mount => crash => non-clustered mount fails => clean slotmap => non-clustered mount succeeds

```
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
mount -t ocfs2 /dev/vdb /mnt
dd if=/dev/vdb bs=1 count=32 skip=287387648
of=/root/slotmap.cluster.mnted  (<== backup slot info)
umount /mnt
dd if=/root/slotmap.cluster.mnted of=/dev/vdb seek=287387648 bs=1
count=32 (<== overwrite)

mount -t ocfs2 -o nocluster /dev/vdb /mnt   <== failure
mount -t ocfs2 /dev/vdb /mnt && umount /mnt <== clean slot 0
mount -t ocfs2 -o nocluster /dev/vdb /mnt   <== success
```

4.2 non-clustered mount => crash => clustered mount fails => clean slotmap => clustered mount succeeds

```
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
mount -t ocfs2 -o nocluster /dev/vdb /mnt
dd if=/dev/vdb bs=1 count=32 skip=287387648
of=/root/slotmap.nocluster.mnted
umount /mnt
dd if=/root/slotmap.nocluster.mnted of=/dev/vdb seek=287387648 bs=1
count=32

mount -t ocfs2 /dev/vdb /mnt   <== failure
mount -t ocfs2 -o nocluster /dev/vdb /mnt && umount /mnt <== clean slot
0
mount -t ocfs2 /dev/vdb /mnt   <== success
```

<5> MMP test

5.1 node1 noclustered mount => node 2 noclustered mount

disable mmp

```
node1:
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
mount -t ocfs2 -o nocluster /dev/vdb /mnt

node2:
mount -t ocfs2 -o nocluster /dev/vdb /mnt (<== success)
```

enable mmp

```
node1:
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
tunefs.ocfs2 --fs-features=mmp /dev/vdb
mount -t ocfs2 -o nocluster /dev/vdb /mnt

node2:
mount -t ocfs2 -o nocluster /dev/vdb /mnt (<== wait ~12s[*], failure by
mmp)
```

[*] 12s:
sleep (OCFS2_MMP_MIN_CHECK_INTERVAL * 2 + 1) then detect mmp_seq was changed, then failed.

5.2 node1 clustered mount => node 2 clustered mount

see case 3.2

5.3 node1 noclustered mount => node 2 noclustered mount

see case 3.4

5.4 remount test

5.4.1 non-clustered mount (run commands on same node)

```
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
tunefs.ocfs2 --fs-features=mmp /dev/vdb

mount -t ocfs2 -o nocluster /dev/vdb /mnt
ps axj | grep kmmpd                            (<== will show kmmpd)
PAGER=cat debugfs.ocfs2 -R "slotmap" /dev/vdb  (<== show
'OCFS2_MMP_SEQ')

mount -o remount,ro,nocluster /dev/vdb /mnt    (<== kmmpd will stop)
ps axj | grep kmmpd  (<== won't show kmmpd)
PAGER=cat debugfs.ocfs2 -R "slotmap" /dev/vdb  (<== show
'OCFS2_MMP_SEQ_CLEAN')

mount -o remount,rw,nocluster /dev/vdb /mnt    (<== kmmpd will start)
ps axj | grep kmmpd  (<== will show kmmpd)
PAGER=cat debugfs.ocfs2 -R "slotmap" /dev/vdb  (<== show
'OCFS2_MMP_SEQ')
```

5.4.2 clustered mount

```
mkfs.ocfs2 --cluster-stack=pcmk --cluster-name=hacluster -N 4 /dev/vdb
tunefs.ocfs2 --fs-features=mmp /dev/vdb

mount -t ocfs2 /dev/vdb /mnt                   (<== clustered mount
won't create kmmpd)
PAGER=cat debugfs.ocfs2 -R "slotmap" /dev/vdb  (<== show
'OCFS2_VALID_CLUSTER')

mount -o remount,ro /dev/vdb /mnt
PAGER=cat debugfs.ocfs2 -R "slotmap" /dev/vdb  (<== show
'OCFS2_VALID_CLUSTER')

mount -o remount,rw /dev/vdb /mnt              (<== wait for ~22s by mmp
start)
PAGER=cat debugfs.ocfs2 -R "slotmap" /dev/vdb  (<== show
'OCFS2_VALID_CLUSTER')
```

Heming Zhao (4):
ocfs2: Fix freeing uninitialized resource on ocfs2_dlm_shutdown
ocfs2: add mlog ML_WARNING support
re-enable "ocfs2: mount shared volume without ha stack"
ocfs2: introduce ext4 MMP feature

 fs/ocfs2/cluster/masklog.c |   3 +
 fs/ocfs2/cluster/masklog.h |   9 +-
 fs/ocfs2/dlmglue.c         |   3 +
 fs/ocfs2/ocfs2.h           |   6 +-
 fs/ocfs2/ocfs2_fs.h        |  13 +-
 fs/ocfs2/slot_map.c        | 479 +++++++++++++++++++++++++++++++++++--
 fs/ocfs2/slot_map.h        |   3 +
 fs/ocfs2/super.c           |  42 +++-
 8 files changed, 527 insertions(+), 31 deletions(-)
Signature · 7 lines
2.37.1


_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
#1 lore raw

PATCH 1/4 ocfs2: Fix freeing uninitialized resource on ocfs2_dlm_shutdown

On local mount mode, there is no dlm resource initalized. If ocfs2_mount_volume() fails in ocfs2_find_slot(), error handling flow will call ocfs2_dlm_shutdown(), then does dlm resource cleanup job, which will trigger kernel crash.

Fixes 0737e01de9c4 ("ocfs2: ocfs2_mount_volume does cleanup job before

return error")

---
 fs/ocfs2/dlmglue.c | 3 +++
 1 file changed, 3 insertions(+)
fs/ocfs2/dlmglue.c +3 -0
index 801e60bab955..1438ac14940b 100644
@@ -3385,6 +3385,9 @@ int ocfs2_dlm_init(struct ocfs2_super *osb)
 void ocfs2_dlm_shutdown(struct ocfs2_super *osb,
 			int hangup_pending)
 {
+	if (ocfs2_mount_local(osb))
+		return;
+
 	ocfs2_drop_osb_locks(osb);
 
 	/*
Signature · 7 lines
2.37.1


_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
#2 lore raw

Re: [Ocfs2-devel] [PATCH 1/4] ocfs2: Fix freeing uninitialized resource on ocfs2_dlm_shutdown

Quoted 7 lines · Heming Zhao

On local mount mode, there is no dlm resource initalized. If ocfs2_mount_volume() fails in ocfs2_find_slot(), error handling flow will call ocfs2_dlm_shutdown(), then does dlm resource cleanup job, which will trigger kernel crash.

Fixes 0737e01de9c4 ("ocfs2: ocfs2_mount_volume does cleanup job before

return error")

Should be put at the same line.

Quoted 10 lines
---
 fs/ocfs2/dlmglue.c | 3 +++
 1 file changed, 3 insertions(+)
fs/ocfs2/dlmglue.c +0 -0
index 801e60bab955..1438ac14940b 100644

fs/ocfs2/dlmglue.c int ocfs2_dlm_init(struct ocfs2_super *osb) · line 3390

Quoted 6 lines
fs/ocfs2/dlmglue.c +3 -0
@@ -3385,6 +3385,9 @@ int ocfs2_dlm_init(struct ocfs2_super *osb)
 void ocfs2_dlm_shutdown(struct ocfs2_super *osb,
 			int hangup_pending)
 {
+	if (ocfs2_mount_local(osb))
+		return;
+

IMO, we have to do part of ocfs2_dlm_shutdown() jobs such as ocfs2_lock_res_free(), which will remove lockres from d_lockres_tracking added by ocfs2_xxx_lock_res_init().

Before commit 0737e01de9c4, it seems this issue also exists since osb->cconn is already set under local mount mode.

Thanks,
Joseph

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#3 lore raw

Re: [Ocfs2-devel] [PATCH 1/4] ocfs2: Fix freeing uninitialized resource on ocfs2_dlm_shutdown

Quoted 8 lines · Joseph Qi
Quoted 5 lines · Heming Zhao

On local mount mode, there is no dlm resource initalized. If ocfs2_mount_volume() fails in ocfs2_find_slot(), error handling flow will call ocfs2_dlm_shutdown(), then does dlm resource cleanup job, which will trigger kernel crash.

Quoted 4 lines
Quoted 2 lines
Fixes 0737e01de9c4 ("ocfs2: ocfs2_mount_volume does cleanup job before

return error")

Should be put at the same line.

OK

Quoted 6 lines
Quoted 6 lines
---
 fs/ocfs2/dlmglue.c | 3 +++
 1 file changed, 3 insertions(+)
Quoted 16 lines

fs/ocfs2/dlmglue.c int ocfs2_dlm_init(struct ocfs2_super *osb) · line 3390

Quoted 11 lines
fs/ocfs2/dlmglue.c +3 -0
index 801e60bab955..1438ac14940b 100644
@@ -3385,6 +3385,9 @@ int ocfs2_dlm_init(struct ocfs2_super *osb)
 void ocfs2_dlm_shutdown(struct ocfs2_super *osb,
 			int hangup_pending)
 {
+	if (ocfs2_mount_local(osb))
+		return;
+

IMO, we have to do part of ocfs2_dlm_shutdown() jobs such as ocfs2_lock_res_free(), which will remove lockres from d_lockres_tracking added by ocfs2_xxx_lock_res_init().

ocfs2_dlm_shutdown does the cleanup job for ocfs2_dlm_init. This patch fixed crash in local mount error flow.
In local mount mode, ocfs2_dlm_init does nothing, which should make ocfs2_dlm_shutdown do nothing.

And I checked all calling ocfs2_dlm_shutdown cases:
1. mount flow:

ocfs2_fill_super
+ xxx =fails=> label:out_super (checked, work fine)
|
+ ocfs2_mount_volume =fails=> label:out_debugfs (checked, work fine) | |
| + xxx =fails=> cleanup everything before returning
|
+ xxx =fails=> label:out_dismout
At this time, dlm has been init successfully, we can call all lines of ocfs2_dlm_shutdown.

2. ocfs2_dismount_volume => 'osb->cconn' is true.
this MUST be dlm successfully init case. everything looks fine.

In previous mail/patch: [PATCH] test error handling during mount stage, I may forget to test local mount mode. So this crash didn't be triggered.

Quoted 2 lines

Before commit 0737e01de9c4, it seems this issue also exists since osb->cconn is already set under local mount mode.

Yes. The bug exists since local mount feature was introduced, commit number:
c271c5c22b0a7ca45fda15f1f4d258bca36a5b94. I will change 'Fixes' on next version.

(Hope 'CC' list takes effect for this mail. -_-# )

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#4 lore raw

Re: [Ocfs2-devel] [PATCH 1/4] ocfs2: Fix freeing uninitialized resource on ocfs2_dlm_shutdown

Quoted 38 lines · Heming Zhao
Quoted 8 lines · Joseph Qi
Quoted 5 lines · Heming Zhao

On local mount mode, there is no dlm resource initalized. If ocfs2_mount_volume() fails in ocfs2_find_slot(), error handling flow will call ocfs2_dlm_shutdown(), then does dlm resource cleanup job, which will trigger kernel crash.

Quoted 4 lines
Quoted 2 lines
Fixes 0737e01de9c4 ("ocfs2: ocfs2_mount_volume does cleanup job before

return error")

Should be put at the same line.

OK

Quoted 6 lines
Quoted 6 lines
---
 fs/ocfs2/dlmglue.c | 3 +++
 1 file changed, 3 insertions(+)
Quoted 16 lines

fs/ocfs2/dlmglue.c int ocfs2_dlm_init(struct ocfs2_super *osb) · line 3390

Quoted 11 lines
fs/ocfs2/dlmglue.c +3 -0
index 801e60bab955..1438ac14940b 100644
@@ -3385,6 +3385,9 @@ int ocfs2_dlm_init(struct ocfs2_super *osb)
 void ocfs2_dlm_shutdown(struct ocfs2_super *osb,
 			int hangup_pending)
 {
+	if (ocfs2_mount_local(osb))
+		return;
+

IMO, we have to do part of ocfs2_dlm_shutdown() jobs such as ocfs2_lock_res_free(), which will remove lockres from d_lockres_tracking added by ocfs2_xxx_lock_res_init().

Quoted 5 lines

ocfs2_dlm_shutdown does the cleanup job for ocfs2_dlm_init. This patch fixed crash in local mount error flow.
In local mount mode, ocfs2_dlm_init does nothing, which should make ocfs2_dlm_shutdown do nothing.

Not exactly, even in local mount we have to initialize lockres like super lock.

Quoted 18 lines

And I checked all calling ocfs2_dlm_shutdown cases:
1. mount flow:

ocfs2_fill_super
+ xxx =fails=> label:out_super (checked, work fine)
|
+ ocfs2_mount_volume =fails=> label:out_debugfs (checked, work fine) | |
| + xxx =fails=> cleanup everything before returning
|
+ xxx =fails=> label:out_dismout
At this time, dlm has been init successfully, we can call all lines of ocfs2_dlm_shutdown.

2. ocfs2_dismount_volume => 'osb->cconn' is true.
this MUST be dlm successfully init case. everything looks fine.

Quoted 9 lines

In previous mail/patch: [PATCH] test error handling during mount stage, I may forget to test local mount mode. So this crash didn't be triggered.

Quoted 2 lines

Before commit 0737e01de9c4, it seems this issue also exists since osb->cconn is already set under local mount mode.

Yes. The bug exists since local mount feature was introduced, commit number:
c271c5c22b0a7ca45fda15f1f4d258bca36a5b94. I will change 'Fixes' on next version.

This patch can be a separate fix and we don't have to bind with nocluster mount feature, even though you've triggered it by related local mount.

Thanks,
Joseph

Quoted 4 lines

(Hope 'CC' list takes effect for this mail. -_-# )

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#5 lore raw

Re: [Ocfs2-devel] [PATCH 1/4] ocfs2: Fix freeing uninitialized resource on ocfs2_dlm_shutdown

Quoted 41 lines · Joseph Qi
Quoted 38 lines · Heming Zhao
Quoted 8 lines · Joseph Qi
Quoted 5 lines · Heming Zhao

On local mount mode, there is no dlm resource initalized. If ocfs2_mount_volume() fails in ocfs2_find_slot(), error handling flow will call ocfs2_dlm_shutdown(), then does dlm resource cleanup job, which will trigger kernel crash.

Quoted 4 lines
Quoted 2 lines
Fixes 0737e01de9c4 ("ocfs2: ocfs2_mount_volume does cleanup job before

return error")

Should be put at the same line.

OK

Quoted 22 lines
Quoted 11 lines
---
  fs/ocfs2/dlmglue.c | 3 +++
  1 file changed, 3 insertions(+)
fs/ocfs2/dlmglue.c +0 -0
index 801e60bab955..1438ac14940b 100644

fs/ocfs2/dlmglue.c int ocfs2_dlm_init(struct ocfs2_super *osb) · line 3390

Quoted 6 lines
fs/ocfs2/dlmglue.c +3 -0
@@ -3385,6 +3385,9 @@ int ocfs2_dlm_init(struct ocfs2_super *osb)
  void ocfs2_dlm_shutdown(struct ocfs2_super *osb,
  			int hangup_pending)
  {
+	if (ocfs2_mount_local(osb))
+		return;
+

IMO, we have to do part of ocfs2_dlm_shutdown() jobs such as ocfs2_lock_res_free(), which will remove lockres from d_lockres_tracking added by ocfs2_xxx_lock_res_init().

Quoted 7 lines
Quoted 5 lines

ocfs2_dlm_shutdown does the cleanup job for ocfs2_dlm_init. This patch fixed crash in local mount error flow.
In local mount mode, ocfs2_dlm_init does nothing, which should make ocfs2_dlm_shutdown do nothing.

Not exactly, even in local mount we have to initialize lockres like super lock.

Thank you for your patient explaination, I understood your meaning, will change it in next version.
I am very busy these days, it makes my brain stop working.

Quoted 26 lines
Quoted 26 lines

And I checked all calling ocfs2_dlm_shutdown cases:
1. mount flow:

ocfs2_fill_super
+ xxx =fails=> label:out_super (checked, work fine)
|
+ ocfs2_mount_volume =fails=> label:out_debugfs (checked, work fine) | |
| + xxx =fails=> cleanup everything before returning
|
+ xxx =fails=> label:out_dismout
At this time, dlm has been init successfully, we can call all lines of ocfs2_dlm_shutdown.

2. ocfs2_dismount_volume => 'osb->cconn' is true.
this MUST be dlm successfully init case. everything looks fine.

In previous mail/patch: [PATCH] test error handling during mount stage, I may forget to test local mount mode. So this crash didn't be triggered.

Quoted 2 lines

Before commit 0737e01de9c4, it seems this issue also exists since osb->cconn is already set under local mount mode.

Quoted 6 lines
Quoted 3 lines

Yes. The bug exists since local mount feature was introduced, commit number:
c271c5c22b0a7ca45fda15f1f4d258bca36a5b94. I will change 'Fixes' on next version.

This patch can be a separate fix and we don't have to bind with nocluster mount feature, even though you've triggered it by related local mount.

OK.

Quoted 8 lines

Thanks,
Joseph

Quoted 4 lines

(Hope 'CC' list takes effect for this mail. -_-# )

Thanks,
Heming

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#6 lore raw

PATCH 2/4 ocfs2: add mlog ML_WARNING support

In reply to Heming Zhao via Ocfs2-devel

This commit gives new message type for ocfs2.

---
 fs/ocfs2/cluster/masklog.c | 3 +++
 fs/ocfs2/cluster/masklog.h | 9 +++++----
 2 files changed, 8 insertions(+), 4 deletions(-)
fs/ocfs2/cluster/masklog.c +3 -0
index 563881ddbf00..bac3488e8002 100644
@@ -63,6 +63,9 @@ void __mlog_printk(const u64 *mask, const char *func, int line,
 	if (*mask & ML_ERROR) {
 		level = KERN_ERR;
 		prefix = "ERROR: ";
+	} else if (*mask & ML_WARNING) {
+		level = KERN_WARNING;
+		prefix = "WARNING: ";
 	} else if (*mask & ML_NOTICE) {
 		level = KERN_NOTICE;
 	} else {
fs/ocfs2/cluster/masklog.h +5 -4
index b73fc42e46ff..d0bc4fe8cf3d 100644
@@ -86,10 +86,11 @@
 
 /* bits that are infrequently given and frequently matched in the high word */
 #define ML_ERROR	0x1000000000000000ULL /* sent to KERN_ERR */
-#define ML_NOTICE	0x2000000000000000ULL /* setn to KERN_NOTICE */
-#define ML_KTHREAD	0x4000000000000000ULL /* kernel thread activity */
+#define ML_NOTICE	0x2000000000000000ULL /* sent to KERN_NOTICE */
+#define ML_WARNING	0x4000000000000000ULL /* sent to KERN_WARNING */
+#define ML_KTHREAD	0x8000000000000000ULL /* kernel thread activity */
 
-#define MLOG_INITIAL_AND_MASK (ML_ERROR|ML_NOTICE)
+#define MLOG_INITIAL_AND_MASK (ML_ERROR|ML_WARNING|ML_NOTICE)
 #ifndef MLOG_MASK_PREFIX
 #define MLOG_MASK_PREFIX 0
 #endif
@@ -102,7 +103,7 @@
 #if defined(CONFIG_OCFS2_DEBUG_MASKLOG)
 #define ML_ALLOWED_BITS ~0
 #else
-#define ML_ALLOWED_BITS (ML_ERROR|ML_NOTICE)
+#define ML_ALLOWED_BITS (ML_ERROR|ML_WARNING|ML_NOTICE)
 #endif
 
 #define MLOG_MAX_BITS 64
Signature · 7 lines
2.37.1


_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
#7 lore raw

PATCH 3/4 re-enable "ocfs2: mount shared volume without ha stack"

In reply to Heming Zhao via Ocfs2-devel

the key different between local mount and non-clustered mount:
local mount feature (tunefs.ocfs2 --fs-features=[no]local) can't do convert job without ha stack. non-clustered mount feature can run totally without ha stack.

commit 912f655d78c5 ("ocfs2: mount shared volume without ha stack") had bug, then commit c80af0c250c8f8a3c978aa5aafbe9c39b336b813 reverted it.

Let's give some explain for the issue mentioned by commit c80af0c250c8.

Under Junxiao's call trace, in __ocfs2_find_empty_slot(), the 'if' accessment is wrong. sl_node_num could be 0 at o2cb env.

with current information, the trigger flow (base on 912f655d78c5):
1>
nodeA with 'node_num = 0' for mounting. it will succeed. at this time, slotmap extent block will contains es_valid:1 & es_node_num:0 for nodeA
then ocfs2_update_disk_slot() will write back slotmap info to disk.

2>
then, nodeB with 'node_num = 1' for mounting
this time, osb->node_num is 1 (set by config file), osb->preferred is OCFS2_INVALID_SLOT (set by ocfs2_parse_options).

ocfs2_find_slot
+ ocfs2_update_slot_info //read slotmap info from disk
| + set si->si_slots[0].es_valid = 1 & si->si_slots[0].sl_node_num = 0 |
+ __ocfs2_node_num_to_slot //will return -ENOENT.
+ __ocfs2_find_empty_slot
+ if ((preferred >= 0) && (preferred < si->si_num_slots)) | fails enter this 'if' for preferred value is OCFS2_INVALID_SLOT |
+ 'for(i = 0; i < si->si_num_slots; i++)' search slot 0 successfully.
| 'si->si_slots[0].sl_node_num' is false. trigger 'break' condition. |
+ return slot 0.
it will cause nodeB grab nodeA journal dlm lock, then trigger hung.

How to do for this bug?

This commit re-enabled 912f655d78c5, next commit (add MMP support) will fix the issue.

---
 fs/ocfs2/ocfs2.h    |  4 +++-
 fs/ocfs2/slot_map.c | 46 ++++++++++++++++++++++++++-------------------
 fs/ocfs2/super.c    | 21 +++++++++++++++++++++
 3 files changed, 51 insertions(+), 20 deletions(-)
fs/ocfs2/ocfs2.h +3 -1
index 740b64238312..337527571461 100644
@@ -277,6 +277,7 @@ enum ocfs2_mount_options
 	OCFS2_MOUNT_JOURNAL_ASYNC_COMMIT = 1 << 15,  /* Journal Async Commit */
 	OCFS2_MOUNT_ERRORS_CONT = 1 << 16, /* Return EIO to the calling process on error */
 	OCFS2_MOUNT_ERRORS_ROFS = 1 << 17, /* Change filesystem to read-only on error */
+	OCFS2_MOUNT_NOCLUSTER = 1 << 18, /* No cluster aware filesystem mount */
 };
 
 #define OCFS2_OSB_SOFT_RO	0x0001
@@ -672,7 +673,8 @@ static inline int ocfs2_cluster_o2cb_global_heartbeat(struct ocfs2_super *osb)
 
 static inline int ocfs2_mount_local(struct ocfs2_super *osb)
 {
-	return (osb->s_feature_incompat & OCFS2_FEATURE_INCOMPAT_LOCAL_MOUNT);
+	return ((osb->s_feature_incompat & OCFS2_FEATURE_INCOMPAT_LOCAL_MOUNT)
+		|| (osb->s_mount_opt & OCFS2_MOUNT_NOCLUSTER));
 }
 
 static inline int ocfs2_uses_extended_slot_map(struct ocfs2_super *osb)
fs/ocfs2/slot_map.c +27 -19

65 lines not on this page · raw message

fs/ocfs2/super.c +21 -0
index 438be028935d..f7298816d8d9 100644
@@ -172,6 +172,7 @@ enum {
 	Opt_dir_resv_level,
 	Opt_journal_async_commit,
 	Opt_err_cont,
+	Opt_nocluster,
 	Opt_err,
 };
 
@@ -205,6 +206,7 @@ static const match_table_t tokens = {
 	{Opt_dir_resv_level, "dir_resv_level=%u"},
 	{Opt_journal_async_commit, "journal_async_commit"},
 	{Opt_err_cont, "errors=continue"},
+	{Opt_nocluster, "nocluster"},
 	{Opt_err, NULL}
 };
 
@@ -616,6 +618,13 @@ static int ocfs2_remount(struct super_block *sb, int *flags, char *data)
 		goto out;
 	}
 
+	tmp = OCFS2_MOUNT_NOCLUSTER;
+	if ((osb->s_mount_opt & tmp) != (parsed_options.mount_opt & tmp)) {
+		ret = -EINVAL;
+		mlog(ML_ERROR, "Cannot change nocluster option on remount\n");
+		goto out;
+	}
+
 	tmp = OCFS2_MOUNT_HB_LOCAL | OCFS2_MOUNT_HB_GLOBAL |
 		OCFS2_MOUNT_HB_NONE;
 	if ((osb->s_mount_opt & tmp) != (parsed_options.mount_opt & tmp)) {
@@ -856,6 +865,7 @@ static int ocfs2_verify_userspace_stack(struct ocfs2_super *osb,
 	}
 
 	if (ocfs2_userspace_stack(osb) &&
+	    !(osb->s_mount_opt & OCFS2_MOUNT_NOCLUSTER) &&
 	    strncmp(osb->osb_cluster_stack, mopt->cluster_stack,
 		    OCFS2_STACK_LABEL_LEN)) {
 		mlog(ML_ERROR,
@@ -1127,6 +1137,11 @@ static int ocfs2_fill_super(struct super_block *sb, void *data, int silent)
 	       osb->s_mount_opt & OCFS2_MOUNT_DATA_WRITEBACK ? "writeback" :
 	       "ordered");
 
+	if ((osb->s_mount_opt & OCFS2_MOUNT_NOCLUSTER) &&
+	   !(osb->s_feature_incompat & OCFS2_FEATURE_INCOMPAT_LOCAL_MOUNT))
+		printk(KERN_NOTICE "ocfs2: The shared device (%s) is mounted "
+		       "without cluster aware mode.\n", osb->dev_str);
+
 	atomic_set(&osb->vol_state, VOLUME_MOUNTED);
 	wake_up(&osb->osb_mount_event);
 
@@ -1437,6 +1452,9 @@ static int ocfs2_parse_options(struct super_block *sb,
 		case Opt_journal_async_commit:
 			mopt->mount_opt |= OCFS2_MOUNT_JOURNAL_ASYNC_COMMIT;
 			break;
+		case Opt_nocluster:
+			mopt->mount_opt |= OCFS2_MOUNT_NOCLUSTER;
+			break;
 		default:
 			mlog(ML_ERROR,
 			     "Unrecognized mount option \"%s\" "
@@ -1548,6 +1566,9 @@ static int ocfs2_show_options(struct seq_file *s, struct dentry *root)
 	if (opts & OCFS2_MOUNT_JOURNAL_ASYNC_COMMIT)
 		seq_printf(s, ",journal_async_commit");
 
+	if (opts & OCFS2_MOUNT_NOCLUSTER)
+		seq_printf(s, ",nocluster");
+
 	return 0;
 }
 
Signature · 7 lines
2.37.1


_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
#8 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

Hi Heming,

Quoted 5 lines · On Fri, Jul 29, 2022 at 6:15 PM Heming Zhao via Ocfs2-devel…

the key different between local mount and non-clustered mount:
local mount feature (tunefs.ocfs2 --fs-features=[no]local) can't do convert job without ha stack. non-clustered mount feature can run totally without ha stack.

Can you please elaborate on this? Local mounts can run without a cluster stack so I don't see the difference there. We have tunefs.ocfs2 look for and join the cluster so as to avoid corrupting users data - that's a feature, not a bug. So what I'm seeing here is just opening us to potential corruptions. Is there a specific use case here that you're trying to account for? Are you fixing a particular bug?

Thanks,
--Mark

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#9 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

Hello Mark,

Quoted 5 lines · Mark Fasheh

Hi Heming,

Quoted 1 line · On Fri, Jul 29, 2022 at 6:15 PM Heming Zhao via Ocfs2-devel…
Quoted 7 lines
Quoted 4 lines

the key different between local mount and non-clustered mount:
local mount feature (tunefs.ocfs2 --fs-features=[no]local) can't do convert job without ha stack. non-clustered mount feature can run totally without ha stack.

Can you please elaborate on this? Local mounts can run without a cluster stack so I don't see the difference there. We have

I am using pacemaker cluster stack. In my env, the trouble of the converting between local and clustered mounts are only happening on cluster stack.

the non-clustered mount feature (Gang He commit: 912f655d78c5) gave ocfs2 the ability to mount volume at any env (with/without cluster stack).
The 912f655d78c5 derived from SUSE customer complain: User wanted to fsck the backup ocfs2 volume in non-clustered env. They wanted to access the volume quickly and didn't want to take time/resource to set up HA stack. (by the way, pcmk stack at least needs two nodes to set up a cluster.)

Quoted 6 lines

tunefs.ocfs2 look for and join the cluster so as to avoid corrupting users data - that's a feature, not a bug. So what I'm seeing here is just opening us to potential corruptions. Is there a specific use case here that you're trying to account for? Are you fixing a particular bug?

Tunefs.ocfs2 still needs HA/dlm stack to protect joining action. commit 912f655d78c5 works on non-clustered env, which needs other tech (eg. MMP) to protect corrupting.

From my viewpoint, the non-clustered mount code is based on local mount code, which gives more flexible than local mount. non-clustered mount uses unify mount style align with clustered mount. I think users will like more to use non-clustered mount than using tunefs.ocfs2 to change mount type.

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#10 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

I found I didn't include Joseph in previous mail.

Quoted 19 lines · On 8/1/22 09:01, heming.zhao--- via Ocfs2-devel wrote:

Hello Mark,

Quoted 5 lines · Mark Fasheh

Hi Heming,

Quoted 1 line · On Fri, Jul 29, 2022 at 6:15 PM Heming Zhao via Ocfs2-devel…
Quoted 7 lines
Quoted 4 lines

the key different between local mount and non-clustered mount:
local mount feature (tunefs.ocfs2 --fs-features=[no]local) can't do convert job without ha stack. non-clustered mount feature can run totally without ha stack.

Can you please elaborate on this? Local mounts can run without a cluster stack so I don't see the difference there. We have

I am using pacemaker cluster stack. In my env, the trouble of the converting between local and clustered mounts are only happening on cluster stack.

Quoted 6 lines

the non-clustered mount feature (Gang He commit: 912f655d78c5) gave ocfs2 the ability to mount volume at any env (with/without cluster stack).
The 912f655d78c5 derived from SUSE customer complain: User wanted to fsck the backup ocfs2 volume in non-clustered env. They wanted to access the volume quickly and didn't want to take time/resource to set up HA stack. (by the way, pcmk stack at least needs two nodes to set up a cluster.)

Ooh, above "needs two nodes to set up a cluster" is wrong. User could use a special corosync.conf to set up a single node cluster. But anyhow, the key is why we can't bypass the setup ha stack step.

Quoted 11 lines
Quoted 7 lines

tunefs.ocfs2 look for and join the cluster so as to avoid corrupting users data - that's a feature, not a bug. So what I'm seeing here is just opening us to potential corruptions. Is there a specific use case here that you're trying to account for? Are you fixing a particular bug?

Tunefs.ocfs2 still needs HA/dlm stack to protect joining action. commit 912f655d78c5 works on non-clustered env, which needs other tech (eg. MMP) to protect corrupting.

Quoted 7 lines

From my viewpoint, the non-clustered mount code is based on local mount code,
which gives more flexible than local mount. non-clustered mount uses unify mount style align with clustered mount. I think users will like more to use non-clustered mount than using tunefs.ocfs2 to change mount type.

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#11 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

In reply to heming.zhao--- via Ocfs2-devel

Hi Heming,

Quoted 20 lines · On Sun, Jul 31, 2022 at 6:02 PM [email protected] <[email protected]>…

Hello Mark,

Quoted 5 lines · Mark Fasheh

Hi Heming,

Quoted 1 line · On Fri, Jul 29, 2022 at 6:15 PM Heming Zhao via Ocfs2-devel…
Quoted 7 lines
Quoted 4 lines

the key different between local mount and non-clustered mount:
local mount feature (tunefs.ocfs2 --fs-features=[no]local) can't do convert job without ha stack. non-clustered mount feature can run totally without ha stack.

Can you please elaborate on this? Local mounts can run without a cluster stack so I don't see the difference there. We have

I am using pacemaker cluster stack. In my env, the trouble of the converting between local and clustered mounts are only happening on cluster stack.

Quoted 6 lines

the non-clustered mount feature (Gang He commit: 912f655d78c5) gave ocfs2 the ability to mount volume at any env (with/without cluster stack).
The 912f655d78c5 derived from SUSE customer complain: User wanted to fsck the backup ocfs2 volume in non-clustered env. They wanted to access the volume quickly and didn't want to take time/resource to set up HA stack. (by the way, pcmk stack at least needs two nodes to set up a cluster.)

Ok. I had some time to look over the ext4 mmp patches. I feel like there are two questions here:

1) Is MMP a useful feature for Ocfs2

My answer is yes absolutely, the user should have the option to enable this on local mounts.

2) Should we allow the user to bypass our cluster checks?

On this question I'm still a 'no'. I simply haven't seen enough evidence to warrant such a drastic change in policy. Allowing it via mount option too just feels extremely error-prone. I think we need to explore alternative avenues to help
ing the user out here. As you noted in your followup, a single node config is entirely possible in pacemaker (I've run that config myself). Why not provide an easy way for the user to drop down to that sort of a config? I know that's kind
of pushing responsibility for this to the cluster stack, but that's where it belongs in the first place.

Another option might be an 'observer mode' mount, where the node participates in the cluster (and the file system locking) but purely in a read-only fashion.

Quoted 9 lines
Quoted 6 lines

tunefs.ocfs2 look for and join the cluster so as to avoid corrupting users data - that's a feature, not a bug. So what I'm seeing here is just opening us to potential corruptions. Is there a specific use case here that you're trying to account for? Are you fixing a particular bug?

Tunefs.ocfs2 still needs HA/dlm stack to protect joining action. commit 912f655d78c5 works on non-clustered env, which needs other tech (eg. MMP) to protect corrupting.

FWIW, I'm glad that we pulled commit 912f655d78c5 and I do not think we should go back to that paradigm.

Quoted 4 lines

From my viewpoint, the non-clustered mount code is based on local mount code,
which gives more flexible than local mount. non-clustered mount uses unify mount style align with clustered mount. I think users will like more to use non-clustered mount than using tunefs.ocfs2 to change mount type.

Can we get rid of the mount option, and make mmp something that users can turn on for Ocfs2 local mounts? I don't recall if we make a slot map for local mounts or not but it wouldn't be difficult to add that.

Btw, thank you very much for all of these patches, and also thank you for the very detailed test cases in your initial email. I'll try to give the actual code a review as well.

Thanks,
--Mark

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#12 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

Quoted 13 lines · Mark Fasheh

2) Should we allow the user to bypass our cluster checks?

On this question I'm still a 'no'. I simply haven't seen enough evidence to warrant such a drastic change in policy. Allowing it via mount option too just feels extremely error-prone. I think we need to explore alternative avenues to help
ing the user out here. As you noted in your followup, a single node config is entirely possible in pacemaker (I've run that config myself). Why not provide an easy way for the user to drop down to that sort of a config? I know that's kind
of pushing responsibility for this to the cluster stack, but that's where it belongs in the first place.

Quoted 3 lines

Another option might be an 'observer mode' mount, where the node participates in the cluster (and the file system locking) but purely in a read-only fashion.

Thinking about this some more... The only way that this works without potential corruptions is if we always write a periodic mmp sequence, even in clustered mode (which might mean each node writes to its own sector). That way tunefs can always check the disk for a mounted node, even without a cluster stack up. If tunefs sees anyone writing sequences to the disk, it can safely fail the operation. Tunefs also would have to be writing an mmp sequence once it has determined that the disk is not mounted. It could also write some flag alongisde the sequence that says 'tunefs is working on this disk'. If a cluster mount comes up and sees a live sequence with that flag, it will know to fail the mount request as the disk is being modified. Local mounts can also use this to ensure that they are the only mounted node.

As it turns out, we already do pretty much all of the sequence writing already for the o2cb cluster stack - check out cluseter/heartbeat.c. If memory serves, tunefs.ocfs2 has code to write to this heartbeat area as well. For o2cb, we use the disk heartbeat to detect node liveness, and to kill our local node if we see disk timeouts. For pcmk, we shouldn't take any of these actions as it is none of our responsibility. Under pcmk, the heartbeating would be purely for mount protection checks.

The downside to this is that all nodes would be heartbeating to the disk on a regular interval, not just one. To be fair, this is exactly how o2cb works and with the correct timeout choices, we were able to avoid a measurable performance impact, though in any case this might have to be a small price the user pays for cluster aware mount protection.

Let me know what you think.

Thanks,
--Mark

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#13 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

Hi Mark,

Quoted 18 lines · Mark Fasheh
Quoted 13 lines · Mark Fasheh

2) Should we allow the user to bypass our cluster checks?

On this question I'm still a 'no'. I simply haven't seen enough evidence to warrant such a drastic change in policy. Allowing it via mount option too just feels extremely error-prone. I think we need to explore alternative avenues to help
ing the user out here. As you noted in your followup, a single node config is entirely possible in pacemaker (I've run that config myself). Why not provide an easy way for the user to drop down to that sort of a config? I know that's kind
of pushing responsibility for this to the cluster stack, but that's where it belongs in the first place.

Quoted 3 lines

Another option might be an 'observer mode' mount, where the node participates in the cluster (and the file system locking) but purely in a read-only fashion.

Quoted 13 lines

Thinking about this some more... The only way that this works without potential corruptions is if we always write a periodic mmp sequence, even in clustered mode (which might mean each node writes to its own sector). That way tunefs can always check the disk for a mounted node, even without a cluster stack up. If tunefs sees anyone writing sequences to the disk, it can safely fail the operation. Tunefs also would have to be writing an mmp sequence once it has determined that the disk is not mounted. It could also write some flag alongisde the sequence that says 'tunefs is working on this disk'. If a cluster mount comes up and sees a live sequence with that flag, it will know to fail the mount request as the disk is being modified. Local mounts can also use this to ensure that they are the only mounted node.

Above tunefs check & write seq steps are mmp feature work flow.

Your mentioned tunefs work flow matches my patch[4/4] idea, and this

patch does all the works in kernel ocfs2_find_slot().

Because sequences should be saved in SB, update it should grab ->osb_lock,

which influence the performance.

And for saving seqs, for saving space, we won't alloc different disk block for different node.
If multi-nodes share the same disk block (eg, keep using slot_map
for saving seqs),
the updating job will make IO performance issue.

For fixing performance issue, in my view, we should disable mmp sequences

updating when mounting mode is clustered. So I make a rule:

** If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type. **

above rule another meaning:

new coming node mounting type should same with exist mounting type.

and there is a basic knowledge:

current ocfs2 code under cluster stack already have the ability to prevent multiple mount when mounting type is clustered.

(from patch 4/4) there are mount labels:

+#define OCFS2_MMP_SEQ_CLEAN 0xFF4D4D50U /* mmp_seq value for clean unmount */

+#define OCFS2_MMP_SEQ_FSCK 0xE24D4D50U /* mmp_seq value when being fscked */

+#define OCFS2_MMP_SEQ_MAX 0xE24D4D4FU /* maximum valid mmp_seq value */

+#define OCFS2_MMP_SEQ_INIT 0x0 /* mmp_seq init value */

+#define OCFS2_VALID_CLUSTER 0xE24D4D55U /* value for clustered mount

+ under MMP disabled */

+#define OCFS2_VALID_NOCLUSTER 0xE24D4D5AU /* value for noclustered mount

+ under MMP disabled */

whenever mount successfully, there should be three types living labels

in slotmap area:

- OCFS2_MMP_SEQ_CLEAN, OCFS2_VALID_NOCLUSTER - for local/non-clustered mount

- OCFS2_VALID_CLUSTER - for clustered mount

new coming node will check if any slot contains living labels.

whenever unmount successfully, there should be two types left labels in slotmap
area:
-
OCFS2_MMP_SEQ_CLEAN or 0 (zero)

when a node does unmount, according to the mount type, it will clean (zeroed)

or write OCFS2_MMP_SEQ_CLEAN in the slot.

Quoted 9 lines

As it turns out, we already do pretty much all of the sequence writing already for the o2cb cluster stack - check out cluseter/heartbeat.c. If memory serves, tunefs.ocfs2 has code to write to this heartbeat area as well. For o2cb, we use the disk heartbeat to detect node liveness, and to kill our local node if we see disk timeouts. For pcmk, we shouldn't take any of these actions as it is none of our responsibility. Under pcmk, the heartbeating would be purely for mount protection checks.

From my thinking, under cluster stack, there is enough protecting logic. For local/non-clustered mount, mmp will give the ability for detecting node liveness.

So I only enable kmmpd-<dev> kthread for local/non-clustered mount.

Quoted 6 lines

The downside to this is that all nodes would be heartbeating to the disk on a regular interval, not just one. To be fair, this is exactly how o2cb works and with the correct timeout choices, we were able to avoid a measurable performance impact, though in any case this might have to be a small price the user pays for cluster aware mount protection.

I already considered this performance issue in patch [4/4].

Quoted 5 lines

Let me know what you think.

Thanks,
--Mark

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#14 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

In reply to Mark Fasheh via Ocfs2-devel

Hello Mark and Joseph,

please ignore my previous mess formated reply mail

Quoted 18 lines · Mark Fasheh
Quoted 13 lines · Mark Fasheh

2) Should we allow the user to bypass our cluster checks?

On this question I'm still a 'no'. I simply haven't seen enough evidence to warrant such a drastic change in policy. Allowing it via mount option too just feels extremely error-prone. I think we need to explore alternative avenues to help
ing the user out here. As you noted in your followup, a single node config is entirely possible in pacemaker (I've run that config myself). Why not provide an easy way for the user to drop down to that sort of a config? I know that's kind
of pushing responsibility for this to the cluster stack, but that's where it belongs in the first place.

Quoted 3 lines

Another option might be an 'observer mode' mount, where the node participates in the cluster (and the file system locking) but purely in a read-only fashion.

Quoted 13 lines

Thinking about this some more... The only way that this works without potential corruptions is if we always write a periodic mmp sequence, even in clustered mode (which might mean each node writes to its own sector). That way tunefs can always check the disk for a mounted node, even without a cluster stack up. If tunefs sees anyone writing sequences to the disk, it can safely fail the operation. Tunefs also would have to be writing an mmp sequence once it has determined that the disk is not mounted. It could also write some flag alongisde the sequence that says 'tunefs is working on this disk'. If a cluster mount comes up and sees a live sequence with that flag, it will know to fail the mount request as the disk is being modified. Local mounts can also use this to ensure that they are the only mounted node.

Above tunefs check & write seq steps are mmp feature work flow.

Your mentioned tunefs work flow matches my patch[4/4] idea, and this patch does all the works in kernel ocfs2_find_slot().

Because sequences should be saved in SB, update it should grab ->osb_lock, which influence the performance.

And for saving seqs, for saving space, we won't alloc different disk block for different node. If multi-nodes share the same disk block (eg, keep using slot_map for saving seqs), the updating job will make IO performance issue.

For fixing performance issue, in my view, we should disable mmp sequences updating when mounting mode is clustered. So I make a rule:
** If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type. **

above rule another meaning:
new coming node mounting type should same with exist mounting type.

and there is a basic knowledge:
current ocfs2 code under cluster stack already have the ability to prevent multiple mount when mounting type is clustered.

(from patch 4/4) there are mount labels:
+#define OCFS2_MMP_SEQ_CLEAN 0xFF4D4D50U /* mmp_seq value for clean unmount */

+#define OCFS2_MMP_SEQ_FSCK  0xE24D4D50U /* mmp_seq value when being fscked */
+#define OCFS2_MMP_SEQ_MAX   0xE24D4D4FU /* maximum valid mmp_seq value */
+#define OCFS2_MMP_SEQ_INIT  0x0         /* mmp_seq init value */

+#define OCFS2_VALID_CLUSTER 0xE24D4D55U /* value for clustered mount
+ under MMP disabled */
+#define OCFS2_VALID_NOCLUSTER 0xE24D4D5AU /* value for noclustered mount
+ under MMP disabled */

whenever mount successfully, there should be three types living labels in slotmap area:
- OCFS2_MMP_SEQ_CLEAN, OCFS2_VALID_NOCLUSTER - for local/non-clustered mount
- OCFS2_VALID_CLUSTER - for clustered mount

new coming node will check if any slot contains living labels.

whenever unmount successfully, there should be two types left labels in slotmap area:
- OCFS2_MMP_SEQ_CLEAN or 0 (zero)

when a node does unmount, according to the mount type, it will clean (zeroed) or write OCFS2_MMP_SEQ_CLEAN in the slot.

Quoted 8 lines

As it turns out, we already do pretty much all of the sequence writing already for the o2cb cluster stack - check out cluseter/heartbeat.c. If memory serves, tunefs.ocfs2 has code to write to this heartbeat area as well. For o2cb, we use the disk heartbeat to detect node liveness, and to kill our local node if we see disk timeouts. For pcmk, we shouldn't take any of these actions as it is none of our responsibility. Under pcmk, the heartbeating would be purely for mount protection checks.

From my thinking, under cluster stack, there is enough protecting logic.

For local/non-clustered mount, mmp will give the ability for detecting node liveness. So I only enable kmmpd-<dev> kthread for local/non-clustered mount.

Quoted 8 lines

The downside to this is that all nodes would be heartbeating to the disk on a regular interval, not just one. To be fair, this is exactly how o2cb works and with the correct timeout choices, we were able to avoid a measurable performance impact, though in any case this might have to be a small price the user pays for cluster aware mount protection.

I already considered this performance issue in patch [4/4].

Quoted 4 lines

Let me know what you think.

Thanks,
--Mark

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#15 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

In reply to Mark Fasheh via Ocfs2-devel

Hi Mark,

Quoted 31 lines · Mark Fasheh

Hi Heming,

Quoted 20 lines · On Sun, Jul 31, 2022 at 6:02 PM [email protected] <[email protected]>…

Hello Mark,

Quoted 5 lines · Mark Fasheh

Hi Heming,

Quoted 1 line · On Fri, Jul 29, 2022 at 6:15 PM Heming Zhao via Ocfs2-devel…
Quoted 7 lines
Quoted 4 lines

the key different between local mount and non-clustered mount:
local mount feature (tunefs.ocfs2 --fs-features=[no]local) can't do convert job without ha stack. non-clustered mount feature can run totally without ha stack.

Can you please elaborate on this? Local mounts can run without a cluster stack so I don't see the difference there. We have

I am using pacemaker cluster stack. In my env, the trouble of the converting between local and clustered mounts are only happening on cluster stack.

Quoted 6 lines

the non-clustered mount feature (Gang He commit: 912f655d78c5) gave ocfs2 the ability to mount volume at any env (with/without cluster stack).
The 912f655d78c5 derived from SUSE customer complain: User wanted to fsck the backup ocfs2 volume in non-clustered env. They wanted to access the volume quickly and didn't want to take time/resource to set up HA stack. (by the way, pcmk stack at least needs two nodes to set up a cluster.)

Quoted 8 lines

Ok. I had some time to look over the ext4 mmp patches. I feel like there are two questions here:

1) Is MMP a useful feature for Ocfs2

My answer is yes absolutely, the user should have the option to enable this on local mounts.

me too.

Quoted 14 lines

2) Should we allow the user to bypass our cluster checks?

On this question I'm still a 'no'. I simply haven't seen enough evidence to warrant such a drastic change in policy. Allowing it via mount option too just feels extremely error-prone. I think we need to explore alternative avenues to help
ing the user out here. As you noted in your followup, a single node config is entirely possible in pacemaker (I've run that config myself). Why not provide an easy way for the user to drop down to that sort of a config? I know that's kind
of pushing responsibility for this to the cluster stack, but that's where it belongs in the first place.

the reason for creating commit 912f655d78c5d4a is user didn't want to do any

set up job for cluster stack. So any HA solution (eg: automatically config

single node, auto install required software, create VM with already set up HA

stack, etc) is not remove the pain point. Base on bypass cluster setup, there only

left non-clustered mount. And in end user viewpoint, non-clustered mount is

also the most easy way to mount ocfs2 volume.

Quoted 4 lines

Another option might be an 'observer mode' mount, where the node participates in the cluster (and the file system locking) but purely in a read-only fashion.

In my feeling, the 'observer mode' is just read-only mount, we don't need to create it.
For 912f655d78c5d4a, user take a snapshot for ocfs2 volume, and do health-check

on a different env (not production env). there may belong to two different network.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

tunefs.ocfs2 look for and join the cluster so as to avoid corrupting users data - that's a feature, not a bug. So what I'm seeing here is just opening us to potential corruptions. Is there a specific use case here that you're trying to account for? Are you fixing a particular bug?

Quoted 6 lines
Quoted 2 lines

Tunefs.ocfs2 still needs HA/dlm stack to protect joining action. commit 912f655d78c5 works on non-clustered env, which needs other tech (eg. MMP) to protect corrupting.

FWIW, I'm glad that we pulled commit 912f655d78c5 and I do not think we should go back to that paradigm.

If you or other maintainers prefer to pull out non-clustered mount, I respect

the decision.

Quoted 10 lines
Quoted 5 lines

From my viewpoint, the non-clustered mount code is based on local mount code,
which gives more flexible than local mount. non-clustered mount uses unify mount style align with clustered mount. I think users will like more to use non-clustered mount than using tunefs.ocfs2 to change mount type.

Can we get rid of the mount option, and make mmp something that users can turn on for Ocfs2 local mounts? I don't recall if we make a slot map for local mounts or not but it wouldn't be difficult to add that.

From tech side, it's possible to enable mmp in local mount.

Without 912f655d78c5d4a, local-mount uses slot 0 (the first available slot).

Quoted 6 lines

Btw, thank you very much for all of these patches, and also thank you for the very detailed test cases in your initial email. I'll try to give the actual code a review as well.

Thanks,
--Mark

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#16 lore raw

Re: [Ocfs2-devel] [PATCH 3/4] re-enable "ocfs2: mount shared volume without ha stack"

In reply to Mark Fasheh via Ocfs2-devel

Hello Mark and Joseph,

(please ignore my previous reply mail)
I may met thunderbird bug, I clearly remembered I added you (Joseph) in CC for this mail & previous mail. But I only see Mark and ocfs2-devel in the receiver list. and the mail format also mess, thunderbird replace '\n' to '\r\n'.

For fixing this problem, I change neomutt to resend these two mails

Quoted 31 lines · Mark Fasheh

Hi Heming,

Quoted 20 lines · On Sun, Jul 31, 2022 at 6:02 PM [email protected] <[email protected]>…

Hello Mark,

Quoted 5 lines · Mark Fasheh

Hi Heming,

Quoted 1 line · On Fri, Jul 29, 2022 at 6:15 PM Heming Zhao via Ocfs2-devel…
Quoted 7 lines
Quoted 4 lines

the key different between local mount and non-clustered mount:
local mount feature (tunefs.ocfs2 --fs-features=[no]local) can't do convert job without ha stack. non-clustered mount feature can run totally without ha stack.

Can you please elaborate on this? Local mounts can run without a cluster stack so I don't see the difference there. We have

I am using pacemaker cluster stack. In my env, the trouble of the converting between local and clustered mounts are only happening on cluster stack.

Quoted 6 lines

the non-clustered mount feature (Gang He commit: 912f655d78c5) gave ocfs2 the ability to mount volume at any env (with/without cluster stack).
The 912f655d78c5 derived from SUSE customer complain: User wanted to fsck the backup ocfs2 volume in non-clustered env. They wanted to access the volume quickly and didn't want to take time/resource to set up HA stack. (by the way, pcmk stack at least needs two nodes to set up a cluster.)

Quoted 8 lines

Ok. I had some time to look over the ext4 mmp patches. I feel like there are two questions here:

1) Is MMP a useful feature for Ocfs2

My answer is yes absolutely, the user should have the option to enable this on local mounts.

me too.

Quoted 14 lines

2) Should we allow the user to bypass our cluster checks?

On this question I'm still a 'no'. I simply haven't seen enough evidence to warrant such a drastic change in policy. Allowing it via mount option too just feels extremely error-prone. I think we need to explore alternative avenues to help
ing the user out here. As you noted in your followup, a single node config is entirely possible in pacemaker (I've run that config myself). Why not provide an easy way for the user to drop down to that sort of a config? I know that's kind
of pushing responsibility for this to the cluster stack, but that's where it belongs in the first place.

the reason for creating commit 912f655d78c5d4a is user didn't want to do any set up job for cluster stack. So any HA solution (eg: automatically config single node, auto install required software, create VM with already set up HA stack, etc) is not remove the pain point. Base on bypass cluster setup, there only left non-clustered mount. And in end user viewpoint, non-clustered mount is also the most easy way to mount ocfs2 volume.

Quoted 4 lines

Another option might be an 'observer mode' mount, where the node participates in the cluster (and the file system locking) but purely in a read-only fashion.

In my feeling, the 'observer mode' is just read-only mount, we don't need to create it.
For 912f655d78c5d4a, user take a snapshot for ocfs2 volume, and do health-check

on a different env (not production env). there may belong to two different network.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

tunefs.ocfs2 look for and join the cluster so as to avoid corrupting users data - that's a feature, not a bug. So what I'm seeing here is just opening us to potential corruptions. Is there a specific use case here that you're trying to account for? Are you fixing a particular bug?

Quoted 5 lines
Quoted 2 lines

Tunefs.ocfs2 still needs HA/dlm stack to protect joining action. commit 912f655d78c5 works on non-clustered env, which needs other tech (eg. MMP) to protect corrupting.

FWIW, I'm glad that we pulled commit 912f655d78c5 and I do not think we should go back to that paradigm.

If you or other maintainers prefer to pull out non-clustered mount, I respect the decision.

Quoted 11 lines
Quoted 6 lines

From my viewpoint, the non-clustered mount code is based on local mount code,
which gives more flexible than local mount. non-clustered mount uses unify mount style align with clustered mount. I think users will like more to use non-clustered mount than using tunefs.ocfs2 to change mount type.

Can we get rid of the mount option, and make mmp something that users can turn on for Ocfs2 local mounts? I don't recall if we make a slot map for local mounts or not but it wouldn't be difficult to add that.

From tech side, it's possible to enable mmp in local mount.

Without 912f655d78c5d4a, local-mount uses slot 0 (the first available slot).

Quoted 6 lines

Btw, thank you very much for all of these patches, and also thank you for the very detailed test cases in your initial email. I'll try to give the actual code a review as well.

Thanks,
--Mark

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#17 lore raw

PATCH 4/4 ocfs2: introduce ext4 MMP feature

In reply to Heming Zhao via Ocfs2-devel

MMP (multiple mount protection) gives filesystem ability to prevent from being mounted multiple times.

For avoiding data corruption when non-clustered and/or clustered mount are happening at same time, this commit introduced MMP feature. MMP idea is from ext4 MMP (fs/ext4/mmp.c) code. For ocfs2 is a clustered fs and also for compatible with existing slotmap feature, I did some optimization and modification when porting from ext4 to ocfs2.

For optimization:
mmp has a kthread kmmpd-<dev>, which is only created in non-clustered mode.

We set a rule:
If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type.

At last, this commit also fix commit c80af0c250c8 ("Revert "ocfs2:
mount shared volume without ha stack") mentioned issue.

---
 fs/ocfs2/ocfs2.h    |   2 +
 fs/ocfs2/ocfs2_fs.h |  13 +-
 fs/ocfs2/slot_map.c | 459 ++++++++++++++++++++++++++++++++++++++++++--
 fs/ocfs2/slot_map.h |   3 +
 fs/ocfs2/super.c    |  23 ++-
 5 files changed, 479 insertions(+), 21 deletions(-)
fs/ocfs2/ocfs2.h +2 -0
index 337527571461..37a7c5855d07 100644
@@ -337,6 +337,8 @@ struct ocfs2_super
 	unsigned int node_num;
 	int slot_num;
 	int preferred_slot;
+	u16 mmp_update_interval;
+	struct task_struct *mmp_task;
 	int s_sectsize_bits;
 	int s_clustersize;
 	int s_clustersize_bits;
fs/ocfs2/ocfs2_fs.h +9 -4
index 638d875eccc7..015672f75563 100644
@@ -87,7 +87,8 @@
 					 | OCFS2_FEATURE_INCOMPAT_REFCOUNT_TREE \
 					 | OCFS2_FEATURE_INCOMPAT_DISCONTIG_BG	\
 					 | OCFS2_FEATURE_INCOMPAT_CLUSTERINFO \
-					 | OCFS2_FEATURE_INCOMPAT_APPEND_DIO)
+					 | OCFS2_FEATURE_INCOMPAT_APPEND_DIO \
+					 | OCFS2_FEATURE_INCOMPAT_MMP)
 #define OCFS2_FEATURE_RO_COMPAT_SUPP	(OCFS2_FEATURE_RO_COMPAT_UNWRITTEN \
 					 | OCFS2_FEATURE_RO_COMPAT_USRQUOTA \
 					 | OCFS2_FEATURE_RO_COMPAT_GRPQUOTA)
@@ -167,6 +168,11 @@
  */
 #define OCFS2_FEATURE_INCOMPAT_APPEND_DIO	0x8000
 
+/*
+ * Multiple mount protection
+ */
+#define OCFS2_FEATURE_INCOMPAT_MMP	0x10000
+
 /*
  * backup superblock flag is used to indicate that this volume
  * has backup superblocks.
@@ -535,8 +541,7 @@ struct ocfs2_slot_map {
 };
 
 struct ocfs2_extended_slot {
-/*00*/	__u8	es_valid;
-	__u8	es_reserved1[3];
+/*00*/	__le32	es_valid;
 	__le32	es_node_num;
 /*08*/
 };
@@ -611,7 +616,7 @@ struct ocfs2_super_block {
 						     INCOMPAT flag set. */
 /*B8*/	__le16 s_xattr_inline_size;	/* extended attribute inline size
 					   for this fs*/
-	__le16 s_reserved0;
+	__le16 s_mmp_update_interval; /* # seconds to wait in MMP checking */
 	__le32 s_dx_seed[3];		/* seed[0-2] for dx dir hash.
 					 * s_uuid_hash serves as seed[3]. */
 /*C0*/  __le64 s_reserved2[15];		/* Fill out superblock */
fs/ocfs2/slot_map.c +444 -15

540 lines not on this page · raw message

fs/ocfs2/slot_map.h +3 -0
index a43644570b53..d4d147b0c190 100644
@@ -25,4 +25,7 @@ int ocfs2_slot_to_node_num_locked(struct ocfs2_super *osb, int slot_num,
 
 int ocfs2_clear_slot(struct ocfs2_super *osb, int slot_num);
 
+int ocfs2_multi_mount_protect(struct ocfs2_super *osb, int noclustered);
+void ocfs2_stop_mmpd(struct ocfs2_super *osb);
+
 #endif
fs/ocfs2/super.c +21 -2
index f7298816d8d9..b0e76b06efc3 100644
@@ -609,6 +609,7 @@ static int ocfs2_remount(struct super_block *sb, int *flags, char *data)
 	struct mount_options parsed_options;
 	struct ocfs2_super *osb = OCFS2_SB(sb);
 	u32 tmp;
+	int noclustered;
 
 	sync_filesystem(sb);
 
@@ -619,7 +620,8 @@ static int ocfs2_remount(struct super_block *sb, int *flags, char *data)
 	}
 
 	tmp = OCFS2_MOUNT_NOCLUSTER;
-	if ((osb->s_mount_opt & tmp) != (parsed_options.mount_opt & tmp)) {
+	noclustered = osb->s_mount_opt & tmp;
+	if (noclustered != (parsed_options.mount_opt & tmp)) {
 		ret = -EINVAL;
 		mlog(ML_ERROR, "Cannot change nocluster option on remount\n");
 		goto out;
@@ -686,10 +688,20 @@ static int ocfs2_remount(struct super_block *sb, int *flags, char *data)
 			}
 			sb->s_flags &= ~SB_RDONLY;
 			osb->osb_flags &= ~OCFS2_OSB_SOFT_RO;
+			if (OCFS2_HAS_INCOMPAT_FEATURE(sb, OCFS2_FEATURE_INCOMPAT_MMP)) {
+				spin_unlock(&osb->osb_lock);
+				if (ocfs2_multi_mount_protect(osb, noclustered)) {
+					mlog(ML_ERROR, "started MMP failed.\n");
+					ocfs2_stop_mmpd(osb);
+					ret = -EROFS;
+					goto unlocked_osb;
+				}
+			}
 		}
 		trace_ocfs2_remount(sb->s_flags, osb->osb_flags, *flags);
 unlock_osb:
 		spin_unlock(&osb->osb_lock);
+unlocked_osb:
 		/* Enable quota accounting after remounting RW */
 		if (!ret && !(*flags & SB_RDONLY)) {
 			if (sb_any_quota_suspended(sb))
@@ -722,6 +734,8 @@ static int ocfs2_remount(struct super_block *sb, int *flags, char *data)
 		sb->s_flags = (sb->s_flags & ~SB_POSIXACL) |
 			((osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) ?
 							SB_POSIXACL : 0);
+		if (sb_rdonly(osb->sb))
+			ocfs2_stop_mmpd(osb);
 	}
 out:
 	return ret;
@@ -1833,7 +1847,7 @@ static int ocfs2_mount_volume(struct super_block *sb)
 	status = ocfs2_init_local_system_inodes(osb);
 	if (status < 0) {
 		mlog_errno(status);
-		goto out_super_lock;
+		goto out_find_slot;
 	}
 
 	status = ocfs2_check_volume(osb);
@@ -1858,6 +1872,8 @@ static int ocfs2_mount_volume(struct super_block *sb)
 	/* before journal shutdown, we should release slot_info */
 	ocfs2_free_slot_info(osb);
 	ocfs2_journal_shutdown(osb);
+out_find_slot:
+	ocfs2_stop_mmpd(osb);
 out_super_lock:
 	ocfs2_super_unlock(osb, 1);
 out_dlm:
@@ -1878,6 +1894,8 @@ static void ocfs2_dismount_volume(struct super_block *sb, int mnt_err)
 	osb = OCFS2_SB(sb);
 	BUG_ON(!osb);
 
+	ocfs2_stop_mmpd(osb);
+
 	/* Remove file check sysfs related directores/files,
 	 * and wait for the pending file check operations */
 	ocfs2_filecheck_remove_sysfs(osb);
@@ -2086,6 +2104,7 @@ static int ocfs2_initialize_super(struct super_block *sb,
 	snprintf(osb->dev_str, sizeof(osb->dev_str), "%u,%u",
 		 MAJOR(osb->sb->s_dev), MINOR(osb->sb->s_dev));
 
+	osb->mmp_update_interval = le16_to_cpu(di->id2.i_super.s_mmp_update_interval);
 	osb->max_slots = le16_to_cpu(di->id2.i_super.s_max_slots);
 	if (osb->max_slots > OCFS2_MAX_SLOTS || osb->max_slots == 0) {
 		mlog(ML_ERROR, "Invalid number of node slots (%u)\n",
Signature · 7 lines
2.37.1


_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
#18 lore raw

Re: [Ocfs2-devel] [PATCH 4/4] ocfs2: introduce ext4 MMP feature

Quoted 49 lines · Heming Zhao

MMP (multiple mount protection) gives filesystem ability to prevent from being mounted multiple times.

For avoiding data corruption when non-clustered and/or clustered mount are happening at same time, this commit introduced MMP feature. MMP idea is from ext4 MMP (fs/ext4/mmp.c) code. For ocfs2 is a clustered fs and also for compatible with existing slotmap feature, I did some optimization and modification when porting from ext4 to ocfs2.

For optimization:
mmp has a kthread kmmpd-<dev>, which is only created in non-clustered mode.

We set a rule:
If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type.

At last, this commit also fix commit c80af0c250c8 ("Revert "ocfs2:
mount shared volume without ha stack") mentioned issue.

---
  fs/ocfs2/ocfs2.h    |   2 +
  fs/ocfs2/ocfs2_fs.h |  13 +-
  fs/ocfs2/slot_map.c | 459 ++++++++++++++++++++++++++++++++++++++++++--
  fs/ocfs2/slot_map.h |   3 +
  fs/ocfs2/super.c    |  23 ++-
  5 files changed, 479 insertions(+), 21 deletions(-)

... ...
/*C0*/ __le64 s_reserved2[15]; /* Fill out superblock */

fs/ocfs2/slot_map.c +14 -0
index 0b0ae3ebb0cf..86a21140ead6 100644
... ...
+void ocfs2_stop_mmpd(struct ocfs2_super *osb)
+{
+	if (osb->mmp_task) {
+		kthread_stop(osb->mmp_task);
+		osb->mmp_task = NULL;
+	}
+}
+
+/*
+ * Protect the filesystem from being mounted more than once.
+ *
+ * This function was inspired by ext4 MMP feature. Because HA stack
+ * helps ocfs2 to manage nodes join/leave, so we only focus on MMP
+ * under nocluster mode.

fs/ocfs2/slot_map.c void ocfs2_free_slot_info(struct ocfs2_super *osb) · line 682

Quoted 10 lines
fs/ocfs2/slot_map.c +10 -0
+ * Another info is ocfs2 only uses slot 0 on nocuster mode.
+ *
+ * es_valid:
+ *  0: not available
+ *  1: valid, cluster mode
+ *  2: valid, nocluster mod> + *
+ * parameters:
+ *  osb: the struct ocfs2_super
+ *  noclustered: under noclustered mount
+ *  slot: prefer slot number

Sorry, above es_valid & parameters comments are outdated. I will remove them on next version.

Quoted 2 lines

+ */
+int ocfs2_multi_mount_protect(struct ocfs2_super *osb, int noclustered)

I forgot to run checkpatch.pl for all 4 patches. And checkpatch.pl reported many format warning. I prefer these patches as draft version, let's discuss the mmp+noncluster idea is workable/acceptable. If no one reject these code, I will file v1.

(For test cases, I put them in cover letter.)

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#19 lore raw

Re: [Ocfs2-devel] [PATCH 4/4] ocfs2: introduce ext4 MMP feature

In reply to Heming Zhao via Ocfs2-devel
Quoted 13 lines · Heming Zhao

MMP (multiple mount protection) gives filesystem ability to prevent from being mounted multiple times.

For avoiding data corruption when non-clustered and/or clustered mount are happening at same time, this commit introduced MMP feature. MMP idea is from ext4 MMP (fs/ext4/mmp.c) code. For ocfs2 is a clustered fs and also for compatible with existing slotmap feature, I did some optimization and modification when porting from ext4 to ocfs2.

For optimization:
mmp has a kthread kmmpd-<dev>, which is only created in non-clustered mode.

Quoted 6 lines

We set a rule:
If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type.

At last, this commit also fix commit c80af0c250c8 ("Revert "ocfs2:
mount shared volume without ha stack") mentioned issue.

I suggest we re-split this series (especially patch 3 and 4), but not revive a buggy commit first and then another commit fixing it BTW.

Thanks,
Joseph

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#20 lore raw

Re: [Ocfs2-devel] [PATCH 4/4] ocfs2: introduce ext4 MMP feature

Quoted 20 lines · Joseph Qi
Quoted 17 lines · Heming Zhao

MMP (multiple mount protection) gives filesystem ability to prevent from being mounted multiple times.

For avoiding data corruption when non-clustered and/or clustered mount are happening at same time, this commit introduced MMP feature. MMP idea is from ext4 MMP (fs/ext4/mmp.c) code. For ocfs2 is a clustered fs and also for compatible with existing slotmap feature, I did some optimization and modification when porting from ext4 to ocfs2.

For optimization:
mmp has a kthread kmmpd-<dev>, which is only created in non-clustered mode.

We set a rule:
If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type.

Quoted 6 lines
Quoted 2 lines

At last, this commit also fix commit c80af0c250c8 ("Revert "ocfs2:
mount shared volume without ha stack") mentioned issue.

I suggest we re-split this series (especially patch 3 and 4), but not revive a buggy commit first and then another commit fixing it BTW.

No problem. I remembered this fixing style, and won't make this kind of mistake again.

Do we need to wait for Mark's further comment?
It looks Mark didn't like non-clustered mount at his first reply, but he showed another idea for non-clustered mount on the second reply. And I am sure what's his meaning/attitude for non-clustered mount.

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#21 lore raw

Re: [Ocfs2-devel] [PATCH 4/4] ocfs2: introduce ext4 MMP feature

Very weird problem. I added Junxiao & Mark in 'CC' field, but there is only kept 'To' field. I tested to use CC to send mail for other mail address, it worked well.

Quoted 36 lines · On Mon, Aug 08, 2022 at 05:07:14PM +0800, Heming Zhao via Ocfs2-devel wrote:
Quoted 20 lines · Joseph Qi
Quoted 17 lines · Heming Zhao

MMP (multiple mount protection) gives filesystem ability to prevent from being mounted multiple times.

For avoiding data corruption when non-clustered and/or clustered mount are happening at same time, this commit introduced MMP feature. MMP idea is from ext4 MMP (fs/ext4/mmp.c) code. For ocfs2 is a clustered fs and also for compatible with existing slotmap feature, I did some optimization and modification when porting from ext4 to ocfs2.

For optimization:
mmp has a kthread kmmpd-<dev>, which is only created in non-clustered mode.

We set a rule:
If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type.

Quoted 6 lines
Quoted 2 lines

At last, this commit also fix commit c80af0c250c8 ("Revert "ocfs2:
mount shared volume without ha stack") mentioned issue.

I suggest we re-split this series (especially patch 3 and 4), but not revive a buggy commit first and then another commit fixing it BTW.

No problem. I remembered this fixing style, and won't make this kind of mistake again.

Do we need to wait for Mark's further comment?
It looks Mark didn't like non-clustered mount at his first reply, but he showed another idea for non-clustered mount on the second reply. And I am sure what's his meaning/attitude for non-clustered mount.

Quoted 7 lines

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#22 lore raw

Re: [Ocfs2-devel] [PATCH 4/4] ocfs2: introduce ext4 MMP feature

In reply to Heming Zhao via Ocfs2-devel
Quoted 28 lines · Heming Zhao
Quoted 20 lines · Joseph Qi
Quoted 17 lines · Heming Zhao

MMP (multiple mount protection) gives filesystem ability to prevent from being mounted multiple times.

For avoiding data corruption when non-clustered and/or clustered mount are happening at same time, this commit introduced MMP feature. MMP idea is from ext4 MMP (fs/ext4/mmp.c) code. For ocfs2 is a clustered fs and also for compatible with existing slotmap feature, I did some optimization and modification when porting from ext4 to ocfs2.

For optimization:
mmp has a kthread kmmpd-<dev>, which is only created in non-clustered mode.

We set a rule:
If last mount didn't do unmount, (eg: crash), the next mount MUST be same mount type.

Quoted 6 lines
Quoted 2 lines

At last, this commit also fix commit c80af0c250c8 ("Revert "ocfs2:
mount shared volume without ha stack") mentioned issue.

I suggest we re-split this series (especially patch 3 and 4), but not revive a buggy commit first and then another commit fixing it BTW.

Quoted 8 lines

No problem. I remembered this fixing style, and won't make this kind of mistake again.

Do we need to wait for Mark's further comment?
It looks Mark didn't like non-clustered mount at his first reply, but he showed another idea for non-clustered mount on the second reply. And I am sure what's his meaning/attitude for non-clustered mount.

Yes, let's take Mark's input first for the next version.

Thanks,
Joseph

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel