Skip to contentSkip to conversation

ocfs2: protect get_block with inode cluster lock

PATCH New Junxiao Bi 5 participants 7 messages Sep 10, 2019 – Sep 12, 2019 OCFS2

lore
Share & tools
lore

Junxiao Bi

author
#0 lore raw

PATCH 1/1 ocfs2: protect get_block with inode cluster lock

Inode cluster lock should be acquired to avoid extent tree changed by other cluster nodes.

---
 fs/ocfs2/aops.c | 6 ++++++
 1 file changed, 6 insertions(+)
fs/ocfs2/aops.c +6 -0
index a4c905d6b575..5ae7253d04b0 100644
@@ -127,9 +127,15 @@ static int ocfs2_lock_get_block(struct inode *inode, sector_t iblock,
 	int ret = 0;
 	struct ocfs2_inode_info *oi = OCFS2_I(inode);
 
+	ret = ocfs2_inode_lock(inode, NULL, 0);
+	if (ret) {
+		mlog_errno(ret);
+		return ret;
+	}
 	down_read(&oi->ip_alloc_sem);
 	ret = ocfs2_get_block(inode, iblock, bh_result, create);
 	up_read(&oi->ip_alloc_sem);
+	ocfs2_inode_unlock(inode, 0);
 
 	return ret;
 }
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/1 ocfs2: protect get_block with inode cluster lock

Hi Junxiao,

Quoted 3 lines · Junxiao Bi

Inode cluster lock should be acquired to avoid extent tree changed by other cluster nodes.

Could you please elaborate more on the real issue? IIUC, ocfs2_lock_get_block() will only be called under:
1) rw = READ;
2) rw = WRITE && overwrite.

Thanks,
Joseph

Quoted 26 lines
---
 fs/ocfs2/aops.c | 6 ++++++
 1 file changed, 6 insertions(+)
fs/ocfs2/aops.c +6 -0
index a4c905d6b575..5ae7253d04b0 100644
@@ -127,9 +127,15 @@ static int ocfs2_lock_get_block(struct inode *inode, sector_t iblock,
 	int ret = 0;
 	struct ocfs2_inode_info *oi = OCFS2_I(inode);
 
+	ret = ocfs2_inode_lock(inode, NULL, 0);
+	if (ret) {
+		mlog_errno(ret);
+		return ret;
+	}
 	down_read(&oi->ip_alloc_sem);
 	ret = ocfs2_get_block(inode, iblock, bh_result, create);
 	up_read(&oi->ip_alloc_sem);
+	ocfs2_inode_unlock(inode, 0);
 
 	return ret;
 }
#2 lore raw

PATCH 1/1 ocfs2: protect get_block with inode cluster lock

In reply to Junxiao Bi

Hi Junxiao,

As _RW_ cluster lock with EX level is held by local node, is that possible other node has a chance to touch the extent tree belonging to the same inode?

Thanks,

Changwei

Quoted 28 lines · Junxiao Bi

Inode cluster lock should be acquired to avoid extent tree changed by other cluster nodes.

---
  fs/ocfs2/aops.c | 6 ++++++
  1 file changed, 6 insertions(+)
fs/ocfs2/aops.c +6 -0
index a4c905d6b575..5ae7253d04b0 100644
@@ -127,9 +127,15 @@ static int ocfs2_lock_get_block(struct inode *inode, sector_t iblock,
  	int ret = 0;
  	struct ocfs2_inode_info *oi = OCFS2_I(inode);
 
+	ret = ocfs2_inode_lock(inode, NULL, 0);
+	if (ret) {
+		mlog_errno(ret);
+		return ret;
+	}
  	down_read(&oi->ip_alloc_sem);
  	ret = ocfs2_get_block(inode, iblock, bh_result, create);
  	up_read(&oi->ip_alloc_sem);
+	ocfs2_inode_unlock(inode, 0);
 
  	return ret;
  }
author
#3 lore raw

PATCH 1/1 ocfs2: protect get_block with inode cluster lock

Hi Joseph & Changwei,

This is found by just reviewing source code, according the lock semantic, for accessing extent tree, alloc_sem and ip cluster lock should be acquired. ocfs2_dio_wr_get_block() had these two lock acquired. rw_lock can protect the tree from the write, but i am not sure whether there were other flow changing extent tree without rw lock held.

Thanks,

Junxiao.

Quoted 41 lines · Changwei Ge

Hi Junxiao,

As _RW_ cluster lock with EX level is held by local node, is that possible other node has a chance to touch the extent tree belonging to the same inode?

Thanks,

Changwei

Quoted 27 lines · Junxiao Bi

Inode cluster lock should be acquired to avoid extent tree changed by other cluster nodes.

---
? fs/ocfs2/aops.c | 6 ++++++
? 1 file changed, 6 insertions(+)

fs/ocfs2/aops.c +5 -0
index a4c905d6b575..5ae7253d04b0 100644
@@ -127,9 +127,15 @@ static int ocfs2_lock_get_block(struct inode 
 *inode, sector_t iblock,
 ????? int ret = 0;
 ????? struct ocfs2_inode_info *oi = OCFS2_I(inode);
 ? +??? ret = ocfs2_inode_lock(inode, NULL, 0);
+??? if (ret) {
+??????? mlog_errno(ret);
+??????? return ret;
+??? }
 ????? down_read(&oi->ip_alloc_sem);
 ????? ret = ocfs2_get_block(inode, iblock, bh_result, create);
 ????? up_read(&oi->ip_alloc_sem);
+??? ocfs2_inode_unlock(inode, 0);
 ? ????? return ret;
 ? }
#4 lore raw

PATCH 1/1 ocfs2: protect get_block with inode cluster lock

Hi Guys,

I went through the ocfs2 code today.
I think there should be a ocfs2_inode_lock() in the ocfs2_lock_get_block() function.
Why?
ocfs2_rw_lock is used to protect read/write operation from the different nodes,
but it cannot synchronize the inode meta-data changes, that means one node cannot on-time see the meta-data changes(e.g. extent blocks) from the other nodes. The meta-data consistency should be guaranteed by ocfs2_inode_lock.
Anyway,
this ocfs2_inode_lock should be added here, but we should do a full testing after this patch, to make sure there is not any deadlock. Second, according to my comments, I think the patch reporter should be able to give an example for why we need to add this patch.

Thanks
Gang

Quoted 68 lines

-----Original Message-----
From: ocfs2-devel-bounces at oss.oracle.com [mailto:ocfs2-devel-bounces at oss.oracle.com] On Behalf Of Junxiao Bi
Sent: 2019?9?12? 1:40
To: jiangqi903 at gmail.com; Changwei Ge <[email protected]>; ocfs2-devel at oss.oracle.com
Subject: Re: [Ocfs2-devel] [PATCH] ocfs2: protect get_block with inode cluster lock

Hi Joseph & Changwei,

This is found by just reviewing source code, according the lock semantic, for accessing extent tree, alloc_sem and ip cluster lock should be acquired. ocfs2_dio_wr_get_block() had these two lock acquired. rw_lock can protect the tree from the write, but i am not sure whether there were other flow changing extent tree without rw lock held.

Thanks,

Junxiao.

Quoted 33 lines · Changwei Ge

Hi Junxiao,

As _RW_ cluster lock with EX level is held by local node, is that possible other node has a chance to touch the extent tree belonging to the same inode?

Thanks,

Changwei

Quoted 19 lines · Junxiao Bi

Inode cluster lock should be acquired to avoid extent tree changed by other cluster nodes.

---
? fs/ocfs2/aops.c | 6 ++++++
? 1 file changed, 6 insertions(+)

fs/ocfs2/aops.c index renamed from fs/ocfs2/aops.c +0 -0

a4c905d6b575..5ae7253d04b0 100644

fs/ocfs2/aops.c +2 -0
@@ -127,9 +127,15 @@ static int ocfs2_lock_get_block(struct inode
 *inode, sector_t iblock,
 ????? int ret = 0;
 ????? struct ocfs2_inode_info *oi = OCFS2_I(inode);
 ? +??? ret = ocfs2_inode_lock(inode, NULL, 0);
+??? if (ret) {
+??????? mlog_errno(ret);
Quoted 8 lines

fs/ocfs2/aops.c static int ocfs2_lock_get_block(struct inode · line 140

Quoted 8 lines
fs/ocfs2/aops.c +3 -0
+??????? return ret;
+??? }
 ????? down_read(&oi->ip_alloc_sem);
 ????? ret = ocfs2_get_block(inode, iblock, bh_result, create);
 ????? up_read(&oi->ip_alloc_sem);
+??? ocfs2_inode_unlock(inode, 0);
 ? ????? return ret;
 ? }

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#5 lore raw

PATCH 1/1 ocfs2: protect get_block with inode cluster lock

Hi Gang,

IIUC, you worried that local node might use the old inode meta like 'inode size", right?

If so, in ocfs2_direct_IO() which is the caller of ocfs2_lock_get_block() and ocfs2_dio_wr_get_block() is already using inode meta.

Should we also lock inode there?

I think the way how we guarantee what you mentioned is we acquire inode cluster lock with RW cluster lock held(in ocfs2_file_write_iter()), after which we a see a consistent inode.

I suppose if other node wants to change inode extent tree(truncating, appending, punching hole), it should
acquire first, by which during local node write operation, we can use a stable extent tree.

Thank,

Changwei

Quoted 86 lines · Gang He

Hi Guys,

I went through the ocfs2 code today.
I think there should be a ocfs2_inode_lock() in the ocfs2_lock_get_block() function.
Why?
ocfs2_rw_lock is used to protect read/write operation from the different nodes,
but it cannot synchronize the inode meta-data changes, that means one node cannot on-time see the meta-data changes(e.g. extent blocks) from the other nodes. The meta-data consistency should be guaranteed by ocfs2_inode_lock.
Anyway,
this ocfs2_inode_lock should be added here, but we should do a full testing after this patch, to make sure there is not any deadlock. Second, according to my comments, I think the patch reporter should be able to give an example for why we need to add this patch.

Thanks
Gang

Quoted 59 lines

-----Original Message-----
From: ocfs2-devel-bounces at oss.oracle.com [mailto:ocfs2-devel-bounces at oss.oracle.com] On Behalf Of Junxiao Bi
Sent: 2019?9?12? 1:40
To: jiangqi903 at gmail.com; Changwei Ge <[email protected]>; ocfs2-devel at oss.oracle.com
Subject: Re: [Ocfs2-devel] [PATCH] ocfs2: protect get_block with inode cluster lock

Hi Joseph & Changwei,

This is found by just reviewing source code, according the lock semantic, for accessing extent tree, alloc_sem and ip cluster lock should be acquired. ocfs2_dio_wr_get_block() had these two lock acquired. rw_lock can protect the tree from the write, but i am not sure whether there were other flow changing extent tree without rw lock held.

Thanks,

Junxiao.

Quoted 37 lines · Changwei Ge

Hi Junxiao,

As _RW_ cluster lock with EX level is held by local node, is that possible other node has a chance to touch the extent tree belonging to the same inode?

Thanks,

Changwei

Quoted 23 lines · Junxiao Bi

Inode cluster lock should be acquired to avoid extent tree changed by other cluster nodes.

---
 ? fs/ocfs2/aops.c | 6 ++++++

? 1 file changed, 6 insertions(+)

fs/ocfs2/aops.c index renamed from fs/ocfs2/aops.c +0 -0

a4c905d6b575..5ae7253d04b0 100644

fs/ocfs2/aops.c +4 -0
@@ -127,9 +127,15 @@ static int ocfs2_lock_get_block(struct inode
 *inode, sector_t iblock,
 ????? int ret = 0;
 ????? struct ocfs2_inode_info *oi = OCFS2_I(inode);
 ? +??? ret = ocfs2_inode_lock(inode, NULL, 0);
+??? if (ret) {
+??????? mlog_errno(ret);
+??????? return ret;
+??? }
 ????? down_read(&oi->ip_alloc_sem);
 ????? ret = ocfs2_get_block(inode, iblock, bh_result, create);
Quoted 8 lines
Quoted 4 lines

fs/ocfs2/aops.c static int ocfs2_lock_get_block(struct inode · line 140

Quoted 4 lines
fs/ocfs2/aops.c +1 -0
 ????? up_read(&oi->ip_alloc_sem);
+??? ocfs2_inode_unlock(inode, 0);
 ? ????? return ret;
 ? }

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

#6 lore raw

PATCH 1/1 ocfs2: protect get_block with inode cluster lock

Hi ChangWei,

Please see my comments inline.

Quoted 10 lines

-----Original Message-----
From: ocfs2-devel-bounces at oss.oracle.com [mailto:ocfs2-devel-bounces at oss.oracle.com] On Behalf Of Changwei Ge
Sent: 2019?9?12? 18:26
To: jiangqi903 at gmail.com; Changwei Ge <[email protected]>; Junxiao Bi <[email protected]>; ocfs2-devel at oss.oracle.com; Gang He <[email protected]>
Subject: Re: [Ocfs2-devel] [PATCH] ocfs2: protect get_block with inode cluster lock

Quoted 4 lines

Hi Gang,

IIUC, you worried that local node might use the old inode meta like 'inode size", right?

Inode block and extent blocks for a file.
My means, if there is not inode meta lock acquire, the inode meta-blocks changes from the other node cannot write-back to the disk immediately, since meta-data block write-back depends on lock downgrade hook.

Quoted 10 lines

If so, in ocfs2_direct_IO() which is the caller of ocfs2_lock_get_block() and ocfs2_dio_wr_get_block() is already using inode meta.

Should we also lock inode there?

I think the way how we guarantee what you mentioned is we acquire inode cluster lock with RW cluster lock held(in ocfs2_file_write_iter()), after which we a see a consistent inode.

My concerns are,
If the direct-io inode has been in the memory in the past (maybe this is second read for this node).
Then, before do the direct-io read, the caller cannot see the meta-data block changes, which were just made from the other nodes. In this case, if we can hit the problem?

Thanks
Gang

Quoted 109 lines

I suppose if other node wants to change inode extent tree(truncating, appending, punching hole), it should acquire first, by which during local node write operation, we can use a stable extent tree.

Thank,

Changwei

Quoted 4 lines · Gang He

Hi Guys,

I went through the ocfs2 code today.
I think there should be a ocfs2_inode_lock() in the ocfs2_lock_get_block()

function.

Quoted 3 lines

Why?
ocfs2_rw_lock is used to protect read/write operation from the different nodes, but it cannot synchronize the inode meta-data changes, that

means one node cannot on-time see the meta-data changes(e.g. extent blocks) from the other nodes.

Quoted 3 lines

The meta-data consistency should be guaranteed by ocfs2_inode_lock. Anyway,
this ocfs2_inode_lock should be added here, but we should do a full testing

after this patch, to make sure there is not any deadlock.

Quoted 1 line

Second, according to my comments, I think the patch reporter should be

able to give an example for why we need to add this patch.

Quoted 17 lines

Thanks
Gang

Quoted 13 lines

-----Original Message-----
From: ocfs2-devel-bounces at oss.oracle.com [mailto:ocfs2-devel-bounces at oss.oracle.com] On Behalf Of Junxiao Bi
Sent: 2019?9?12? 1:40
To: jiangqi903 at gmail.com; Changwei Ge <[email protected]>; ocfs2-devel at oss.oracle.com
Subject: Re: [Ocfs2-devel] [PATCH] ocfs2: protect get_block with inode cluster lock

Hi Joseph & Changwei,

This is found by just reviewing source code, according the lock semantic, for accessing extent tree, alloc_sem and ip cluster lock should be

acquired.

Quoted 54 lines
Quoted 46 lines

ocfs2_dio_wr_get_block() had these two lock acquired. rw_lock can protect the tree from the write, but i am not sure whether there were other flow changing extent tree without rw lock held.

Thanks,

Junxiao.

Quoted 37 lines · Changwei Ge

Hi Junxiao,

As _RW_ cluster lock with EX level is held by local node, is that possible other node has a chance to touch the extent tree belonging to the same inode?

Thanks,

Changwei

Quoted 23 lines · Junxiao Bi

Inode cluster lock should be acquired to avoid extent tree changed by other cluster nodes.

---
 ? fs/ocfs2/aops.c | 6 ++++++

? 1 file changed, 6 insertions(+)

fs/ocfs2/aops.c index renamed from fs/ocfs2/aops.c +0 -0

a4c905d6b575..5ae7253d04b0 100644

fs/ocfs2/aops.c +4 -0
@@ -127,9 +127,15 @@ static int ocfs2_lock_get_block(struct inode
 *inode, sector_t iblock,
 ????? int ret = 0;
 ????? struct ocfs2_inode_info *oi = OCFS2_I(inode);
 ? +??? ret = ocfs2_inode_lock(inode, NULL, 0);
+??? if (ret) {
+??????? mlog_errno(ret);
+??????? return ret;
+??? }
 ????? down_read(&oi->ip_alloc_sem);
 ????? ret = ocfs2_get_block(inode, iblock, bh_result, create);
Quoted 8 lines
Quoted 4 lines

fs/ocfs2/aops.c static int ocfs2_lock_get_block(struct inode · line 140

Quoted 4 lines
fs/ocfs2/aops.c +1 -0
 ????? up_read(&oi->ip_alloc_sem);
+??? ocfs2_inode_unlock(inode, 0);
 ? ????? return ret;
 ? }

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Quoted 4 lines

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel