ocfs2: fix a deadlock case

PATCH In review Joseph Qi 2 participants 6 messages Jan 21, 2022 – Jan 27, 2022

Subsystems (2)

OCFS2 Journalling layer for block devices (JBD2)

lore

Share & tools

lore

Joseph Qi

author maintainer

2022-01-21 07:12 UTC #0 lore raw

cover ocfs2: fix a deadlock case

This is trying to fix a deadlock case in ocfs2.
We firstly export jbd2 symbols jbd2_journal_[grab|put]_journal_head as preparation and later use them in ocfs2 insread of jbd_[lock|unlock]_bh_journal_head to fix the deadlock.

Joseph Qi (2):
jbd2: export jbd2_journal_[grab|put]_journal_head
ocfs2: fix a deadlock when commit trans

 fs/jbd2/journal.c   |  2 ++
 fs/ocfs2/suballoc.c | 25 +++++++++++--------------
 2 files changed, 13 insertions(+), 14 deletions(-)

Signature · 1 line

2.19.1.6.gb485710b

Joseph Qi

author maintainer

2022-01-21 07:12 UTC #1 lore raw

PATCH 1/2 jbd2: export jbd2_journal_[grab|put]_journal_head

This exports symbols jbd2_journal_[grab|put]_journal_head, which will be used outside modules, e.g. ocfs2.

Signed-off-by Joseph Qi <[email protected]>

---
 fs/jbd2/journal.c | 2 ++
 1 file changed, 2 insertions(+)

fs/jbd2/journal.c +2 -0

index 0b86a4365b66..e9f0c72f6664 100644
@@ -2972,6 +2972,7 @@ struct journal_head *jbd2_journal_grab_journal_head(struct buffer_head *bh)
 	jbd_unlock_bh_journal_head(bh);
 	return jh;
 }
+EXPORT_SYMBOL(jbd2_journal_grab_journal_head);
 
 static void __journal_remove_journal_head(struct buffer_head *bh)
 {

@@ -3024,6 +3025,7 @@ void jbd2_journal_put_journal_head(struct journal_head *jh)
 		jbd_unlock_bh_journal_head(bh);
 	}
 }
+EXPORT_SYMBOL(jbd2_journal_put_journal_head);
 
 /*
  * Initialize jbd inode head

Signature · 7 lines

2.19.1.6.gb485710b


_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Joseph Qi

author maintainer

2022-01-21 07:12 UTC #2 lore raw

PATCH 2/2 ocfs2: fix a deadlock when commit trans

In reply to Joseph Qi

commit 6f1b228529ae introduces a regression which can deadlock as follows:

Task1:                              Task2:
jbd2_journal_commit_transaction     ocfs2_test_bg_bit_allocatable
spin_lock(&jh->b_state_lock)        jbd_lock_bh_journal_head
__jbd2_journal_remove_checkpoint    spin_lock(&jh->b_state_lock)

jbd2_journal_put_journal_head
jbd_lock_bh_journal_head

Task1 and Task2 lock bh->b_state and jh->b_state_lock in different order, which finally result in a deadlock.

So use jbd2_journal_[grab|put]_journal_head instead in ocfs2_test_bg_bit_allocatable() to fix it.

Reported-by Gautham Ananthakrishna <[email protected]>

Fixes 6f1b228529ae ("ocfs2: fix race between searching chunks and release journal_head from buffer_head")

Cc <[email protected]>

Signed-off-by Joseph Qi <[email protected]>

---
 fs/ocfs2/suballoc.c | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)

fs/ocfs2/suballoc.c +11 ~~-14~~

index 481017e1dac5..166c8918c825 100644
@@ -1251,26 +1251,23 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
 {
 	struct ocfs2_group_desc *bg = (struct ocfs2_group_desc *) bg_bh->b_data;
 	struct journal_head *jh;
-	int ret = 1;
+	int ret;
 
 	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
 		return 0;
 
-	if (!buffer_jbd(bg_bh))
+	jh = jbd2_journal_grab_journal_head(bg_bh);
+	if (!jh)
 		return 1;
 
-	jbd_lock_bh_journal_head(bg_bh);
-	if (buffer_jbd(bg_bh)) {
-		jh = bh2jh(bg_bh);
-		spin_lock(&jh->b_state_lock);
-		bg = (struct ocfs2_group_desc *) jh->b_committed_data;
-		if (bg)
-			ret = !ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap);
-		else
-			ret = 1;
-		spin_unlock(&jh->b_state_lock);
-	}
-	jbd_unlock_bh_journal_head(bg_bh);
+	spin_lock(&jh->b_state_lock);
+	bg = (struct ocfs2_group_desc *) jh->b_committed_data;
+	if (bg)
+		ret = !ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap);
+	else
+		ret = 1;
+	spin_unlock(&jh->b_state_lock);
+	jbd2_journal_put_journal_head(jh);
 
 	return ret;
 }

Signature · 1 line

2.19.1.6.gb485710b

Gautham Ananthakrishna

2022-01-23 05:31 UTC #3 lore raw

RE: [PATCH 2/2] ocfs2: fix a deadlock when commit trans

Hi,
This deadlock was originally reported by [email protected] Could you please add Saeed as the reportedby.

Thanks,
Gautham.

-----Original Message-----
From: Joseph Qi <[email protected]>
Sent: Friday, January 21, 2022 12:42 PM
To: [email protected]; [email protected]; [email protected]
Cc: Gautham Ananthakrishna <[email protected]>; [email protected]; [email protected]
Subject: [PATCH 2/2] ocfs2: fix a deadlock when commit trans

commit 6f1b228529ae introduces a regression which can deadlock as follows:

Task1:                              Task2:
jbd2_journal_commit_transaction     ocfs2_test_bg_bit_allocatable
spin_lock(&jh->b_state_lock)        jbd_lock_bh_journal_head
__jbd2_journal_remove_checkpoint    spin_lock(&jh->b_state_lock)

jbd2_journal_put_journal_head
jbd_lock_bh_journal_head

Task1 and Task2 lock bh->b_state and jh->b_state_lock in different order, which finally result in a deadlock.

So use jbd2_journal_[grab|put]_journal_head instead in ocfs2_test_bg_bit_allocatable() to fix it.

Reported-by Gautham Ananthakrishna <[email protected]>

Fixes 6f1b228529ae ("ocfs2: fix race between searching chunks and release journal_head from buffer_head")

Cc <[email protected]>

Signed-off-by Joseph Qi <[email protected]>

---
 fs/ocfs2/suballoc.c | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)

fs/ocfs2/suballoc.c +11 ~~-15~~

@@ -1251,26 +1251,23 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,  {
 	struct ocfs2_group_desc *bg = (struct ocfs2_group_desc *) bg_bh->b_data;
 	struct journal_head *jh;
-	int ret = 1;
+	int ret;
 
 	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
 		return 0;
 
-	if (!buffer_jbd(bg_bh))
+	jh = jbd2_journal_grab_journal_head(bg_bh);
+	if (!jh)
 		return 1;
 
-	jbd_lock_bh_journal_head(bg_bh);
-	if (buffer_jbd(bg_bh)) {
-		jh = bh2jh(bg_bh);
-		spin_lock(&jh->b_state_lock);
-		bg = (struct ocfs2_group_desc *) jh->b_committed_data;
-		if (bg)
-			ret = !ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap);
-		else
-			ret = 1;
-		spin_unlock(&jh->b_state_lock);
-	}
-	jbd_unlock_bh_journal_head(bg_bh);
+	spin_lock(&jh->b_state_lock);
+	bg = (struct ocfs2_group_desc *) jh->b_committed_data;
+	if (bg)
+		ret = !ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap);
+	else
+		ret = 1;
+	spin_unlock(&jh->b_state_lock);
+	jbd2_journal_put_journal_head(jh);
 
 	return ret;
 }
--

2.19.1.6.gb485710b

Joseph Qi

author maintainer

2022-01-24 02:37 UTC #4 lore raw

Re: [PATCH 2/2] ocfs2: fix a deadlock when commit trans

Sure, will do it in v2.
So could this patch resolve your issue?

Thanks,
Joseph

Quoted 78 lines · Gautham Ananthakrishna

Hi,
This deadlock was originally reported by [email protected] Could you please add Saeed as the reportedby.
Thanks,
Gautham.
-----Original Message-----
From: Joseph Qi <[email protected]>
Sent: Friday, January 21, 2022 12:42 PM
To: [email protected]; [email protected]; [email protected]
Cc: Gautham Ananthakrishna <[email protected]>; [email protected]; [email protected]
Subject: [PATCH 2/2] ocfs2: fix a deadlock when commit trans
commit 6f1b228529ae introduces a regression which can deadlock as follows:
Task1:                              Task2:
jbd2_journal_commit_transaction     ocfs2_test_bg_bit_allocatable
spin_lock(&jh->b_state_lock)        jbd_lock_bh_journal_head
__jbd2_journal_remove_checkpoint    spin_lock(&jh->b_state_lock)
jbd2_journal_put_journal_head
jbd_lock_bh_journal_head
Task1 and Task2 lock bh->b_state and jh->b_state_lock in different order, which finally result in a deadlock.
So use jbd2_journal_[grab|put]_journal_head instead in ocfs2_test_bg_bit_allocatable() to fix it.
Reported-by Gautham Ananthakrishna <[email protected]>
Fixes 6f1b228529ae ("ocfs2: fix race between searching chunks and release journal_head from buffer_head")
Cc <[email protected]>
Signed-off-by Joseph Qi <[email protected]>
---
 fs/ocfs2/suballoc.c | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)
fs/ocfs2/suballoc.c +11 ~~-15~~
@@ -1251,26 +1251,23 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,  {
 	struct ocfs2_group_desc *bg = (struct ocfs2_group_desc *) bg_bh->b_data;
 	struct journal_head *jh;
-	int ret = 1;
+	int ret;
 
 	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
 		return 0;
 
-	if (!buffer_jbd(bg_bh))
+	jh = jbd2_journal_grab_journal_head(bg_bh);
+	if (!jh)
 		return 1;
 
-	jbd_lock_bh_journal_head(bg_bh);
-	if (buffer_jbd(bg_bh)) {
-		jh = bh2jh(bg_bh);
-		spin_lock(&jh->b_state_lock);
-		bg = (struct ocfs2_group_desc *) jh->b_committed_data;
-		if (bg)
-			ret = !ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap);
-		else
-			ret = 1;
-		spin_unlock(&jh->b_state_lock);
-	}
-	jbd_unlock_bh_journal_head(bg_bh);
+	spin_lock(&jh->b_state_lock);
+	bg = (struct ocfs2_group_desc *) jh->b_committed_data;
+	if (bg)
+		ret = !ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap);
+	else
+		ret = 1;
+	spin_unlock(&jh->b_state_lock);
+	jbd2_journal_put_journal_head(jh);
 
 	return ret;
 }
--
2.19.1.6.gb485710b

Gautham Ananthakrishna

2022-01-27 12:46 UTC #5 lore raw

RE: [PATCH 2/2] ocfs2: fix a deadlock when commit trans

Yes. The patch has resolved the issue.

Thanks,
Gautham.

-----Original Message-----
From: Joseph Qi <[email protected]>
Sent: Monday, January 24, 2022 8:08 AM
To: Gautham Ananthakrishna <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Saeed Mirzamohammadi <[email protected]>
Subject: Re: [PATCH 2/2] ocfs2: fix a deadlock when commit trans

Sure, will do it in v2.
So could this patch resolve your issue?

Thanks,
Joseph

Quoted 78 lines · Gautham Ananthakrishna

Hi,
This deadlock was originally reported by [email protected] Could you please add Saeed as the reportedby.
Thanks,
Gautham.
-----Original Message-----
From: Joseph Qi <[email protected]>
Sent: Friday, January 21, 2022 12:42 PM
To: [email protected]; [email protected]; [email protected]
Cc: Gautham Ananthakrishna <[email protected]>; [email protected]; [email protected]
Subject: [PATCH 2/2] ocfs2: fix a deadlock when commit trans
commit 6f1b228529ae introduces a regression which can deadlock as follows:
Task1:                              Task2:
jbd2_journal_commit_transaction     ocfs2_test_bg_bit_allocatable
spin_lock(&jh->b_state_lock)        jbd_lock_bh_journal_head
__jbd2_journal_remove_checkpoint    spin_lock(&jh->b_state_lock)
jbd2_journal_put_journal_head
jbd_lock_bh_journal_head
Task1 and Task2 lock bh->b_state and jh->b_state_lock in different order, which finally result in a deadlock.
So use jbd2_journal_[grab|put]_journal_head instead in ocfs2_test_bg_bit_allocatable() to fix it.
Reported-by Gautham Ananthakrishna <[email protected]>
Fixes 6f1b228529ae ("ocfs2: fix race between searching chunks and release journal_head from buffer_head")
Cc <[email protected]>
Signed-off-by Joseph Qi <[email protected]>
---
 fs/ocfs2/suballoc.c | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)
fs/ocfs2/suballoc.c +11 ~~-15~~
@@ -1251,26 +1251,23 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,  {
 	struct ocfs2_group_desc *bg = (struct ocfs2_group_desc *) bg_bh->b_data;
 	struct journal_head *jh;
-	int ret = 1;
+	int ret;
 
 	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
 		return 0;
 
-	if (!buffer_jbd(bg_bh))
+	jh = jbd2_journal_grab_journal_head(bg_bh);
+	if (!jh)
 		return 1;
 
-	jbd_lock_bh_journal_head(bg_bh);
-	if (buffer_jbd(bg_bh)) {
-		jh = bh2jh(bg_bh);
-		spin_lock(&jh->b_state_lock);
-		bg = (struct ocfs2_group_desc *) jh->b_committed_data;
-		if (bg)
-			ret = !ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap);
-		else
-			ret = 1;
-		spin_unlock(&jh->b_state_lock);
-	}
-	jbd_unlock_bh_journal_head(bg_bh);
+	spin_lock(&jh->b_state_lock);
+	bg = (struct ocfs2_group_desc *) jh->b_committed_data;
+	if (bg)
+		ret = !ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap);
+	else
+		ret = 1;
+	spin_unlock(&jh->b_state_lock);
+	jbd2_journal_put_journal_head(jh);
 
 	return ret;
 }
--
2.19.1.6.gb485710b