Skip to contentSkip to conversation

f2fs: fix to avoid potential deadlock

PATCH In review Chao Yu 2 participants 14 messages Jan 27, 2022 – Mar 3, 2022 F2FS file system

lore
Share & tools
lore

Chao Yu

author maintainer
#0 lore raw

PATCH 1/1 f2fs: fix to avoid potential deadlock

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback] [T:checkpoint]
- wb_writeback
- blk_start_plug
bio contains NodeA was plugged in writeback threads
- do_writepages -- sync write inodeB, inc wb_sync_req[DATA]
- f2fs_write_data_pages
- f2fs_write_single_data_page -- write last dirty page
- f2fs_do_write_data_page
- set_page_writeback -- clear page dirty flag and PAGECACHE_TAG_DIRTY tag in radix tree
- f2fs_outplace_write_data
- f2fs_update_data_blkaddr
- f2fs_wait_on_page_writeback -- wait NodeA to writeback here
- inode_dec_dirty_pages
- writeback_sb_inodes
- writeback_single_inode
- do_writepages
- f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
- wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
- requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
- blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
 fs/f2fs/data.c | 6 +++++-
 fs/f2fs/node.c | 6 +++++-
 2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	/* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
 	if (wbc->sync_mode == WB_SYNC_ALL)
 		atomic_inc(&sbi->wb_sync_req[DATA]);
-	else if (atomic_read(&sbi->wb_sync_req[DATA]))
+	else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
 		goto skip_write;
+	}
 
 	if (__should_serialize_io(inode, wbc)) {
 		mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
 
 	if (wbc->sync_mode == WB_SYNC_ALL)
 		atomic_inc(&sbi->wb_sync_req[NODE]);
-	else if (atomic_read(&sbi->wb_sync_req[NODE]))
+	else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
 		goto skip_write;
+	}
 
 	trace_f2fs_writepages(mapping->host, wbc, NODE);
 
Signature · 1 line
2.32.0
maintainer
#1 lore raw
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback] [T:checkpoint]
- wb_writeback
- blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

Quoted 69 lines

- do_writepages -- sync write inodeB, inc wb_sync_req[DATA]
- f2fs_write_data_pages
- f2fs_write_single_data_page -- write last dirty page
- f2fs_do_write_data_page
- set_page_writeback -- clear page dirty flag and PAGECACHE_TAG_DIRTY tag in radix tree
- f2fs_outplace_write_data
- f2fs_update_data_blkaddr
- f2fs_wait_on_page_writeback -- wait NodeA to writeback here
- inode_dec_dirty_pages
- writeback_sb_inodes
- writeback_single_inode
- do_writepages
- f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
- wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
- requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
- blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
 fs/f2fs/data.c | 6 +++++-
 fs/f2fs/node.c | 6 +++++-
 2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	/* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
 	if (wbc->sync_mode == WB_SYNC_ALL)
 		atomic_inc(&sbi->wb_sync_req[DATA]);
-	else if (atomic_read(&sbi->wb_sync_req[DATA]))
+	else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
 		goto skip_write;
+	}
 
 	if (__should_serialize_io(inode, wbc)) {
 		mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
 
 	if (wbc->sync_mode == WB_SYNC_ALL)
 		atomic_inc(&sbi->wb_sync_req[NODE]);
-	else if (atomic_read(&sbi->wb_sync_req[NODE]))
+	else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
 		goto skip_write;
+	}
 
 	trace_f2fs_writepages(mapping->host, wbc, NODE);
 
Signature · 1 line
2.32.0
author maintainer
#2 lore raw
Quoted 11 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback] [T:checkpoint]
- wb_writeback
- blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Thanks,

Quoted 70 lines
Quoted 70 lines

- do_writepages -- sync write inodeB, inc wb_sync_req[DATA]
- f2fs_write_data_pages
- f2fs_write_single_data_page -- write last dirty page
- f2fs_do_write_data_page
- set_page_writeback -- clear page dirty flag and PAGECACHE_TAG_DIRTY tag in radix tree
- f2fs_outplace_write_data
- f2fs_update_data_blkaddr
- f2fs_wait_on_page_writeback -- wait NodeA to writeback here
- inode_dec_dirty_pages
- writeback_sb_inodes
- writeback_single_inode
- do_writepages
- f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
- wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
- requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
- blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
  fs/f2fs/data.c | 6 +++++-
  fs/f2fs/node.c | 6 +++++-
  2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
  	/* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
  	if (wbc->sync_mode == WB_SYNC_ALL)
  		atomic_inc(&sbi->wb_sync_req[DATA]);
-	else if (atomic_read(&sbi->wb_sync_req[DATA]))
+	else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
  		goto skip_write;
+	}
 
  	if (__should_serialize_io(inode, wbc)) {
  		mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
 
  	if (wbc->sync_mode == WB_SYNC_ALL)
  		atomic_inc(&sbi->wb_sync_req[NODE]);
-	else if (atomic_read(&sbi->wb_sync_req[NODE]))
+	else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
  		goto skip_write;
+	}
 
  	trace_f2fs_writepages(mapping->host, wbc, NODE);
 
Signature · 1 line
2.32.0
maintainer
#3 lore raw
Quoted 10 lines · Chao Yu
Quoted 9 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback] [T:checkpoint]
- wb_writeback
- blk_start_plug
bio contains NodeA was plugged in writeback threads

Quoted 5 lines
Quoted 2 lines

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Wasn't that added into a different task->plug?

Quoted 69 lines

Thanks,

Quoted 66 lines
Quoted 66 lines

- do_writepages -- sync write inodeB, inc wb_sync_req[DATA]
- f2fs_write_data_pages
- f2fs_write_single_data_page -- write last dirty page
- f2fs_do_write_data_page
- set_page_writeback -- clear page dirty flag and PAGECACHE_TAG_DIRTY tag in radix tree
- f2fs_outplace_write_data
- f2fs_update_data_blkaddr
- f2fs_wait_on_page_writeback -- wait NodeA to writeback here
- inode_dec_dirty_pages
- writeback_sb_inodes
- writeback_single_inode
- do_writepages
- f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
- wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
- requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
- blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
  fs/f2fs/data.c | 6 +++++-
  fs/f2fs/node.c | 6 +++++-
  2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
  	/* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
  	if (wbc->sync_mode == WB_SYNC_ALL)
  		atomic_inc(&sbi->wb_sync_req[DATA]);
-	else if (atomic_read(&sbi->wb_sync_req[DATA]))
+	else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
  		goto skip_write;
+	}
  	if (__should_serialize_io(inode, wbc)) {
  		mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
  	if (wbc->sync_mode == WB_SYNC_ALL)
  		atomic_inc(&sbi->wb_sync_req[NODE]);
-	else if (atomic_read(&sbi->wb_sync_req[NODE]))
+	else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
  		goto skip_write;
+	}
  	trace_f2fs_writepages(mapping->host, wbc, NODE);
Signature · 1 line
2.32.0
author maintainer
#4 lore raw
Quoted 11 lines · Jaegeuk Kim
Quoted 10 lines · Chao Yu
Quoted 9 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback] [T:checkpoint]
- wb_writeback
- blk_start_plug
bio contains NodeA was plugged in writeback threads

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

Thanks,

Quoted 70 lines
Quoted 70 lines

Thanks,

Quoted 66 lines
Quoted 66 lines

- do_writepages -- sync write inodeB, inc wb_sync_req[DATA]
- f2fs_write_data_pages
- f2fs_write_single_data_page -- write last dirty page
- f2fs_do_write_data_page
- set_page_writeback -- clear page dirty flag and PAGECACHE_TAG_DIRTY tag in radix tree
- f2fs_outplace_write_data
- f2fs_update_data_blkaddr
- f2fs_wait_on_page_writeback -- wait NodeA to writeback here
- inode_dec_dirty_pages
- writeback_sb_inodes
- writeback_single_inode
- do_writepages
- f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
- wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
- requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
- blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
   fs/f2fs/data.c | 6 +++++-
   fs/f2fs/node.c | 6 +++++-
   2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
   	/* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
   	if (wbc->sync_mode == WB_SYNC_ALL)
   		atomic_inc(&sbi->wb_sync_req[DATA]);
-	else if (atomic_read(&sbi->wb_sync_req[DATA]))
+	else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
   		goto skip_write;
+	}
   	if (__should_serialize_io(inode, wbc)) {
   		mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
   	if (wbc->sync_mode == WB_SYNC_ALL)
   		atomic_inc(&sbi->wb_sync_req[NODE]);
-	else if (atomic_read(&sbi->wb_sync_req[NODE]))
+	else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
   		goto skip_write;
+	}
   	trace_f2fs_writepages(mapping->host, wbc, NODE);
Signature · 1 line
2.32.0
maintainer
#5 lore raw
Quoted 18 lines · Chao Yu
Quoted 17 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback] [T:checkpoint]
- wb_writeback
- blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Quoted 6 lines
Quoted 1 line

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

I think so.

Quoted 73 lines

Thanks,

Quoted 70 lines
Quoted 70 lines

Thanks,

Quoted 66 lines
Quoted 66 lines

- do_writepages -- sync write inodeB, inc wb_sync_req[DATA]
- f2fs_write_data_pages
- f2fs_write_single_data_page -- write last dirty page
- f2fs_do_write_data_page
- set_page_writeback -- clear page dirty flag and PAGECACHE_TAG_DIRTY tag in radix tree
- f2fs_outplace_write_data
- f2fs_update_data_blkaddr
- f2fs_wait_on_page_writeback -- wait NodeA to writeback here
- inode_dec_dirty_pages
- writeback_sb_inodes
- writeback_single_inode
- do_writepages
- f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
- wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
- requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
- blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
   fs/f2fs/data.c | 6 +++++-
   fs/f2fs/node.c | 6 +++++-
   2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
   	/* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
   	if (wbc->sync_mode == WB_SYNC_ALL)
   		atomic_inc(&sbi->wb_sync_req[DATA]);
-	else if (atomic_read(&sbi->wb_sync_req[DATA]))
+	else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
   		goto skip_write;
+	}
   	if (__should_serialize_io(inode, wbc)) {
   		mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
   	if (wbc->sync_mode == WB_SYNC_ALL)
   		atomic_inc(&sbi->wb_sync_req[NODE]);
-	else if (atomic_read(&sbi->wb_sync_req[NODE]))
+	else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
   		goto skip_write;
+	}
   	trace_f2fs_writepages(mapping->host, wbc, NODE);
Signature · 1 line
2.32.0
author maintainer
#6 lore raw
Quoted 19 lines · Jaegeuk Kim
Quoted 18 lines · Chao Yu
Quoted 17 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback] [T:checkpoint]
- wb_writeback
- blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

I think so.

I guess block plug helper says it doesn't allow to use nested plug, so there is only one plug in kworker thread?

void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)

{
	struct task_struct *tsk = current;

	/*
	 * If this is a nested plug, don't actually assign it.
	 */
	if (tsk->plug)
		return;

...
}

Thanks,

Quoted 74 lines
Quoted 74 lines

Thanks,

Quoted 70 lines
Quoted 70 lines

Thanks,

Quoted 66 lines
Quoted 66 lines

- do_writepages -- sync write inodeB, inc wb_sync_req[DATA]
- f2fs_write_data_pages
- f2fs_write_single_data_page -- write last dirty page
- f2fs_do_write_data_page
- set_page_writeback -- clear page dirty flag and PAGECACHE_TAG_DIRTY tag in radix tree
- f2fs_outplace_write_data
- f2fs_update_data_blkaddr
- f2fs_wait_on_page_writeback -- wait NodeA to writeback here
- inode_dec_dirty_pages
- writeback_sb_inodes
- writeback_single_inode
- do_writepages
- f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
- wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
- requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
- blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
    fs/f2fs/data.c | 6 +++++-
    fs/f2fs/node.c | 6 +++++-
    2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
    	/* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
    	if (wbc->sync_mode == WB_SYNC_ALL)
    		atomic_inc(&sbi->wb_sync_req[DATA]);
-	else if (atomic_read(&sbi->wb_sync_req[DATA]))
+	else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
    		goto skip_write;
+	}
    	if (__should_serialize_io(inode, wbc)) {
    		mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
    	if (wbc->sync_mode == WB_SYNC_ALL)
    		atomic_inc(&sbi->wb_sync_req[NODE]);
-	else if (atomic_read(&sbi->wb_sync_req[NODE]))
+	else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+		/* to avoid potential deadlock */
+		if (current->plug)
+			blk_finish_plug(current->plug);
    		goto skip_write;
+	}
    	trace_f2fs_writepages(mapping->host, wbc, NODE);
Signature · 1 line
2.32.0
author maintainer
#7 lore raw
Quoted 36 lines · Chao Yu
Quoted 19 lines · Jaegeuk Kim
Quoted 18 lines · Chao Yu
Quoted 17 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback]                [T:checkpoint]
- wb_writeback
    - blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

I think so.

I guess block plug helper says it doesn't allow to use nested plug, so there is only one plug in kworker thread?

void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) {
    struct task_struct *tsk = current;

Quoted 7 lines

    /*
     * If this is a nested plug, don't actually assign it. */
    if (tsk->plug)
        return;
...
}

Any further comments?

Thanks,

Quoted 83 lines

Thanks,

Quoted 66 lines
Quoted 66 lines

Thanks,

Quoted 62 lines
Quoted 62 lines

Thanks,

Quoted 58 lines
Quoted 58 lines

                    - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
                     - f2fs_write_data_pages
                      - f2fs_write_single_data_page -- write last dirty page
                       - f2fs_do_write_data_page
                        - set_page_writeback  -- clear page dirty flag and
                        PAGECACHE_TAG_DIRTY tag in radix tree
                        - f2fs_outplace_write_data
                         - f2fs_update_data_blkaddr
                          - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
                       - inode_dec_dirty_pages
    - writeback_sb_inodes
     - writeback_single_inode
      - do_writepages
       - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
        - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
     - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
    - blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")

---
    fs/f2fs/data.c | 6 +++++-
    fs/f2fs/node.c | 6 +++++-

    2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
         /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[DATA]);
-    else if (atomic_read(&sbi->wb_sync_req[DATA]))
+    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         if (__should_serialize_io(inode, wbc)) {
             mutex_lock(&sbi->writepages);
fs/f2fs/node.c +1 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[NODE]);
-    else if (atomic_read(&sbi->wb_sync_req[NODE]))
+    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
fs/f2fs/node.c +4 -0
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         trace_f2fs_writepages(mapping->host, wbc, NODE);
Signature · 1 line
2.32.0

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

author maintainer
#8 lore raw

ping,

Quoted 138 lines · Chao Yu
Quoted 36 lines · Chao Yu
Quoted 19 lines · Jaegeuk Kim
Quoted 18 lines · Chao Yu
Quoted 17 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback]                [T:checkpoint]
- wb_writeback
    - blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

I think so.

I guess block plug helper says it doesn't allow to use nested plug, so there is only one plug in kworker thread?

void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) {
     struct task_struct *tsk = current;

Quoted 7 lines

     /*
      * If this is a nested plug, don't actually assign it. */
     if (tsk->plug)
         return;
...
}

Any further comments?

Thanks,

Quoted 75 lines

Thanks,

Quoted 72 lines
Quoted 72 lines

Thanks,

Quoted 68 lines
Quoted 68 lines

Thanks,

Quoted 64 lines
Quoted 64 lines

                    - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
                     - f2fs_write_data_pages
                      - f2fs_write_single_data_page -- write last dirty page
                       - f2fs_do_write_data_page
                        - set_page_writeback  -- clear page dirty flag and
                        PAGECACHE_TAG_DIRTY tag in radix tree
                        - f2fs_outplace_write_data
                         - f2fs_update_data_blkaddr
                          - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
                       - inode_dec_dirty_pages
    - writeback_sb_inodes
     - writeback_single_inode
      - do_writepages
       - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
        - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
     - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
    - blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")

---
    fs/f2fs/data.c | 6 +++++-
    fs/f2fs/node.c | 6 +++++-

    2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
         /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[DATA]);
-    else if (atomic_read(&sbi->wb_sync_req[DATA]))
+    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         if (__should_serialize_io(inode, wbc)) {
             mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[NODE]);
-    else if (atomic_read(&sbi->wb_sync_req[NODE]))
+    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         trace_f2fs_writepages(mapping->host, wbc, NODE);
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signature · 1 line
2.32.0

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

maintainer
#9 lore raw
Quoted 35 lines · Chao Yu

ping,

Quoted 32 lines · Chao Yu
Quoted 31 lines · Chao Yu
Quoted 19 lines · Jaegeuk Kim
Quoted 18 lines · Chao Yu
Quoted 17 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback]                [T:checkpoint]
- wb_writeback
    - blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

I think so.

I guess block plug helper says it doesn't allow to use nested plug, so there is only one plug in kworker thread?

Is there only one kworker thread that flushes node and inode pages?

Quoted 106 lines
Quoted 106 lines
Quoted 5 lines

void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) {
     struct task_struct *tsk = current;

Quoted 7 lines

     /*
      * If this is a nested plug, don't actually assign it. */
     if (tsk->plug)
         return;
...
}

Any further comments?

Thanks,

Quoted 75 lines

Thanks,

Quoted 72 lines
Quoted 72 lines

Thanks,

Quoted 68 lines
Quoted 68 lines

Thanks,

Quoted 64 lines
Quoted 64 lines

                    - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
                     - f2fs_write_data_pages
                      - f2fs_write_single_data_page -- write last dirty page
                       - f2fs_do_write_data_page
                        - set_page_writeback  -- clear page dirty flag and
                        PAGECACHE_TAG_DIRTY tag in radix tree
                        - f2fs_outplace_write_data
                         - f2fs_update_data_blkaddr
                          - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
                       - inode_dec_dirty_pages
    - writeback_sb_inodes
     - writeback_single_inode
      - do_writepages
       - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
        - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
     - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
    - blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")

---
    fs/f2fs/data.c | 6 +++++-
    fs/f2fs/node.c | 6 +++++-

    2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
         /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[DATA]);
-    else if (atomic_read(&sbi->wb_sync_req[DATA]))
+    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         if (__should_serialize_io(inode, wbc)) {
             mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[NODE]);
-    else if (atomic_read(&sbi->wb_sync_req[NODE]))
+    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         trace_f2fs_writepages(mapping->host, wbc, NODE);
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signature · 1 line
2.32.0

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

author maintainer
#10 lore raw
Quoted 32 lines · Jaegeuk Kim
Quoted 31 lines · Chao Yu

ping,

Quoted 28 lines · Chao Yu
Quoted 27 lines · Chao Yu
Quoted 26 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 10 lines · Chao Yu
Quoted 9 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback]                [T:checkpoint]
- wb_writeback
    - blk_start_plug
bio contains NodeA was plugged in writeback threads

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

I think so.

I guess block plug helper says it doesn't allow to use nested plug, so there is only one plug in kworker thread?

Is there only one kworker thread that flushes node and inode pages?

IIRC, =one kworker per block device?

Thanks,

Quoted 103 lines
Quoted 103 lines
Quoted 103 lines
Quoted 6 lines

void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) {
     struct task_struct *tsk = current;

Quoted 7 lines

     /*
      * If this is a nested plug, don't actually assign it. */
     if (tsk->plug)
         return;
...
}

Any further comments?

Thanks,

Quoted 83 lines

Thanks,

Quoted 66 lines
Quoted 66 lines

Thanks,

Quoted 62 lines
Quoted 62 lines

Thanks,

Quoted 58 lines
Quoted 58 lines

                    - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
                     - f2fs_write_data_pages
                      - f2fs_write_single_data_page -- write last dirty page
                       - f2fs_do_write_data_page
                        - set_page_writeback  -- clear page dirty flag and
                        PAGECACHE_TAG_DIRTY tag in radix tree
                        - f2fs_outplace_write_data
                         - f2fs_update_data_blkaddr
                          - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
                       - inode_dec_dirty_pages
    - writeback_sb_inodes
     - writeback_single_inode
      - do_writepages
       - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
        - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
     - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
    - blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
     fs/f2fs/data.c | 6 +++++-
     fs/f2fs/node.c | 6 +++++-
     2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
         /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[DATA]);
-    else if (atomic_read(&sbi->wb_sync_req[DATA]))
+    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         if (__should_serialize_io(inode, wbc)) {
             mutex_lock(&sbi->writepages);
fs/f2fs/node.c +1 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[NODE]);
-    else if (atomic_read(&sbi->wb_sync_req[NODE]))
+    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
fs/f2fs/node.c +4 -0
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         trace_f2fs_writepages(mapping->host, wbc, NODE);
Signature · 1 line
2.32.0

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

maintainer
#11 lore raw
Quoted 33 lines · Chao Yu
Quoted 32 lines · Jaegeuk Kim
Quoted 31 lines · Chao Yu

ping,

Quoted 28 lines · Chao Yu
Quoted 27 lines · Chao Yu
Quoted 26 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 10 lines · Chao Yu
Quoted 9 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback]                [T:checkpoint]
- wb_writeback
    - blk_start_plug
bio contains NodeA was plugged in writeback threads

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

I think so.

I guess block plug helper says it doesn't allow to use nested plug, so there is only one plug in kworker thread?

Is there only one kworker thread that flushes node and inode pages?

IIRC, =one kworker per block device?

If there's one kworker only, f2fs_write_node_pages() should have flushed its plug?

Quoted 110 lines

Thanks,

Quoted 107 lines
Quoted 107 lines
Quoted 107 lines
Quoted 6 lines

void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) {
     struct task_struct *tsk = current;

Quoted 7 lines

     /*
      * If this is a nested plug, don't actually assign it. */
     if (tsk->plug)
         return;
...
}

Any further comments?

Thanks,

Quoted 75 lines

Thanks,

Quoted 72 lines
Quoted 72 lines

Thanks,

Quoted 68 lines
Quoted 68 lines

Thanks,

Quoted 64 lines
Quoted 64 lines

                    - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
                     - f2fs_write_data_pages
                      - f2fs_write_single_data_page -- write last dirty page
                       - f2fs_do_write_data_page
                        - set_page_writeback  -- clear page dirty flag and
                        PAGECACHE_TAG_DIRTY tag in radix tree
                        - f2fs_outplace_write_data
                         - f2fs_update_data_blkaddr
                          - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
                       - inode_dec_dirty_pages
    - writeback_sb_inodes
     - writeback_single_inode
      - do_writepages
       - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
        - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
     - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
    - blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
     fs/f2fs/data.c | 6 +++++-
     fs/f2fs/node.c | 6 +++++-
     2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
         /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[DATA]);
-    else if (atomic_read(&sbi->wb_sync_req[DATA]))
+    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         if (__should_serialize_io(inode, wbc)) {
             mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
         if (wbc->sync_mode == WB_SYNC_ALL)
             atomic_inc(&sbi->wb_sync_req[NODE]);
-    else if (atomic_read(&sbi->wb_sync_req[NODE]))
+    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
             goto skip_write;
+    }
         trace_f2fs_writepages(mapping->host, wbc, NODE);
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signature · 1 line
2.32.0

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

author maintainer
#12 lore raw
Quoted 39 lines · Jaegeuk Kim
Quoted 38 lines · Chao Yu
Quoted 37 lines · Jaegeuk Kim
Quoted 35 lines · Chao Yu

ping,

Quoted 32 lines · Chao Yu
Quoted 31 lines · Chao Yu
Quoted 19 lines · Jaegeuk Kim
Quoted 18 lines · Chao Yu
Quoted 17 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen between kworker and checkpoint as below:

[T:writeback]                [T:checkpoint]
- wb_writeback
    - blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

I think so.

I guess block plug helper says it doesn't allow to use nested plug, so there is only one plug in kworker thread?

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Is there only one kworker thread that flushes node and inode pages?

IIRC, =one kworker per block device?

If there's one kworker only, f2fs_write_node_pages() should have flushed its plug?

No, f2fs_write_node_pages() failed to attach local plug into current->plug due to current has attached plug from wb_writeback(), and also, f2fs_write_node_pages() will fail to flush current->plug due to its local plug doesn't match current->plug.

void blk_start_plug_nr_ios()

{
	if (tsk->plug)
		return;

...
}

void blk_finish_plug(struct blk_plug *plug)

{
	if (plug == current->plug) {
		__blk_flush_plug(plug, false);
		current->plug = NULL;
	}
}

Thanks,

Quoted 111 lines
Quoted 111 lines

Thanks,

Quoted 107 lines
Quoted 107 lines
Quoted 107 lines
Quoted 6 lines

void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) {
     struct task_struct *tsk = current;

Quoted 7 lines

     /*
      * If this is a nested plug, don't actually assign it. */
     if (tsk->plug)
         return;
...
}

Any further comments?

Thanks,

Quoted 75 lines

Thanks,

Quoted 72 lines
Quoted 72 lines

Thanks,

Quoted 68 lines
Quoted 68 lines

Thanks,

Quoted 64 lines
Quoted 64 lines

                    - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
                     - f2fs_write_data_pages
                      - f2fs_write_single_data_page -- write last dirty page
                       - f2fs_do_write_data_page
                        - set_page_writeback  -- clear page dirty flag and
                        PAGECACHE_TAG_DIRTY tag in radix tree
                        - f2fs_outplace_write_data
                         - f2fs_update_data_blkaddr
                          - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
                       - inode_dec_dirty_pages
    - writeback_sb_inodes
     - writeback_single_inode
      - do_writepages
       - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
        - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
     - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
    - blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via blk_finish_plug(current->plug) once we'v skipped writeback in writepages() due to valid sbi->wb_sync_req[DATA/NODE].

Fixes 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
---
      fs/f2fs/data.c | 6 +++++-
      fs/f2fs/node.c | 6 +++++-
      2 files changed, 10 insertions(+), 2 deletions(-)
fs/f2fs/data.c +5 -1
index 76d6fe7b0c8f..932a4c81acaf 100644
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
          /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
          if (wbc->sync_mode == WB_SYNC_ALL)
              atomic_inc(&sbi->wb_sync_req[DATA]);
-    else if (atomic_read(&sbi->wb_sync_req[DATA]))
+    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
              goto skip_write;
+    }
          if (__should_serialize_io(inode, wbc)) {
              mutex_lock(&sbi->writepages);
fs/f2fs/node.c +5 -1
index 556fcd8457f3..69c6bcaf5aae 100644
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
          if (wbc->sync_mode == WB_SYNC_ALL)
              atomic_inc(&sbi->wb_sync_req[NODE]);
-    else if (atomic_read(&sbi->wb_sync_req[NODE]))
+    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
              goto skip_write;
+    }
          trace_f2fs_writepages(mapping->host, wbc, NODE);
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signature · 1 line
2.32.0

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

maintainer
#13 lore raw
Quoted 58 lines · Chao Yu
Quoted 39 lines · Jaegeuk Kim
Quoted 38 lines · Chao Yu
Quoted 37 lines · Jaegeuk Kim
Quoted 35 lines · Chao Yu

ping,

Quoted 32 lines · Chao Yu
Quoted 31 lines · Chao Yu
Quoted 19 lines · Jaegeuk Kim
Quoted 18 lines · Chao Yu
Quoted 17 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 11 lines · Jaegeuk Kim
Quoted 7 lines · Chao Yu

Quoted from Jing Xia's report, there is a potential deadlock may happen
between kworker and checkpoint as below:

[T:writeback]                [T:checkpoint]
- wb_writeback
    - blk_start_plug
bio contains NodeA was plugged in writeback threads

I'm still trying to understand more precisely. So, how is it possible to have bio having node write in this current context?

IMO, after above blk_start_plug(), it may plug some inode's node page in kworker during writebacking node_inode's data page (which should be node page)?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Wasn't that added into a different task->plug?

I'm not sure I've got your concern correctly...

Do you mean NodeA and other IOs from do_writepages() were plugged in different local plug variables?

I think so.

I guess block plug helper says it doesn't allow to use nested plug, so there is only one plug in kworker thread?

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Is there only one kworker thread that flushes node and inode pages?

IIRC, =one kworker per block device?

If there's one kworker only, f2fs_write_node_pages() should have flushed its plug?

No, f2fs_write_node_pages() failed to attach local plug into current->plug due to current has attached plug from wb_writeback(), and also, f2fs_write_node_pages() will fail to flush current->plug due to its local plug doesn't match current->plug.

void blk_start_plug_nr_ios()

{
	if (tsk->plug)
		return;

...
}

Quoted 7 lines

void blk_finish_plug(struct blk_plug *plug)

{
	if (plug == current->plug) {
		__blk_flush_plug(plug, false);
		current->plug = NULL;
	}
}

Ah, okay. Now I see. Thanks for the chasing down.

Quoted 114 lines

Thanks,

Quoted 111 lines
Quoted 111 lines

Thanks,

Quoted 107 lines
Quoted 107 lines
Quoted 107 lines
Quoted 6 lines

void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) {
     struct task_struct *tsk = current;

Quoted 7 lines

     /*
      * If this is a nested plug, don't actually assign it. */
     if (tsk->plug)
         return;
...
}

Any further comments?

Thanks,

Quoted 75 lines

Thanks,

Quoted 72 lines
Quoted 72 lines

Thanks,

Quoted 68 lines
Quoted 68 lines

Thanks,

Quoted 64 lines
Quoted 64 lines

                    - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
                     - f2fs_write_data_pages
                      - f2fs_write_single_data_page -- write last dirty page
                       - f2fs_do_write_data_page
                        - set_page_writeback  -- clear page dirty flag and
                        PAGECACHE_TAG_DIRTY tag in radix tree
                        - f2fs_outplace_write_data
                         - f2fs_update_data_blkaddr
                          - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
                       - inode_dec_dirty_pages
    - writeback_sb_inodes
     - writeback_single_inode
      - do_writepages
       - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
        - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
     - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
    - blk_finish_plug

Let's try to avoid deadlock condition by forcing unplugging previous bio via
blk_finish_plug(current->plug) once we'v skipped writeback in writepages()
due to valid sbi->wb_sync_req[DATA/NODE].

Fixes: 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
Signed-off-by: Zhiguo Niu <[email protected]>
Signed-off-by: Jing Xia <[email protected]>
Signed-off-by: Chao Yu <[email protected]>
---
    fs/f2fs/data.c | 6 +++++-
    fs/f2fs/node.c | 6 +++++-
    2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 76d6fe7b0c8f..932a4c81acaf 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
        /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
        if (wbc->sync_mode == WB_SYNC_ALL)
            atomic_inc(&sbi->wb_sync_req[DATA]);
-    else if (atomic_read(&sbi->wb_sync_req[DATA]))
+    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
            goto skip_write;
+    }
        if (__should_serialize_io(inode, wbc)) {
            mutex_lock(&sbi->writepages);
diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
index 556fcd8457f3..69c6bcaf5aae 100644
--- a/fs/f2fs/node.c
+++ b/fs/f2fs/node.c
@@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
        if (wbc->sync_mode == WB_SYNC_ALL)
            atomic_inc(&sbi->wb_sync_req[NODE]);
-    else if (atomic_read(&sbi->wb_sync_req[NODE]))
+    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
+        /* to avoid potential deadlock */
+        if (current->plug)
+            blk_finish_plug(current->plug);
            goto skip_write;
+    }
        trace_f2fs_writepages(mapping->host, wbc, NODE);

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
2.32.0

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel