Skip to contentSkip to conversation

f2fs: don't skip writeback of quota data

PATCH In review Jaegeuk Kim 2 participants 15 messages Jul 9, 2020 – Jul 16, 2020 F2FS file system

lore
Share & tools
lore

Jaegeuk Kim

author maintainer
#0 lore raw

PATCH 1/1 f2fs: don't skip writeback of quota data

It doesn't need to bypass flushing quota data in background.

---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +1 -1
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))
Signature · 1 line
2.27.0.383.g050319c2ae-goog
#1 lore raw
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

Thanks,

Quoted 20 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +1 -1
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))
author maintainer
#2 lore raw
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Thanks,

Quoted 23 lines

Thanks,

Quoted 20 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +1 -1
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))
#3 lore raw
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

author maintainer
#4 lore raw
Quoted 12 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Quoted 5 lines

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Quoted 31 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

#5 lore raw
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

Quoted 34 lines
Quoted 32 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

.

author maintainer
#6 lore raw
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M
Quoted 3 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make dirty pages in more fine granularity.

Quoted 35 lines
Quoted 35 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

.

#7 lore raw
Quoted 46 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
Quoted 8 lines
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M

Does this only happen during umount? If so, will below change help?

	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+			!is_sbi_flag_set(sbi, SBI_IS_CLOSE) &&
			wbc->sync_mode == WB_SYNC_NONE &&
			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
			f2fs_available_free_memory(sbi, DIRTY_DENTS))
		goto skip_write;
Quoted 7 lines
Quoted 4 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make

No guarantee that there is no dirty dentry being created after cp_error, right?

e.g.

Thread A Thread B
- f2fs_create
- bypass f2fs_cp_error
- set cp_error
- create dirty dentry

BTW, do you know what __writeback_inodes_sb_nr is waiting for?

Quoted 39 lines

dirty pages in more fine granularity.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

Quoted 2 lines
Quoted 2 lines

.

.

author maintainer
#8 lore raw
Quoted 56 lines · Chao Yu
Quoted 46 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
Quoted 8 lines
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M
Quoted 1 line

Does this only happen during umount? If so, will below change help?

Will give it a try. It's quite flaky so may take some time.

Quoted 68 lines
	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
(quoted diff) +1 -0
+			!is_sbi_flag_set(sbi, SBI_IS_CLOSE) &&
			wbc->sync_mode == WB_SYNC_NONE &&
			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
			f2fs_available_free_memory(sbi, DIRTY_DENTS))
		goto skip_write;
Quoted 7 lines
Quoted 4 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make

No guarantee that there is no dirty dentry being created after cp_error, right?

e.g.

Thread A Thread B
- f2fs_create
- bypass f2fs_cp_error
- set cp_error
- create dirty dentry

BTW, do you know what __writeback_inodes_sb_nr is waiting for?

Quoted 39 lines

dirty pages in more fine granularity.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

Quoted 2 lines
Quoted 2 lines

.

.

author maintainer
#9 lore raw
In reply to Chao Yu
Quoted 60 lines · Chao Yu
Quoted 46 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
Quoted 8 lines
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M

Does this only happen during umount? If so, will below change help?

	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
(quoted diff) +1 -0
+			!is_sbi_flag_set(sbi, SBI_IS_CLOSE) &&
Quoted 4 lines
			wbc->sync_mode == WB_SYNC_NONE &&
			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
			f2fs_available_free_memory(sbi, DIRTY_DENTS))
		goto skip_write;

Hmm, this doesn't work. The writeback was called before put_super? I'll try the original patch one more time.

Quoted 61 lines
Quoted 8 lines
Quoted 5 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make

No guarantee that there is no dirty dentry being created after cp_error, right?

e.g.

Thread A Thread B
- f2fs_create
- bypass f2fs_cp_error
- set cp_error
- create dirty dentry

BTW, do you know what __writeback_inodes_sb_nr is waiting for?

Quoted 39 lines

dirty pages in more fine granularity.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

Quoted 2 lines
Quoted 2 lines

.

.

#10 lore raw
Quoted 59 lines · Jaegeuk Kim
Quoted 58 lines · Chao Yu
Quoted 46 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
Quoted 8 lines
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M

Does this only happen during umount? If so, will below change help?

Quoted 8 lines
Quoted 6 lines
	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
(quoted diff) +1 -0
+			!is_sbi_flag_set(sbi, SBI_IS_CLOSE) &&
			wbc->sync_mode == WB_SYNC_NONE &&
			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
			f2fs_available_free_memory(sbi, DIRTY_DENTS))
		goto skip_write;

Hmm, this doesn't work. The writeback was called before put_super?

Oops, still be confused about this issue. :(

Thanks,

Quoted 65 lines

I'll try the original patch one more time.

Quoted 14 lines
Quoted 8 lines
Quoted 5 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make

No guarantee that there is no dirty dentry being created after cp_error, right?

e.g.

Quoted 47 lines

Thread A Thread B
- f2fs_create
- bypass f2fs_cp_error
- set cp_error
- create dirty dentry

BTW, do you know what __writeback_inodes_sb_nr is waiting for?

Quoted 39 lines

dirty pages in more fine granularity.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

Quoted 2 lines
Quoted 2 lines

.

.

.

author maintainer
#11 lore raw
Quoted 69 lines · Chao Yu
Quoted 59 lines · Jaegeuk Kim
Quoted 58 lines · Chao Yu
Quoted 46 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
Quoted 8 lines
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M

Does this only happen during umount? If so, will below change help?

Quoted 8 lines
Quoted 6 lines
	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
(quoted diff) +1 -0
+			!is_sbi_flag_set(sbi, SBI_IS_CLOSE) &&
			wbc->sync_mode == WB_SYNC_NONE &&
			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
			f2fs_available_free_memory(sbi, DIRTY_DENTS))
		goto skip_write;

Hmm, this doesn't work. The writeback was called before put_super?

Quoted 1 line

Oops, still be confused about this issue. :(

Huam, I hit the problem with the patch.
I need to return back and think in other way. :(

Quoted 68 lines

Thanks,

Quoted 65 lines

I'll try the original patch one more time.

Quoted 14 lines
Quoted 8 lines
Quoted 5 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make

No guarantee that there is no dirty dentry being created after cp_error, right?

e.g.

Quoted 47 lines

Thread A Thread B
- f2fs_create
- bypass f2fs_cp_error
- set cp_error
- create dirty dentry

BTW, do you know what __writeback_inodes_sb_nr is waiting for?

Quoted 39 lines

dirty pages in more fine granularity.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

Quoted 2 lines
Quoted 2 lines

.

.

.

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#12 lore raw
Quoted 70 lines · Jaegeuk Kim
Quoted 69 lines · Chao Yu
Quoted 67 lines · Jaegeuk Kim
Quoted 60 lines · Chao Yu
Quoted 46 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
Quoted 8 lines
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M

Does this only happen during umount? If so, will below change help?

	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
(quoted diff) +1 -0
+			!is_sbi_flag_set(sbi, SBI_IS_CLOSE) &&
Quoted 4 lines
			wbc->sync_mode == WB_SYNC_NONE &&
			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
			f2fs_available_free_memory(sbi, DIRTY_DENTS))
		goto skip_write;

Hmm, this doesn't work. The writeback was called before put_super?

Quoted 4 lines
Quoted 1 line

Oops, still be confused about this issue. :(

Huam, I hit the problem with the patch.
I need to return back and think in other way. :(

Still quota data was left? what about dentry?

Thanks,

Quoted 71 lines
Quoted 69 lines

Thanks,

Quoted 65 lines

I'll try the original patch one more time.

Quoted 14 lines
Quoted 8 lines
Quoted 5 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make

No guarantee that there is no dirty dentry being created after cp_error, right?

e.g.

Quoted 47 lines

Thread A Thread B
- f2fs_create
- bypass f2fs_cp_error
- set cp_error
- create dirty dentry

BTW, do you know what __writeback_inodes_sb_nr is waiting for?

Quoted 39 lines

dirty pages in more fine granularity.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

Quoted 2 lines
Quoted 2 lines

.

.

.

.

author maintainer
#13 lore raw
Quoted 71 lines · Chao Yu
Quoted 70 lines · Jaegeuk Kim
Quoted 69 lines · Chao Yu
Quoted 67 lines · Jaegeuk Kim
Quoted 60 lines · Chao Yu
Quoted 46 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
Quoted 8 lines
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M

Does this only happen during umount? If so, will below change help?

	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
(quoted diff) +1 -0
+			!is_sbi_flag_set(sbi, SBI_IS_CLOSE) &&
Quoted 4 lines
			wbc->sync_mode == WB_SYNC_NONE &&
			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
			f2fs_available_free_memory(sbi, DIRTY_DENTS))
		goto skip_write;

Hmm, this doesn't work. The writeback was called before put_super?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Oops, still be confused about this issue. :(

Huam, I hit the problem with the patch.
I need to return back and think in other way. :(

Still quota data was left? what about dentry?

Huh, at this time, only node and meta pages were left as dirty.

Quoted 74 lines

Thanks,

Quoted 71 lines
Quoted 69 lines

Thanks,

Quoted 65 lines

I'll try the original patch one more time.

Quoted 14 lines
Quoted 8 lines
Quoted 5 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make

No guarantee that there is no dirty dentry being created after cp_error, right?

e.g.

Quoted 47 lines

Thread A Thread B
- f2fs_create
- bypass f2fs_cp_error
- set cp_error
- create dirty dentry

BTW, do you know what __writeback_inodes_sb_nr is waiting for?

Quoted 39 lines

dirty pages in more fine granularity.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines
---
 fs/f2fs/data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/f2fs/data.c +0 -0
index 44645f4f914b6..72e8b50e588c1 100644
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
 	if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
 		goto skip_write;
 
Quoted 6 lines

fs/f2fs/data.c static int __f2fs_write_data_pages(struct address_space *mapping, · line 3154

Quoted 6 lines
fs/f2fs/data.c +1 -1
-	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+	if (S_ISDIR(inode->i_mode) &&
 			wbc->sync_mode == WB_SYNC_NONE &&
 			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
 			f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

Quoted 2 lines
Quoted 2 lines

.

.

.

.

#14 lore raw
Quoted 72 lines · Jaegeuk Kim
Quoted 71 lines · Chao Yu
Quoted 70 lines · Jaegeuk Kim
Quoted 69 lines · Chao Yu
Quoted 67 lines · Jaegeuk Kim
Quoted 60 lines · Chao Yu
Quoted 46 lines · Jaegeuk Kim
Quoted 24 lines · Chao Yu
Quoted 16 lines · Jaegeuk Kim
Quoted 15 lines · Chao Yu
Quoted 10 lines · Jaegeuk Kim
Quoted 5 lines · Chao Yu
Quoted 1 line · Jaegeuk Kim

It doesn't need to bypass flushing quota data in background.

The condition is used to flush quota data in batch to avoid random small-sized udpate, did you hit any problem here?

I suspect this causes fault injection test being stuck by waiting for inode writeback completion. With this patch, it has been running w/o any issue so far. I keep an eye on this.

Hmmm.. so that this patch may not fix the root cause, and it may hiding the issue deeper.

Quoted 6 lines
Quoted 2 lines

How about just keeping this patch in our private branch to let fault injection test not be stuck? until we find the root cause in upstream codes.

Well, I don't think this hides something. When the issue happens, I saw inodes being stuck due to writeback while only quota has some dirty data. At that time, there was no dirty data page from other inodes.

Quoted 7 lines

Okay,

Quoted 3 lines

More specifically, I suspect __writeback_inodes_sb_nr() gives WB_SYNC_NONE and waits for wb_wait_for_completion().

Did you record any callstack after the issue happened?

I found this.

[213389.297642]  __schedule+0x2dd/0x780^M
[213389.299224]  schedule+0x55/0xc0^M
[213389.300745]  wb_wait_for_completion+0x56/0x90^M
[213389.302469]  ? wait_woken+0x80/0x80^M
[213389.303997]  __writeback_inodes_sb_nr+0xa8/0xd0^M
[213389.305760]  writeback_inodes_sb+0x4b/0x60^M
[213389.307439]  sync_filesystem+0x2e/0xa0^M
[213389.308999]  generic_shutdown_super+0x27/0x110^M
[213389.310738]  kill_block_super+0x27/0x50^M
[213389.312327]  kill_f2fs_super+0x76/0xe0 [f2fs]^M
[213389.314014]  deactivate_locked_super+0x3b/0x80^M
Quoted 8 lines
[213389.315692]  deactivate_super+0x3e/0x50^M
[213389.317226]  cleanup_mnt+0x109/0x160^M
[213389.318718]  __cleanup_mnt+0x12/0x20^M
[213389.320177]  task_work_run+0x70/0xb0^M
[213389.321609]  exit_to_usermode_loop+0x131/0x160^M
[213389.323306]  do_syscall_64+0x170/0x1b0^M
[213389.324762]  entry_SYSCALL_64_after_hwframe+0x44/0xa9^M
[213389.326477] RIP: 0033:0x7fc4b5e6a35b^M

Does this only happen during umount? If so, will below change help?

	if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
(quoted diff) +1 -0
+			!is_sbi_flag_set(sbi, SBI_IS_CLOSE) &&
Quoted 4 lines
			wbc->sync_mode == WB_SYNC_NONE &&
			get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
			f2fs_available_free_memory(sbi, DIRTY_DENTS))
		goto skip_write;

Hmm, this doesn't work. The writeback was called before put_super?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Oops, still be confused about this issue. :(

Huam, I hit the problem with the patch.
I need to return back and think in other way. :(

Still quota data was left? what about dentry?

Huh, at this time, only node and meta pages were left as dirty.

I saw both meta and node's writepages() called from background will skip flushing dirty pages if dirty page count didn't touch the threshold.

	/* collect a number of dirty node pages and write together */
	if (wbc->sync_mode != WB_SYNC_ALL &&
			get_pages(sbi, F2FS_DIRTY_NODES) <
					nr_pages_to_skip(sbi, NODE))
		goto skip_write;

	/* collect a number of dirty meta pages and write together */
	if (wbc->sync_mode != WB_SYNC_ALL &&
			get_pages(sbi, F2FS_DIRTY_META) <
					nr_pages_to_skip(sbi, META))
		goto skip_write;

At that time, would syncfs (type sync command via shell) help?

Quoted 77 lines
Quoted 75 lines

Thanks,

Quoted 71 lines
Quoted 69 lines

Thanks,

Quoted 65 lines

I'll try the original patch one more time.

Quoted 14 lines
Quoted 8 lines
Quoted 5 lines

Still I'm confused that why directory's data written could be skipped, but quota's data couldn't, what's the difference?

I suspect different blocking timing from cp_error between quota and dentry. e.g., we block dir operations right after cp_error, while quota can make

No guarantee that there is no dirty dentry being created after cp_error, right?

e.g.

Quoted 47 lines

Thread A Thread B
- f2fs_create
- bypass f2fs_cp_error
- set cp_error
- create dirty dentry

BTW, do you know what __writeback_inodes_sb_nr is waiting for?

Quoted 39 lines

dirty pages in more fine granularity.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks,

Quoted 28 lines

Thanks,

Quoted 17 lines

Thanks,

Quoted 14 lines

Signed-off-by: Jaegeuk Kim <[email protected]>
---
fs/f2fs/data.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 44645f4f914b6..72e8b50e588c1 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -3148,7 +3148,7 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
goto skip_write;

Quoted 6 lines
Quoted 6 lines

- if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+ if (S_ISDIR(inode->i_mode) &&
wbc->sync_mode == WB_SYNC_NONE &&
get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
f2fs_available_free_memory(sbi, DIRTY_DENTS))

.

Quoted 2 lines
Quoted 2 lines

.

.

.

.

.