Skip to contentSkip to conversation

f2fs: dump more information when checkpoint was blocked for long time

PATCH In review Chao Yu 2 participants 4 messages Jul 29, 2025 – Jul 30, 2025 F2FS file system

lore
Share & tools
lore

Chao Yu

author maintainer
#0 lore raw

PATCH 1/1 f2fs: dump more information when checkpoint was blocked for long time

generic/299 w/ mode=lfs will cause long time latency, let's dump more information once we hit case.

CP merge:
- Queued : 0
- Issued : 1
- Total : 1
- Cur time : 7565(ms)
- Peak time : 7565(ms)

F2FS-fs (vdc): checkpoint was blocked for 7565 ms, affecting 1 tasks

CPU: 8 UID: 0 PID: 1614 Comm: f2fs_ckpt-253:3 Tainted: G           O        6.16.0-rc3+ #406 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE
Call Trace:
 dump_stack_lvl+0x6e/0xa0
 __checkpoint_and_complete_reqs+0x1a6/0x1d0
 issue_checkpoint_thread+0x4b/0x140
 kthread+0x10d/0x250
 ret_from_fork+0x164/0x190
 ret_from_fork_asm+0x1a/0x30
Cc Jan Prusakowski <[email protected]>
---
 fs/f2fs/checkpoint.c | 9 ++++++++-
 fs/f2fs/f2fs.h       | 3 +++
 2 files changed, 11 insertions(+), 1 deletion(-)
fs/f2fs/checkpoint.c +8 -1
index db3831f7f2f5..b0dcaa8dc40d 100644
@@ -1788,8 +1788,15 @@ static void __checkpoint_and_complete_reqs(struct f2fs_sb_info *sbi)
 
 	spin_lock(&cprc->stat_lock);
 	cprc->cur_time = (unsigned int)div64_u64(sum_diff, count);
-	if (cprc->peak_time < cprc->cur_time)
+	if (cprc->peak_time < cprc->cur_time) {
 		cprc->peak_time = cprc->cur_time;
+
+		if (unlikely(cprc->peak_time >= CP_LONG_LATENCY_THRESHOLD)) {
+			f2fs_warn_ratelimited(sbi, "checkpoint was blocked for %u ms, affecting %llu tasks",
+					cprc->peak_time, count);
+			dump_stack();
+		}
+	}
 	spin_unlock(&cprc->stat_lock);
 }
 
fs/f2fs/f2fs.h +3 -0
index 46d23c2c086c..3130ca6a4770 100644
@@ -350,6 +350,9 @@ struct ckpt_req_control {
 	unsigned int peak_time;		/* peak wait time in msec until now */
 };
 
+/* a time threshold that checkpoint was blocked for, unit: ms */
+#define CP_LONG_LATENCY_THRESHOLD	5000
+
 /* for the bitmap indicate blocks to be discarded */
 struct discard_entry {
 	struct list_head list;	/* list head */
Signature · 1 line
2.49.0
maintainer
#1 lore raw
Quoted 11 lines · Chao Yu

generic/299 w/ mode=lfs will cause long time latency, let's dump more information once we hit case.

CP merge:
- Queued : 0
- Issued : 1
- Total : 1
- Cur time : 7565(ms)
- Peak time : 7565(ms)

F2FS-fs (vdc): checkpoint was blocked for 7565 ms, affecting 1 tasks

Quoted 9 lines
CPU: 8 UID: 0 PID: 1614 Comm: f2fs_ckpt-253:3 Tainted: G           O        6.16.0-rc3+ #406 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE
Call Trace:
 dump_stack_lvl+0x6e/0xa0
 __checkpoint_and_complete_reqs+0x1a6/0x1d0
 issue_checkpoint_thread+0x4b/0x140
 kthread+0x10d/0x250
 ret_from_fork+0x164/0x190
 ret_from_fork_asm+0x1a/0x30

Can we add more information for debugging this?

Quoted 45 lines
Cc Jan Prusakowski <[email protected]>
---
 fs/f2fs/checkpoint.c | 9 ++++++++-
 fs/f2fs/f2fs.h       | 3 +++
 2 files changed, 11 insertions(+), 1 deletion(-)
fs/f2fs/checkpoint.c +8 -1
index db3831f7f2f5..b0dcaa8dc40d 100644
@@ -1788,8 +1788,15 @@ static void __checkpoint_and_complete_reqs(struct f2fs_sb_info *sbi)
 
 	spin_lock(&cprc->stat_lock);
 	cprc->cur_time = (unsigned int)div64_u64(sum_diff, count);
-	if (cprc->peak_time < cprc->cur_time)
+	if (cprc->peak_time < cprc->cur_time) {
 		cprc->peak_time = cprc->cur_time;
+
+		if (unlikely(cprc->peak_time >= CP_LONG_LATENCY_THRESHOLD)) {
+			f2fs_warn_ratelimited(sbi, "checkpoint was blocked for %u ms, affecting %llu tasks",
+					cprc->peak_time, count);
+			dump_stack();
+		}
+	}
 	spin_unlock(&cprc->stat_lock);
 }
 
fs/f2fs/f2fs.h +3 -0
index 46d23c2c086c..3130ca6a4770 100644
@@ -350,6 +350,9 @@ struct ckpt_req_control {
 	unsigned int peak_time;		/* peak wait time in msec until now */
 };
 
+/* a time threshold that checkpoint was blocked for, unit: ms */
+#define CP_LONG_LATENCY_THRESHOLD	5000
+
 /* for the bitmap indicate blocks to be discarded */
 struct discard_entry {
 	struct list_head list;	/* list head */
Signature · 1 line
2.49.0
author maintainer
#2 lore raw
Quoted 22 lines · Jaegeuk Kim
Quoted 11 lines · Chao Yu

generic/299 w/ mode=lfs will cause long time latency, let's dump more information once we hit case.

CP merge:
- Queued : 0
- Issued : 1
- Total : 1
- Cur time : 7565(ms)
- Peak time : 7565(ms)

F2FS-fs (vdc): checkpoint was blocked for 7565 ms, affecting 1 tasks

Quoted 9 lines
CPU: 8 UID: 0 PID: 1614 Comm: f2fs_ckpt-253:3 Tainted: G           O        6.16.0-rc3+ #406 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE
Call Trace:
 dump_stack_lvl+0x6e/0xa0
 __checkpoint_and_complete_reqs+0x1a6/0x1d0
 issue_checkpoint_thread+0x4b/0x140
 kthread+0x10d/0x250
 ret_from_fork+0x164/0x190
 ret_from_fork_asm+0x1a/0x30
Quoted 1 line

Can we add more information for debugging this?

Oops, I need to dump information in user's thread instead of f2fs_ckpt.

Thanks,

Quoted 46 lines
Quoted 46 lines
Cc Jan Prusakowski <[email protected]>
---
 fs/f2fs/checkpoint.c | 9 ++++++++-
 fs/f2fs/f2fs.h       | 3 +++
 2 files changed, 11 insertions(+), 1 deletion(-)
fs/f2fs/checkpoint.c +8 -1
index db3831f7f2f5..b0dcaa8dc40d 100644
@@ -1788,8 +1788,15 @@ static void __checkpoint_and_complete_reqs(struct f2fs_sb_info *sbi)
 
 	spin_lock(&cprc->stat_lock);
 	cprc->cur_time = (unsigned int)div64_u64(sum_diff, count);
-	if (cprc->peak_time < cprc->cur_time)
+	if (cprc->peak_time < cprc->cur_time) {
 		cprc->peak_time = cprc->cur_time;
+
+		if (unlikely(cprc->peak_time >= CP_LONG_LATENCY_THRESHOLD)) {
+			f2fs_warn_ratelimited(sbi, "checkpoint was blocked for %u ms, affecting %llu tasks",
+					cprc->peak_time, count);
+			dump_stack();
+		}
+	}
 	spin_unlock(&cprc->stat_lock);
 }
 
fs/f2fs/f2fs.h +3 -0
index 46d23c2c086c..3130ca6a4770 100644
@@ -350,6 +350,9 @@ struct ckpt_req_control {
 	unsigned int peak_time;		/* peak wait time in msec until now */
 };
 
+/* a time threshold that checkpoint was blocked for, unit: ms */
+#define CP_LONG_LATENCY_THRESHOLD	5000
+
 /* for the bitmap indicate blocks to be discarded */
 struct discard_entry {
 	struct list_head list;	/* list head */
Signature · 1 line
2.49.0
author maintainer
#3 lore raw
In reply to Jaegeuk Kim
Quoted 22 lines · Jaegeuk Kim
Quoted 11 lines · Chao Yu

generic/299 w/ mode=lfs will cause long time latency, let's dump more information once we hit case.

CP merge:
- Queued : 0
- Issued : 1
- Total : 1
- Cur time : 7565(ms)
- Peak time : 7565(ms)

F2FS-fs (vdc): checkpoint was blocked for 7565 ms, affecting 1 tasks

Quoted 9 lines
CPU: 8 UID: 0 PID: 1614 Comm: f2fs_ckpt-253:3 Tainted: G           O        6.16.0-rc3+ #406 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE
Call Trace:
 dump_stack_lvl+0x6e/0xa0
 __checkpoint_and_complete_reqs+0x1a6/0x1d0
 issue_checkpoint_thread+0x4b/0x140
 kthread+0x10d/0x250
 ret_from_fork+0x164/0x190
 ret_from_fork_asm+0x1a/0x30
Quoted 1 line

Can we add more information for debugging this?

I guess we can add more stats in write_checkpoint() to see which stage we cost most of the latency.

Thanks,

Quoted 46 lines
Quoted 46 lines
Cc Jan Prusakowski <[email protected]>
---
 fs/f2fs/checkpoint.c | 9 ++++++++-
 fs/f2fs/f2fs.h       | 3 +++
 2 files changed, 11 insertions(+), 1 deletion(-)
fs/f2fs/checkpoint.c +8 -1
index db3831f7f2f5..b0dcaa8dc40d 100644
@@ -1788,8 +1788,15 @@ static void __checkpoint_and_complete_reqs(struct f2fs_sb_info *sbi)
 
 	spin_lock(&cprc->stat_lock);
 	cprc->cur_time = (unsigned int)div64_u64(sum_diff, count);
-	if (cprc->peak_time < cprc->cur_time)
+	if (cprc->peak_time < cprc->cur_time) {
 		cprc->peak_time = cprc->cur_time;
+
+		if (unlikely(cprc->peak_time >= CP_LONG_LATENCY_THRESHOLD)) {
+			f2fs_warn_ratelimited(sbi, "checkpoint was blocked for %u ms, affecting %llu tasks",
+					cprc->peak_time, count);
+			dump_stack();
+		}
+	}
 	spin_unlock(&cprc->stat_lock);
 }
 
fs/f2fs/f2fs.h +3 -0
index 46d23c2c086c..3130ca6a4770 100644
@@ -350,6 +350,9 @@ struct ckpt_req_control {
 	unsigned int peak_time;		/* peak wait time in msec until now */
 };
 
+/* a time threshold that checkpoint was blocked for, unit: ms */
+#define CP_LONG_LATENCY_THRESHOLD	5000
+
 /* for the bitmap indicate blocks to be discarded */
 struct discard_entry {
 	struct list_head list;	/* list head */
Signature · 1 line
2.49.0