Skip to contentSkip to conversation

f2fs: accurately adjust free_sections during free_segment_range

PATCH Superseded Daeho Jeong 3 participants 12 messages Aug 18 – Aug 25 F2FS file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 f2fs: accurately adjust free_sections during free_segment_range

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them under segmap_lock on exit.

---
 fs/f2fs/gc.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
fs/f2fs/gc.c +14 -1
index 787133ee2eb2..f3a6fc6d08ae 100644
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
 static int free_segment_range(struct f2fs_sb_info *sbi,
 				unsigned int secs, bool dry_run)
 {
-	unsigned int next_inuse, start, end;
+	unsigned int secno, next_inuse, start, end, end_secno;
 	struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+	unsigned int freed_secs = 0;
 	int gc_mode, gc_type;
 	int err = 0;
 	int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
 	MAIN_SECS(sbi) -= secs;
 	start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
 	end = MAIN_SEGS(sbi) - 1;
+	end_secno = GET_SEC_FROM_SEG(sbi, end);
 
 	mutex_lock(&DIRTY_I(sbi)->seglist_lock);
 	for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2221,6 +2223,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
 			sbi->next_victim_seg[gc_type] = NULL_SEGNO;
 	mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
 
+	spin_lock(&FREE_I(sbi)->segmap_lock);
+	for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+		if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+			freed_secs++;
+	}
+	FREE_I(sbi)->free_sections -= freed_secs;
+	spin_unlock(&FREE_I(sbi)->segmap_lock);
+
 	/* Move out cursegs from the target range */
 	for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
 		err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2245,6 +2255,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
 		f2fs_bug_on(sbi, 1);
 	}
 out:
+	spin_lock(&FREE_I(sbi)->segmap_lock);
+	FREE_I(sbi)->free_sections += freed_secs;
+	spin_unlock(&FREE_I(sbi)->segmap_lock);
 	MAIN_SECS(sbi) += secs;
 	return err;
 }
Signature · 1 line
2.55.0.691.gc56d675ccc-goog
maintainer
#1 lore raw
Quoted 9 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Can you please show me an example for above case? I didn't get it.

BTW, it needs to rebase this patch on dev-test branch.

Thanks,

Quoted 59 lines

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them under segmap_lock on exit.

---
 fs/f2fs/gc.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
fs/f2fs/gc.c +14 -1
index 787133ee2eb2..f3a6fc6d08ae 100644
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
 static int free_segment_range(struct f2fs_sb_info *sbi,
 				unsigned int secs, bool dry_run)
 {
-	unsigned int next_inuse, start, end;
+	unsigned int secno, next_inuse, start, end, end_secno;
 	struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+	unsigned int freed_secs = 0;
 	int gc_mode, gc_type;
 	int err = 0;
 	int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
 	MAIN_SECS(sbi) -= secs;
 	start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
 	end = MAIN_SEGS(sbi) - 1;
+	end_secno = GET_SEC_FROM_SEG(sbi, end);
 
 	mutex_lock(&DIRTY_I(sbi)->seglist_lock);
 	for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2221,6 +2223,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
 			sbi->next_victim_seg[gc_type] = NULL_SEGNO;
 	mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
 
+	spin_lock(&FREE_I(sbi)->segmap_lock);
+	for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+		if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+			freed_secs++;
+	}
+	FREE_I(sbi)->free_sections -= freed_secs;
+	spin_unlock(&FREE_I(sbi)->segmap_lock);
+
 	/* Move out cursegs from the target range */
 	for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
 		err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2245,6 +2255,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
 		f2fs_bug_on(sbi, 1);
 	}
 out:
+	spin_lock(&FREE_I(sbi)->segmap_lock);
+	FREE_I(sbi)->free_sections += freed_secs;
+	spin_unlock(&FREE_I(sbi)->segmap_lock);
 	MAIN_SECS(sbi) += secs;
 	return err;
 }
#2 lore raw
Quoted 7 lines · Chao Yu
Quoted 5 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

Quoted 6 lines
Quoted 4 lines

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Can you please show me an example for above case? I didn't get it.

Here is a concrete example explaining why this adjustment is needed:

Suppose:
- Total main sections: MAIN_SECS = 100 (sections 0 .. 99)
- Total free sections: free_sections = 30
- We want to shrink the filesystem by 10 sections (secs = 10, range 90 .. 99).
- Within the truncated range (sections 90 .. 99):
* 6 sections are already free (free_secmap bit is 0)
* 4 sections are in-use with valid blocks that need to be migrated by GC.
When free_segment_range() enters:
1. MAIN_SECS is temporarily reduced from 100 to 90 so that new block allocations are constrained to sections 0 .. 89.
2. The actual number of free sections available in the reduced range (0 .. 89) is only 24 (30 - 6 = 24).
3. Without this patch:
- free_sections remains 30 while MAIN_SECS is 90.
- During the subsequent GC migrations, free section checks (such as has_not_enough_free_secs()) will over-estimate available space by 6 sections in the active 0 .. 89 range.
- If free_segment_range() fails midway (e.g. -EAGAIN), free_sections accounting becomes inconsistent.
With this patch:
- We count the 6 already-free sections in the truncated range (90 .. 99) and deduct them from free_sections upon entering (30 - 6 = 24), perfectly matching the actual free sections in the active range 0 .. 89.
- On exit, the deducted amount is restored, keeping free_sections consistent throughout the entire resize lifecycle.

Hope this clarifies the scenario!

Thanks,

Quoted 65 lines

BTW, it needs to rebase this patch on dev-test branch.

Thanks,

Quoted 59 lines

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them under segmap_lock on exit.

---
 fs/f2fs/gc.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
fs/f2fs/gc.c +14 -1
index 787133ee2eb2..f3a6fc6d08ae 100644
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
 static int free_segment_range(struct f2fs_sb_info *sbi,
                              unsigned int secs, bool dry_run)
 {
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
      struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
      int gc_mode, gc_type;
      int err = 0;
      int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
      MAIN_SECS(sbi) -= secs;
      start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
      end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
      mutex_lock(&DIRTY_I(sbi)->seglist_lock);
      for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2221,6 +2223,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
                      sbi->next_victim_seg[gc_type] = NULL_SEGNO;
      mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
 
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+
      /* Move out cursegs from the target range */
      for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
              err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2245,6 +2255,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
              f2fs_bug_on(sbi, 1);
      }
 out:
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
      MAIN_SECS(sbi) += secs;
      return err;
 }
maintainer
#3 lore raw
Quoted 8 lines · Daeho Jeong
Quoted 7 lines · Chao Yu
Quoted 5 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Can you please show me an example for above case? I didn't get it.

Here is a concrete example explaining why this adjustment is needed:

Thanks for the detailed explanation.

Quoted 12 lines

Suppose:
- Total main sections: MAIN_SECS = 100 (sections 0 .. 99)
- Total free sections: free_sections = 30
- We want to shrink the filesystem by 10 sections (secs = 10, range 90 .. 99).
- Within the truncated range (sections 90 .. 99):
* 6 sections are already free (free_secmap bit is 0)
* 4 sections are in-use with valid blocks that need to be migrated by GC.
When free_segment_range() enters:
1. MAIN_SECS is temporarily reduced from 100 to 90 so that new block allocations are constrained to sections 0 .. 89.
2. The actual number of free sections available in the reduced range

Quoted 5 lines
    (0 .. 89) is only 24 (30 - 6 = 24).

3. Without this patch:
- free_sections remains 30 while MAIN_SECS is 90.
- During the subsequent GC migrations, free section checks (such as has_not_enough_free_secs()) will over-estimate available space by 6

But free_segment_range() won't call into has_not_enough_free_secs(), if I'm not missing anything.

Not sure, maybe you mean other threads will call has_not_enough_free_secs(), are you worried about that we may miss chances to call fggc earlier in below cases:

f2fs_balance_fs -> has_enough_free_secs -> f2fs_gc? I guess it will blocked on gc_lock.

Quoted 3 lines

sections in the active 0 .. 89 range.
- If free_segment_range() fails midway (e.g. -EAGAIN), free_sections accounting becomes inconsistent.

Why free_sections accounting becomes inconsistent if free_segment_range() fails midway?

Thanks,

Quoted 76 lines

With this patch:
- We count the 6 already-free sections in the truncated range (90 .. 99) and deduct them from free_sections upon entering (30 - 6 = 24), perfectly matching the actual free sections in the active range 0 .. 89.
- On exit, the deducted amount is restored, keeping free_sections consistent throughout the entire resize lifecycle.

Hope this clarifies the scenario!

Thanks,

Quoted 65 lines

BTW, it needs to rebase this patch on dev-test branch.

Thanks,

Quoted 59 lines

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them under segmap_lock on exit.

---
  fs/f2fs/gc.c | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)
fs/f2fs/gc.c +14 -1
index 787133ee2eb2..f3a6fc6d08ae 100644
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
  static int free_segment_range(struct f2fs_sb_info *sbi,
                               unsigned int secs, bool dry_run)
  {
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
       struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
       int gc_mode, gc_type;
       int err = 0;
       int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
       MAIN_SECS(sbi) -= secs;
       start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
       end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
       mutex_lock(&DIRTY_I(sbi)->seglist_lock);
       for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2221,6 +2223,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
                       sbi->next_victim_seg[gc_type] = NULL_SEGNO;
       mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
 
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+
       /* Move out cursegs from the target range */
       for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
               err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2245,6 +2255,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
               f2fs_bug_on(sbi, 1);
       }
  out:
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
       MAIN_SECS(sbi) += secs;
       return err;
  }
#4 lore raw
Quoted 39 lines · Chao Yu
Quoted 8 lines · Daeho Jeong
Quoted 7 lines · Chao Yu
Quoted 5 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Can you please show me an example for above case? I didn't get it.

Here is a concrete example explaining why this adjustment is needed:

Thanks for the detailed explanation.

Quoted 12 lines

Suppose:
- Total main sections: MAIN_SECS = 100 (sections 0 .. 99)
- Total free sections: free_sections = 30
- We want to shrink the filesystem by 10 sections (secs = 10, range 90 .. 99).
- Within the truncated range (sections 90 .. 99):
* 6 sections are already free (free_secmap bit is 0)
* 4 sections are in-use with valid blocks that need to be migrated by GC.
When free_segment_range() enters:
1. MAIN_SECS is temporarily reduced from 100 to 90 so that new block allocations are constrained to sections 0 .. 89.
2. The actual number of free sections available in the reduced range

Quoted 5 lines
    (0 .. 89) is only 24 (30 - 6 = 24).

3. Without this patch:
- free_sections remains 30 while MAIN_SECS is 90.
- During the subsequent GC migrations, free section checks (such as has_not_enough_free_secs()) will over-estimate available space by 6

Quoted 2 lines

But free_segment_range() won't call into has_not_enough_free_secs(), if I'm not missing anything.

Oh, right. I missed it's FG_GC.
I think we can drop this patch.

Thanks.

Quoted 94 lines

Not sure, maybe you mean other threads will call has_not_enough_free_secs(), are you worried about that we may miss chances to call fggc earlier in below cases:

f2fs_balance_fs -> has_enough_free_secs -> f2fs_gc? I guess it will blocked on gc_lock.

Quoted 3 lines

sections in the active 0 .. 89 range.
- If free_segment_range() fails midway (e.g. -EAGAIN), free_sections accounting becomes inconsistent.

Why free_sections accounting becomes inconsistent if free_segment_range() fails midway?

Thanks,

Quoted 76 lines

With this patch:
- We count the 6 already-free sections in the truncated range (90 .. 99) and deduct them from free_sections upon entering (30 - 6 = 24), perfectly matching the actual free sections in the active range 0 .. 89.
- On exit, the deducted amount is restored, keeping free_sections consistent throughout the entire resize lifecycle.

Hope this clarifies the scenario!

Thanks,

Quoted 65 lines

BTW, it needs to rebase this patch on dev-test branch.

Thanks,

Quoted 59 lines

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them under segmap_lock on exit.

---
  fs/f2fs/gc.c | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)
fs/f2fs/gc.c +14 -1
index 787133ee2eb2..f3a6fc6d08ae 100644
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
  static int free_segment_range(struct f2fs_sb_info *sbi,
                               unsigned int secs, bool dry_run)
  {
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
       struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
       int gc_mode, gc_type;
       int err = 0;
       int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
       MAIN_SECS(sbi) -= secs;
       start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
       end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
       mutex_lock(&DIRTY_I(sbi)->seglist_lock);
       for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2221,6 +2223,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
                       sbi->next_victim_seg[gc_type] = NULL_SEGNO;
       mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
 
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+
       /* Move out cursegs from the target range */
       for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
               err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2245,6 +2255,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
               f2fs_bug_on(sbi, 1);
       }
  out:
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
       MAIN_SECS(sbi) += secs;
       return err;
  }
#5 lore raw
Quoted 43 lines · Chao Yu
Quoted 15 lines · Daeho Jeong
Quoted 13 lines · Chao Yu
Quoted 12 lines · Daeho Jeong
Quoted 9 lines

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Quoted 11 lines
Quoted 3 lines
Quoted 1 line

Can you please show me an example for above case? I didn't get it.

Here is a concrete example explaining why this adjustment is needed:

Thanks for the detailed explanation.

Quoted 5 lines

Suppose:
- Total main sections: MAIN_SECS = 100 (sections 0 .. 99)
- Total free sections: free_sections = 30
- We want to shrink the filesystem by 10 sections (secs = 10, range

90 .. 99).

Quoted 3 lines
Quoted 3 lines

- Within the truncated range (sections 90 .. 99):
* 6 sections are already free (free_secmap bit is 0)
* 4 sections are in-use with valid blocks that need to be migrated

by GC.

Quoted 11 lines
Quoted 10 lines

When free_segment_range() enters:
1. MAIN_SECS is temporarily reduced from 100 to 90 so that new block allocations are constrained to sections 0 .. 89.
2. The actual number of free sections available in the reduced range

    (0 .. 89) is only 24 (30 - 6 = 24).

3. Without this patch:
- free_sections remains 30 while MAIN_SECS is 90.
- During the subsequent GC migrations, free section checks (such as has_not_enough_free_secs()) will over-estimate available space
by 6

Quoted 8 lines
Quoted 2 lines

But free_segment_range() won't call into has_not_enough_free_secs(), if I'm not missing anything.

Oh, right. I missed it's FG_GC.
I think we can drop this patch.

Thanks.

I wonder if the free-section adjustment could still be relevant to the SSR/LFS allocation decision during resize.

Could you please also check whether free_sections may affect allocation through the following paths?

free_segment_range()

    -> f2fs_allocate_segment_for_resize()
      -> f2fs_need_SSR()
      -> new_curseg()
        -> get_new_segment()

Also, when a current segment becomes full during block migration:

free_segment_range()

    -> f2fs_gc_range()
      -> do_garbage_collect()
        -> f2fs_allocate_data_block()
          -> need_new_seg()
            -> f2fs_need_SSR()
            -> new_curseg()
              -> get_new_segment()

If free_sections still include free sections in the range being removed, f2fs_need_SSR() may choose LFS allocation instead of SSR. Since get_new_segment() searches only within the temporarily reduced MAIN_SECS range, it may fail to find a free section.

Could you please confirm whether this case also needs to be handled?

Thanks,

Quoted 109 lines
Quoted 3 lines

Not sure, maybe you mean other threads will call has_not_enough_free_secs(), are you worried about that we may miss

chances to call fggc earlier in below cases:

Quoted 7 lines

f2fs_balance_fs -> has_enough_free_secs -> f2fs_gc? I guess it will blocked on gc_lock.

Quoted 2 lines

sections in the active 0 .. 89 range.
- If free_segment_range() fails midway (e.g. -EAGAIN),

free_sections

Quoted 9 lines
Quoted 1 line

accounting becomes inconsistent.

Why free_sections accounting becomes inconsistent if free_segment_range() fails midway?

Thanks,

Quoted 2 lines

With this patch:
- We count the 6 already-free sections in the truncated range (90 ..

99) and

Quoted 1 line
Quoted 1 line

deduct them from free_sections upon entering (30 - 6 = 24),

perfectly

Quoted 2 lines
Quoted 2 lines

matching the actual free sections in the active range 0 .. 89.
- On exit, the deducted amount is restored, keeping free_sections

consistent

Quoted 7 lines
Quoted 7 lines

throughout the entire resize lifecycle.

Hope this clarifies the scenario!

Thanks,

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

BTW, it needs to rebase this patch on dev-test branch.

Thanks,

Quoted 4 lines

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them

under segmap_lock on exit.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
---
  fs/f2fs/gc.c | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
fs/f2fs/gc.c index renamed from fs/f2fs/gc.c +0 -0

787133ee2eb2..f3a6fc6d08ae 100644

fs/f2fs/gc.c +0 -0
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
  static int free_segment_range(struct f2fs_sb_info *sbi,
                               unsigned int secs, bool dry_run)
  {
Quoted 8 lines
Quoted 8 lines

fs/f2fs/gc.c static int free_segment_range(struct

Quoted 8 lines
fs/f2fs/gc.c +2 -1
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
       struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
       int gc_mode, gc_type;
       int err = 0;
       int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct

f2fs_sb_info *sbi,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi,

Quoted 8 lines
fs/f2fs/gc.c +1 -1
       MAIN_SECS(sbi) -= secs;
       start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
       end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
       mutex_lock(&DIRTY_I(sbi)->seglist_lock);
       for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++) @@
-2221,6 +2223,14 @@ static int free_segment_range(struct

f2fs_sb_info *sbi,

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
                       sbi->next_victim_seg[gc_type] = NULL_SEGNO;
       mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
fs/f2fs/gc.c +8 -0
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+
       /* Move out cursegs from the target range */
       for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
               err = f2fs_allocate_segment_for_resize(sbi, type,
Quoted 1 line

start, end); @@ -2245,6 +2255,9 @@ static int

free_segment_range(struct f2fs_sb_info *sbi,

Quoted 11 lines
Quoted 10 lines
Quoted 10 lines

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi, · line 2263

Quoted 9 lines
               f2fs_bug_on(sbi, 1);
       }

out:

fs/f2fs/gc.c +3 -0
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
       MAIN_SECS(sbi) += secs;
       return err;
  }

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#6 lore raw
Quoted 75 lines · Yeongjin Gil
Quoted 44 lines · Chao Yu
Quoted 15 lines · Daeho Jeong
Quoted 13 lines · Chao Yu
Quoted 12 lines · Daeho Jeong
Quoted 9 lines

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Quoted 11 lines
Quoted 3 lines
Quoted 1 line

Can you please show me an example for above case? I didn't get it.

Here is a concrete example explaining why this adjustment is needed:

Thanks for the detailed explanation.

Quoted 5 lines

Suppose:
- Total main sections: MAIN_SECS = 100 (sections 0 .. 99)
- Total free sections: free_sections = 30
- We want to shrink the filesystem by 10 sections (secs = 10, range

90 .. 99).

Quoted 3 lines
Quoted 3 lines

- Within the truncated range (sections 90 .. 99):
* 6 sections are already free (free_secmap bit is 0)
* 4 sections are in-use with valid blocks that need to be migrated

by GC.

Quoted 11 lines
Quoted 10 lines

When free_segment_range() enters:
1. MAIN_SECS is temporarily reduced from 100 to 90 so that new block allocations are constrained to sections 0 .. 89.
2. The actual number of free sections available in the reduced range

    (0 .. 89) is only 24 (30 - 6 = 24).

3. Without this patch:
- free_sections remains 30 while MAIN_SECS is 90.
- During the subsequent GC migrations, free section checks (such as has_not_enough_free_secs()) will over-estimate available space
by 6

Quoted 8 lines
Quoted 2 lines

But free_segment_range() won't call into has_not_enough_free_secs(), if I'm not missing anything.

Oh, right. I missed it's FG_GC.
I think we can drop this patch.

Thanks.

I wonder if the free-section adjustment could still be relevant to the SSR/LFS allocation decision during resize.

Could you please also check whether free_sections may affect allocation through the following paths?

free_segment_range()

    -> f2fs_allocate_segment_for_resize()
      -> f2fs_need_SSR()
      -> new_curseg()
        -> get_new_segment()

Also, when a current segment becomes full during block migration:

free_segment_range()

    -> f2fs_gc_range()
      -> do_garbage_collect()
        -> f2fs_allocate_data_block()
          -> need_new_seg()
            -> f2fs_need_SSR()
            -> new_curseg()
              -> get_new_segment()
Quoted 6 lines

If free_sections still include free sections in the range being removed, f2fs_need_SSR() may choose LFS allocation instead of SSR. Since get_new_segment() searches only within the temporarily reduced MAIN_SECS range, it may fail to find a free section.

Could you please confirm whether this case also needs to be handled?

Thank you for pointing this out — this is a very sharp and valid observation!

It makes sense to me. Chao, WDYT?

Quoted 113 lines

Thanks,

Quoted 109 lines
Quoted 3 lines

Not sure, maybe you mean other threads will call has_not_enough_free_secs(), are you worried about that we may miss

chances to call fggc earlier in below cases:

Quoted 7 lines

f2fs_balance_fs -> has_enough_free_secs -> f2fs_gc? I guess it will blocked on gc_lock.

Quoted 2 lines

sections in the active 0 .. 89 range.
- If free_segment_range() fails midway (e.g. -EAGAIN),

free_sections

Quoted 9 lines
Quoted 1 line

accounting becomes inconsistent.

Why free_sections accounting becomes inconsistent if free_segment_range() fails midway?

Thanks,

Quoted 2 lines

With this patch:
- We count the 6 already-free sections in the truncated range (90 ..

99) and

Quoted 1 line
Quoted 1 line

deduct them from free_sections upon entering (30 - 6 = 24),

perfectly

Quoted 2 lines
Quoted 2 lines

matching the actual free sections in the active range 0 .. 89.
- On exit, the deducted amount is restored, keeping free_sections

consistent

Quoted 7 lines
Quoted 7 lines

throughout the entire resize lifecycle.

Hope this clarifies the scenario!

Thanks,

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

BTW, it needs to rebase this patch on dev-test branch.

Thanks,

Quoted 4 lines

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them

under segmap_lock on exit.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
---
  fs/f2fs/gc.c | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
fs/f2fs/gc.c index renamed from fs/f2fs/gc.c +0 -0

787133ee2eb2..f3a6fc6d08ae 100644

fs/f2fs/gc.c +0 -0
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
  static int free_segment_range(struct f2fs_sb_info *sbi,
                               unsigned int secs, bool dry_run)
  {
Quoted 8 lines
Quoted 8 lines

fs/f2fs/gc.c static int free_segment_range(struct

Quoted 8 lines
fs/f2fs/gc.c +2 -1
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
       struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
       int gc_mode, gc_type;
       int err = 0;
       int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct

f2fs_sb_info *sbi,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi,

Quoted 8 lines
fs/f2fs/gc.c +1 -1
       MAIN_SECS(sbi) -= secs;
       start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
       end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
       mutex_lock(&DIRTY_I(sbi)->seglist_lock);
       for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++) @@
-2221,6 +2223,14 @@ static int free_segment_range(struct

f2fs_sb_info *sbi,

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
                       sbi->next_victim_seg[gc_type] = NULL_SEGNO;
       mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
fs/f2fs/gc.c +8 -0
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+
       /* Move out cursegs from the target range */
       for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
               err = f2fs_allocate_segment_for_resize(sbi, type,
Quoted 1 line

start, end); @@ -2245,6 +2255,9 @@ static int

free_segment_range(struct f2fs_sb_info *sbi,

Quoted 11 lines
Quoted 10 lines
Quoted 10 lines

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi, · line 2263

Quoted 9 lines
               f2fs_bug_on(sbi, 1);
       }

out:

fs/f2fs/gc.c +3 -0
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
       MAIN_SECS(sbi) += secs;
       return err;
  }

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

maintainer
#7 lore raw
Quoted 81 lines · Daeho Jeong
Quoted 80 lines · Yeongjin Gil
Quoted 44 lines · Chao Yu
Quoted 15 lines · Daeho Jeong
Quoted 13 lines · Chao Yu
Quoted 12 lines · Daeho Jeong
Quoted 9 lines

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Quoted 11 lines
Quoted 3 lines
Quoted 1 line

Can you please show me an example for above case? I didn't get it.

Here is a concrete example explaining why this adjustment is needed:

Thanks for the detailed explanation.

Quoted 5 lines

Suppose:
- Total main sections: MAIN_SECS = 100 (sections 0 .. 99)
- Total free sections: free_sections = 30
- We want to shrink the filesystem by 10 sections (secs = 10, range

90 .. 99).

Quoted 3 lines
Quoted 3 lines

- Within the truncated range (sections 90 .. 99):
* 6 sections are already free (free_secmap bit is 0)
* 4 sections are in-use with valid blocks that need to be migrated

by GC.

Quoted 11 lines
Quoted 10 lines

When free_segment_range() enters:
1. MAIN_SECS is temporarily reduced from 100 to 90 so that new block allocations are constrained to sections 0 .. 89.
2. The actual number of free sections available in the reduced range

     (0 .. 89) is only 24 (30 - 6 = 24).

3. Without this patch:
- free_sections remains 30 while MAIN_SECS is 90.
- During the subsequent GC migrations, free section checks (such as has_not_enough_free_secs()) will over-estimate available space
by 6

Quoted 8 lines
Quoted 2 lines

But free_segment_range() won't call into has_not_enough_free_secs(), if I'm not missing anything.

Oh, right. I missed it's FG_GC.
I think we can drop this patch.

Thanks.

I wonder if the free-section adjustment could still be relevant to the SSR/LFS allocation decision during resize.

Could you please also check whether free_sections may affect allocation through the following paths?

   free_segment_range()
     -> f2fs_allocate_segment_for_resize()
       -> f2fs_need_SSR()
       -> new_curseg()
         -> get_new_segment()

Also, when a current segment becomes full during block migration:

   free_segment_range()
     -> f2fs_gc_range()
       -> do_garbage_collect()
         -> f2fs_allocate_data_block()
           -> need_new_seg()
             -> f2fs_need_SSR()
             -> new_curseg()
               -> get_new_segment()

If free_sections still include free sections in the range being removed, f2fs_need_SSR() may choose LFS allocation instead of SSR. Since get_new_segment() searches only within the temporarily reduced MAIN_SECS range, it may fail to find a free section.

Quoted 5 lines
Quoted 1 line

Could you please confirm whether this case also needs to be handled?

Thank you for pointing this out — this is a very sharp and valid observation!

It makes sense to me. Chao, WDYT?

Daeho, yeah, I think it's a good catch from Yeongjin.

I suspect this is a bug, right?

For example, f2fs has high BDF value:
0..85 are dirty sections, and has few valid blocks in each section 86..89 are free sections
90..99 are full sections

When we migrate 90..99 to 0..89, if SSR can not used for such case, get_new_segment() may return -ENOSPC, and trigger the panic?

Thanks,

Quoted 114 lines
Quoted 114 lines

Thanks,

Quoted 109 lines
Quoted 3 lines

Not sure, maybe you mean other threads will call has_not_enough_free_secs(), are you worried about that we may miss

chances to call fggc earlier in below cases:

Quoted 7 lines

f2fs_balance_fs -> has_enough_free_secs -> f2fs_gc? I guess it will blocked on gc_lock.

Quoted 2 lines

sections in the active 0 .. 89 range.
- If free_segment_range() fails midway (e.g. -EAGAIN),

free_sections

Quoted 9 lines
Quoted 1 line

accounting becomes inconsistent.

Why free_sections accounting becomes inconsistent if free_segment_range() fails midway?

Thanks,

Quoted 2 lines

With this patch:
- We count the 6 already-free sections in the truncated range (90 ..

99) and

Quoted 1 line
Quoted 1 line

deduct them from free_sections upon entering (30 - 6 = 24),

perfectly

Quoted 2 lines
Quoted 2 lines

matching the actual free sections in the active range 0 .. 89.
- On exit, the deducted amount is restored, keeping free_sections

consistent

Quoted 7 lines
Quoted 7 lines

throughout the entire resize lifecycle.

Hope this clarifies the scenario!

Thanks,

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

BTW, it needs to rebase this patch on dev-test branch.

Thanks,

Quoted 4 lines

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them

under segmap_lock on exit.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
---
   fs/f2fs/gc.c | 15 ++++++++++++++-
   1 file changed, 14 insertions(+), 1 deletion(-)
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
fs/f2fs/gc.c index renamed from fs/f2fs/gc.c +0 -0

787133ee2eb2..f3a6fc6d08ae 100644

fs/f2fs/gc.c +0 -0
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
   static int free_segment_range(struct f2fs_sb_info *sbi,
                                unsigned int secs, bool dry_run)
   {
Quoted 8 lines
Quoted 8 lines

fs/f2fs/gc.c static int free_segment_range(struct

Quoted 8 lines
fs/f2fs/gc.c +2 -1
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
        struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
        int gc_mode, gc_type;
        int err = 0;
        int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct

f2fs_sb_info *sbi,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

fs/f2fs/gc.c static int free_segment_range(struct

Quoted 8 lines
fs/f2fs/gc.c +1 -1
        MAIN_SECS(sbi) -= secs;
        start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
        end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
        mutex_lock(&DIRTY_I(sbi)->seglist_lock);
        for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++) @@
-2221,6 +2223,14 @@ static int free_segment_range(struct

f2fs_sb_info *sbi,

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
                        sbi->next_victim_seg[gc_type] = NULL_SEGNO;
        mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
fs/f2fs/gc.c +8 -0
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+
        /* Move out cursegs from the target range */
        for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
                err = f2fs_allocate_segment_for_resize(sbi, type,
Quoted 1 line

start, end); @@ -2245,6 +2255,9 @@ static int

free_segment_range(struct f2fs_sb_info *sbi,

Quoted 11 lines
Quoted 10 lines
Quoted 10 lines

fs/f2fs/gc.c

Quoted 9 lines
                f2fs_bug_on(sbi, 1);
        }

out:

fs/f2fs/gc.c +3 -0
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
        MAIN_SECS(sbi) += secs;
        return err;
   }

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#8 lore raw
Quoted 93 lines · Chao Yu
Quoted 81 lines · Daeho Jeong
Quoted 80 lines · Yeongjin Gil
Quoted 44 lines · Chao Yu
Quoted 15 lines · Daeho Jeong
Quoted 13 lines · Chao Yu
Quoted 12 lines · Daeho Jeong
Quoted 9 lines

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Quoted 11 lines
Quoted 3 lines
Quoted 1 line

Can you please show me an example for above case? I didn't get it.

Here is a concrete example explaining why this adjustment is needed:

Thanks for the detailed explanation.

Quoted 5 lines

Suppose:
- Total main sections: MAIN_SECS = 100 (sections 0 .. 99)
- Total free sections: free_sections = 30
- We want to shrink the filesystem by 10 sections (secs = 10, range

90 .. 99).

Quoted 3 lines
Quoted 3 lines

- Within the truncated range (sections 90 .. 99):
* 6 sections are already free (free_secmap bit is 0)
* 4 sections are in-use with valid blocks that need to be migrated

by GC.

Quoted 11 lines
Quoted 10 lines

When free_segment_range() enters:
1. MAIN_SECS is temporarily reduced from 100 to 90 so that new block allocations are constrained to sections 0 .. 89.
2. The actual number of free sections available in the reduced range

     (0 .. 89) is only 24 (30 - 6 = 24).

3. Without this patch:
- free_sections remains 30 while MAIN_SECS is 90.
- During the subsequent GC migrations, free section checks (such as has_not_enough_free_secs()) will over-estimate available space
by 6

Quoted 8 lines
Quoted 2 lines

But free_segment_range() won't call into has_not_enough_free_secs(), if I'm not missing anything.

Oh, right. I missed it's FG_GC.
I think we can drop this patch.

Thanks.

I wonder if the free-section adjustment could still be relevant to the SSR/LFS allocation decision during resize.

Could you please also check whether free_sections may affect allocation through the following paths?

   free_segment_range()
     -> f2fs_allocate_segment_for_resize()
       -> f2fs_need_SSR()
       -> new_curseg()
         -> get_new_segment()

Also, when a current segment becomes full during block migration:

   free_segment_range()
     -> f2fs_gc_range()
       -> do_garbage_collect()
         -> f2fs_allocate_data_block()
           -> need_new_seg()
             -> f2fs_need_SSR()
             -> new_curseg()
               -> get_new_segment()

If free_sections still include free sections in the range being removed, f2fs_need_SSR() may choose LFS allocation instead of SSR. Since get_new_segment() searches only within the temporarily reduced MAIN_SECS range, it may fail to find a free section.

Quoted 5 lines
Quoted 1 line

Could you please confirm whether this case also needs to be handled?

Thank you for pointing this out — this is a very sharp and valid observation!

It makes sense to me. Chao, WDYT?

Daeho, yeah, I think it's a good catch from Yeongjin.

I suspect this is a bug, right?

Quoted 7 lines

For example, f2fs has high BDF value:
0..85 are dirty sections, and has few valid blocks in each section 86..89 are free sections
90..99 are full sections

When we migrate 90..99 to 0..89, if SSR can not used for such case, get_new_segment() may return -ENOSPC, and trigger the panic?

Yes, so I think this patch is still valid for the case..

Thanks,

Quoted 118 lines

Thanks,

Quoted 114 lines
Quoted 114 lines

Thanks,

Quoted 109 lines
Quoted 3 lines

Not sure, maybe you mean other threads will call has_not_enough_free_secs(), are you worried about that we may miss

chances to call fggc earlier in below cases:

Quoted 7 lines

f2fs_balance_fs -> has_enough_free_secs -> f2fs_gc? I guess it will blocked on gc_lock.

Quoted 2 lines

sections in the active 0 .. 89 range.
- If free_segment_range() fails midway (e.g. -EAGAIN),

free_sections

Quoted 9 lines
Quoted 1 line

accounting becomes inconsistent.

Why free_sections accounting becomes inconsistent if free_segment_range() fails midway?

Thanks,

Quoted 2 lines

With this patch:
- We count the 6 already-free sections in the truncated range (90 ..

99) and

Quoted 1 line
Quoted 1 line

deduct them from free_sections upon entering (30 - 6 = 24),

perfectly

Quoted 2 lines
Quoted 2 lines

matching the actual free sections in the active range 0 .. 89.
- On exit, the deducted amount is restored, keeping free_sections

consistent

Quoted 7 lines
Quoted 7 lines

throughout the entire resize lifecycle.

Hope this clarifies the scenario!

Thanks,

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

BTW, it needs to rebase this patch on dev-test branch.

Thanks,

Quoted 4 lines

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them

under segmap_lock on exit.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
---
   fs/f2fs/gc.c | 15 ++++++++++++++-
   1 file changed, 14 insertions(+), 1 deletion(-)
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
fs/f2fs/gc.c index renamed from fs/f2fs/gc.c +0 -0

787133ee2eb2..f3a6fc6d08ae 100644

fs/f2fs/gc.c +0 -0
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
   static int free_segment_range(struct f2fs_sb_info *sbi,
                                unsigned int secs, bool dry_run)
   {
Quoted 8 lines
Quoted 8 lines

fs/f2fs/gc.c static int free_segment_range(struct

Quoted 8 lines
fs/f2fs/gc.c +2 -1
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
        struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
        int gc_mode, gc_type;
        int err = 0;
        int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct

f2fs_sb_info *sbi,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

fs/f2fs/gc.c static int free_segment_range(struct

Quoted 8 lines
fs/f2fs/gc.c +1 -1
        MAIN_SECS(sbi) -= secs;
        start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
        end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
        mutex_lock(&DIRTY_I(sbi)->seglist_lock);
        for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++) @@
-2221,6 +2223,14 @@ static int free_segment_range(struct

f2fs_sb_info *sbi,

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
                        sbi->next_victim_seg[gc_type] = NULL_SEGNO;
        mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
fs/f2fs/gc.c +8 -0
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+
        /* Move out cursegs from the target range */
        for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
                err = f2fs_allocate_segment_for_resize(sbi, type,
Quoted 1 line

start, end); @@ -2245,6 +2255,9 @@ static int

free_segment_range(struct f2fs_sb_info *sbi,

Quoted 11 lines
Quoted 10 lines
Quoted 10 lines

fs/f2fs/gc.c

Quoted 9 lines
                f2fs_bug_on(sbi, 1);
        }

out:

fs/f2fs/gc.c +3 -0
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
        MAIN_SECS(sbi) += secs;
        return err;
   }

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

maintainer
#9 lore raw
In reply to Daeho Jeong
Quoted 5 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

Quoted 8 lines

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them under segmap_lock on exit.

I think we need to change commit message a bit to reflect real issues?

Fixes and Cc stable line here.

Quoted 32 lines
---
  fs/f2fs/gc.c | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)
fs/f2fs/gc.c +3 -1
index 787133ee2eb2..f3a6fc6d08ae 100644
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
  static int free_segment_range(struct f2fs_sb_info *sbi,
  				unsigned int secs, bool dry_run)
  {
-	unsigned int next_inuse, start, end;
+	unsigned int secno, next_inuse, start, end, end_secno;
  	struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+	unsigned int freed_secs = 0;
  	int gc_mode, gc_type;
  	int err = 0;
  	int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
  	MAIN_SECS(sbi) -= secs;
  	start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
  	end = MAIN_SEGS(sbi) - 1;
+	end_secno = GET_SEC_FROM_SEG(sbi, end);
 
  	mutex_lock(&DIRTY_I(sbi)->seglist_lock);
  	for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2221,6 +2223,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
  			sbi->next_victim_seg[gc_type] = NULL_SEGNO;

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi, · line 2233

Quoted 10 lines
fs/f2fs/gc.c +8 -0
  	mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
 
+	spin_lock(&FREE_I(sbi)->segmap_lock);
+	for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+		if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+			freed_secs++;
+	}
+	FREE_I(sbi)->free_sections -= freed_secs;
+	spin_unlock(&FREE_I(sbi)->segmap_lock);
+

Before f2fs_gc_range
0..89 have 10 dirty/full sections, 80 free sections. 90..99 have 5 dirty sections, 5 free sections.

FREE_I(sbi)->free_sections -= freed_secs; free_sections = 85 - 5 = 80

After f2fs_gc_range
0..89 have 13 dirty/full sections, 77 free sections. 90..99 have 10 free sections.
free_sections = 77 (.free_sections) + 10 (secs) = 87

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi, · line 2259

Quoted 9 lines
fs/f2fs/gc.c +2 -0
  	/* Move out cursegs from the target range */
  	for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
  		err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2245,6 +2255,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
  		f2fs_bug_on(sbi, 1);
  	}
  out:
+	spin_lock(&FREE_I(sbi)->segmap_lock);
+	FREE_I(sbi)->free_sections += freed_secs;

So, shouldn't be?
FREE_I(sbi)->free_sections += secs;

Thanks,

Quoted 4 lines
fs/f2fs/gc.c +1 -0
+	spin_unlock(&FREE_I(sbi)->segmap_lock);
  	MAIN_SECS(sbi) += secs;
  	return err;
  }
#10 lore raw
Quoted 75 lines · Chao Yu
Quoted 5 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

Quoted 8 lines

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them under segmap_lock on exit.

I think we need to change commit message a bit to reflect real issues?

Fixes and Cc stable line here.

Quoted 32 lines
---
  fs/f2fs/gc.c | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)
fs/f2fs/gc.c +3 -1
index 787133ee2eb2..f3a6fc6d08ae 100644
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
  static int free_segment_range(struct f2fs_sb_info *sbi,
                              unsigned int secs, bool dry_run)
  {
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
      struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
      int gc_mode, gc_type;
      int err = 0;
      int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
      MAIN_SECS(sbi) -= secs;
      start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
      end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
      mutex_lock(&DIRTY_I(sbi)->seglist_lock);
      for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2221,6 +2223,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
                      sbi->next_victim_seg[gc_type] = NULL_SEGNO;

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi, · line 2233

Quoted 10 lines
fs/f2fs/gc.c +8 -0
      mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
 
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+

Before f2fs_gc_range
0..89 have 10 dirty/full sections, 80 free sections. 90..99 have 5 dirty sections, 5 free sections.

FREE_I(sbi)->free_sections -= freed_secs; free_sections = 85 - 5 = 80

After f2fs_gc_range
0..89 have 13 dirty/full sections, 77 free sections. 90..99 have 10 free sections.
free_sections = 77 (.free_sections) + 10 (secs) = 87

Quoted 12 lines

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi, · line 2259

Quoted 9 lines
fs/f2fs/gc.c +2 -0
      /* Move out cursegs from the target range */
      for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
              err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2245,6 +2255,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
              f2fs_bug_on(sbi, 1);
      }
  out:
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;

So, shouldn't be?
FREE_I(sbi)->free_sections += secs;

During f2fs_gc_range(), when the dirty sections in 90..99 are cleaned and freed, __set_test_and_free() is invoked for those sections, which automatically increments FREE_I(sbi)->free_sections for each newly freed section?

Quoted 8 lines

Thanks,

Quoted 4 lines
fs/f2fs/gc.c +1 -0
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
      MAIN_SECS(sbi) += secs;
      return err;
  }
maintainer
#11 lore raw
Quoted 89 lines · Daeho Jeong
Quoted 75 lines · Chao Yu
Quoted 5 lines · Daeho Jeong

From: Daeho Jeong <[email protected]>

In free_segment_range(), MAIN_SECS(sbi) is temporarily reduced by secs while valid blocks in the truncated range are evacuated by GC.

Quoted 8 lines

However, if any sections within the truncated range were already free, failing to deduct them from FREE_I(sbi)->free_sections leads to an over-estimation of available space in the reduced main area, causing inconsistent free section accounting.

Fix this by calculating the number of already-free sections in the truncated range under segmap_lock, deducting them from free_sections upon entering free_segment_range(), and restoring them under segmap_lock on exit.

I think we need to change commit message a bit to reflect real issues?

Fixes and Cc stable line here.

Quoted 32 lines
---
   fs/f2fs/gc.c | 15 ++++++++++++++-
   1 file changed, 14 insertions(+), 1 deletion(-)
fs/f2fs/gc.c +3 -1
index 787133ee2eb2..f3a6fc6d08ae 100644
@@ -2200,8 +2200,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
   static int free_segment_range(struct f2fs_sb_info *sbi,
                               unsigned int secs, bool dry_run)
   {
-     unsigned int next_inuse, start, end;
+     unsigned int secno, next_inuse, start, end, end_secno;
       struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+     unsigned int freed_secs = 0;
       int gc_mode, gc_type;
       int err = 0;
       int type;
@@ -2210,6 +2211,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
       MAIN_SECS(sbi) -= secs;
       start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
       end = MAIN_SEGS(sbi) - 1;
+     end_secno = GET_SEC_FROM_SEG(sbi, end);
 
       mutex_lock(&DIRTY_I(sbi)->seglist_lock);
       for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2221,6 +2223,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
                       sbi->next_victim_seg[gc_type] = NULL_SEGNO;

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi, · line 2233

Quoted 10 lines
fs/f2fs/gc.c +8 -0
       mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
 
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+             if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+                     freed_secs++;
+     }
+     FREE_I(sbi)->free_sections -= freed_secs;
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
+

Before f2fs_gc_range
0..89 have 10 dirty/full sections, 80 free sections. 90..99 have 5 dirty sections, 5 free sections.

FREE_I(sbi)->free_sections -= freed_secs; free_sections = 85 - 5 = 80

After f2fs_gc_range
0..89 have 13 dirty/full sections, 77 free sections. 90..99 have 10 free sections.
free_sections = 77 (.free_sections) + 10 (secs) = 87

Quoted 12 lines

fs/f2fs/gc.c static int free_segment_range(struct f2fs_sb_info *sbi, · line 2259

Quoted 9 lines
fs/f2fs/gc.c +2 -0
       /* Move out cursegs from the target range */
       for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
               err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2245,6 +2255,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
               f2fs_bug_on(sbi, 1);
       }
   out:
+     spin_lock(&FREE_I(sbi)->segmap_lock);
+     FREE_I(sbi)->free_sections += freed_secs;

So, shouldn't be?
FREE_I(sbi)->free_sections += secs;

Quoted 4 lines

During f2fs_gc_range(), when the dirty sections in 90..99 are cleaned and freed, __set_test_and_free() is invoked for those sections, which automatically increments FREE_I(sbi)->free_sections for each newly freed section?

Ah, I see, thanks for correcting me. :)

Thanks,

Quoted 9 lines
Quoted 9 lines

Thanks,

Quoted 4 lines
fs/f2fs/gc.c +1 -0
+     spin_unlock(&FREE_I(sbi)->segmap_lock);
       MAIN_SECS(sbi) += secs;
       return err;
   }