Skip to contentSkip to conversation

mm/page-writeback: Fix performance when BDI's share of ratio is 0.

PATCH Applied Chi Wu 3 participants 6 messages Apr 28, 2021 – May 10, 2021

Subsystems (2)
lore
Share & tools
lore

Chi Wu

author
#0 lore raw

PATCH 1/1 mm/page-writeback: Fix performance when BDI's share of ratio is 0.

Fix performance when BDI's share of ratio is 0.

The issue is similar to commit 74d369443325 ("writeback: Fix performance regression in wb_over_bg_thresh()").

Balance_dirty_pages and the writeback worker will also disagree on whether writeback when a BDI uses BDI_CAP_STRICTLIMIT and BDI's share of the thresh ratio is zero.

For example, A thread on cpu0 writes 32 pages and then balance_dirty_pages, it will wake up background writeback and pauses because wb_dirty > wb->wb_thresh = 0 (share of thresh ratio is zero). A thread may runs on cpu0 again because scheduler prefers pre_cpu. Then writeback worker may runs on other cpus(1,2..) which causes the value of wb_stat(wb, WB_RECLAIMABLE) in wb_over_bg_thresh is 0 and does not writeback and returns.

Thus, balance_dirty_pages keeps looping, sleeping and then waking up the worker who will do nothing. It remains stuck in this state until the writeback worker hit the right dirty cpu or the dirty pages expire.

The fix that we should get the wb_stat_sum radically when thresh is low.

---
 mm/page-writeback.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
mm/page-writeback.c +16 -4
index 0062d5c57d41..bd7052295246 100644
@@ -1945,6 +1945,8 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 	struct dirty_throttle_control * const gdtc = &gdtc_stor;
 	struct dirty_throttle_control * const mdtc = mdtc_valid(&mdtc_stor) ?
 						     &mdtc_stor : NULL;
+	unsigned long reclaimable;
+	unsigned long thresh;
 
 	/*
 	 * Similar to balance_dirty_pages() but ignores pages being written
@@ -1957,8 +1959,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 	if (gdtc->dirty > gdtc->bg_thresh)
 		return true;
 
-	if (wb_stat(wb, WB_RECLAIMABLE) >
-	    wb_calc_thresh(gdtc->wb, gdtc->bg_thresh))
+	thresh = wb_calc_thresh(gdtc->wb, gdtc->bg_thresh);
+	if (thresh < 2 * wb_stat_error())
+		reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+	else
+		reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+	if (reclaimable > thresh)
 		return true;
 
 	if (mdtc) {
@@ -1972,8 +1979,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 		if (mdtc->dirty > mdtc->bg_thresh)
 			return true;
 
-		if (wb_stat(wb, WB_RECLAIMABLE) >
-		    wb_calc_thresh(mdtc->wb, mdtc->bg_thresh))
+		thresh = wb_calc_thresh(mdtc->wb, mdtc->bg_thresh);
+		if (thresh < 2 * wb_stat_error())
+			reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+		else
+			reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+		if (reclaimable > thresh)
 			return true;
 	}
 
Signature · 1 line
2.17.1
author
#1 lore raw

Ping...

Quoted 78 lines · Chi Wu <[email protected]> 于2021年4月29日周四 上午6:51写道:

Fix performance when BDI's share of ratio is 0.

The issue is similar to commit 74d369443325 ("writeback: Fix performance regression in wb_over_bg_thresh()").

Balance_dirty_pages and the writeback worker will also disagree on whether writeback when a BDI uses BDI_CAP_STRICTLIMIT and BDI's share of the thresh ratio is zero.

For example, A thread on cpu0 writes 32 pages and then balance_dirty_pages, it will wake up background writeback and pauses because wb_dirty > wb->wb_thresh = 0 (share of thresh ratio is zero). A thread may runs on cpu0 again because scheduler prefers pre_cpu. Then writeback worker may runs on other cpus(1,2..) which causes the value of wb_stat(wb, WB_RECLAIMABLE) in wb_over_bg_thresh is 0 and does not writeback and returns.

Thus, balance_dirty_pages keeps looping, sleeping and then waking up the worker who will do nothing. It remains stuck in this state until the writeback worker hit the right dirty cpu or the dirty pages expire.

The fix that we should get the wb_stat_sum radically when thresh is low.

---
 mm/page-writeback.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
mm/page-writeback.c +0 -0
index 0062d5c57d41..bd7052295246 100644
@@ -1945,6 +1945,8 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
        struct dirty_throttle_control * const gdtc = &gdtc_stor;
        struct dirty_throttle_control * const mdtc =

mdtc_valid(&mdtc_stor) ?

                                                     &mdtc_stor : NULL;
mm/page-writeback.c +16 -5
+       unsigned long reclaimable;
+       unsigned long thresh;
 
        /*
         * Similar to balance_dirty_pages() but ignores pages being written
@@ -1957,8 +1959,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
        if (gdtc->dirty > gdtc->bg_thresh)
                return true;
 
-       if (wb_stat(wb, WB_RECLAIMABLE) >
-           wb_calc_thresh(gdtc->wb, gdtc->bg_thresh))
+       thresh = wb_calc_thresh(gdtc->wb, gdtc->bg_thresh);
+       if (thresh < 2 * wb_stat_error())
+               reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+       else
+               reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+       if (reclaimable > thresh)
                return true;
 
        if (mdtc) {
@@ -1972,8 +1979,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
                if (mdtc->dirty > mdtc->bg_thresh)
                        return true;
 
-               if (wb_stat(wb, WB_RECLAIMABLE) >
-                   wb_calc_thresh(mdtc->wb, mdtc->bg_thresh))
+               thresh = wb_calc_thresh(mdtc->wb, mdtc->bg_thresh);
+               if (thresh < 2 * wb_stat_error())
+                       reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+               else
+                       reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+               if (reclaimable > thresh)
                        return true;
        }
 
--

2.17.1

author
#2 lore raw
In reply to Chi Wu

Ping...

Quoted 77 lines · Chi Wu <[email protected]> 于2021年4月29日周四 上午6:51写道:

Fix performance when BDI's share of ratio is 0.

The issue is similar to commit 74d369443325 ("writeback: Fix performance regression in wb_over_bg_thresh()").

Balance_dirty_pages and the writeback worker will also disagree on whether writeback when a BDI uses BDI_CAP_STRICTLIMIT and BDI's share of the thresh ratio is zero.

For example, A thread on cpu0 writes 32 pages and then balance_dirty_pages, it will wake up background writeback and pauses because wb_dirty > wb->wb_thresh = 0 (share of thresh ratio is zero). A thread may runs on cpu0 again because scheduler prefers pre_cpu. Then writeback worker may runs on other cpus(1,2..) which causes the value of wb_stat(wb, WB_RECLAIMABLE) in wb_over_bg_thresh is 0 and does not writeback and returns.

Thus, balance_dirty_pages keeps looping, sleeping and then waking up the worker who will do nothing. It remains stuck in this state until the writeback worker hit the right dirty cpu or the dirty pages expire.

The fix that we should get the wb_stat_sum radically when thresh is low.

---
 mm/page-writeback.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
mm/page-writeback.c +16 -5
index 0062d5c57d41..bd7052295246 100644
@@ -1945,6 +1945,8 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
        struct dirty_throttle_control * const gdtc = &gdtc_stor;
        struct dirty_throttle_control * const mdtc = mdtc_valid(&mdtc_stor) ?
                                                     &mdtc_stor : NULL;
+       unsigned long reclaimable;
+       unsigned long thresh;
 
        /*
         * Similar to balance_dirty_pages() but ignores pages being written
@@ -1957,8 +1959,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
        if (gdtc->dirty > gdtc->bg_thresh)
                return true;
 
-       if (wb_stat(wb, WB_RECLAIMABLE) >
-           wb_calc_thresh(gdtc->wb, gdtc->bg_thresh))
+       thresh = wb_calc_thresh(gdtc->wb, gdtc->bg_thresh);
+       if (thresh < 2 * wb_stat_error())
+               reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+       else
+               reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+       if (reclaimable > thresh)
                return true;
 
        if (mdtc) {
@@ -1972,8 +1979,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
                if (mdtc->dirty > mdtc->bg_thresh)
                        return true;
 
-               if (wb_stat(wb, WB_RECLAIMABLE) >
-                   wb_calc_thresh(mdtc->wb, mdtc->bg_thresh))
+               thresh = wb_calc_thresh(mdtc->wb, mdtc->bg_thresh);
+               if (thresh < 2 * wb_stat_error())
+                       reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+               else
+                       reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+               if (reclaimable > thresh)
                        return true;
        }
 
--

2.17.1

maintainer
#3 lore raw
In reply to Chi Wu
Quoted 17 lines · On Thu, 29 Apr 2021 06:50:46 +0800 Chi Wu <[email protected]> wrote:

Fix performance when BDI's share of ratio is 0.

The issue is similar to commit 74d369443325 ("writeback: Fix performance regression in wb_over_bg_thresh()").

Balance_dirty_pages and the writeback worker will also disagree on whether writeback when a BDI uses BDI_CAP_STRICTLIMIT and BDI's share of the thresh ratio is zero.

For example, A thread on cpu0 writes 32 pages and then balance_dirty_pages, it will wake up background writeback and pauses because wb_dirty > wb->wb_thresh = 0 (share of thresh ratio is zero). A thread may runs on cpu0 again because scheduler prefers pre_cpu. Then writeback worker may runs on other cpus(1,2..) which causes the value of wb_stat(wb, WB_RECLAIMABLE) in wb_over_bg_thresh is 0 and does not writeback and returns.

Quoted 5 lines

Thus, balance_dirty_pages keeps looping, sleeping and then waking up the worker who will do nothing. It remains stuck in this state until the writeback worker hit the right dirty cpu or the dirty pages expire.

The fix that we should get the wb_stat_sum radically when thresh is low.

(optimistically Cc's various people who might remember how this code works)

Quoted 1 line

Thanks. I'll add it for some testing and hopefully someone will find the time to review this.

Quoted 51 lines
---
 mm/page-writeback.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
mm/page-writeback.c +16 -4
index 0062d5c57d41..bd7052295246 100644
@@ -1945,6 +1945,8 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 	struct dirty_throttle_control * const gdtc = &gdtc_stor;
 	struct dirty_throttle_control * const mdtc = mdtc_valid(&mdtc_stor) ?
 						     &mdtc_stor : NULL;
+	unsigned long reclaimable;
+	unsigned long thresh;
 
 	/*
 	 * Similar to balance_dirty_pages() but ignores pages being written
@@ -1957,8 +1959,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 	if (gdtc->dirty > gdtc->bg_thresh)
 		return true;
 
-	if (wb_stat(wb, WB_RECLAIMABLE) >
-	    wb_calc_thresh(gdtc->wb, gdtc->bg_thresh))
+	thresh = wb_calc_thresh(gdtc->wb, gdtc->bg_thresh);
+	if (thresh < 2 * wb_stat_error())
+		reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+	else
+		reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+	if (reclaimable > thresh)
 		return true;
 
 	if (mdtc) {
@@ -1972,8 +1979,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 		if (mdtc->dirty > mdtc->bg_thresh)
 			return true;
 
-		if (wb_stat(wb, WB_RECLAIMABLE) >
-		    wb_calc_thresh(mdtc->wb, mdtc->bg_thresh))
+		thresh = wb_calc_thresh(mdtc->wb, mdtc->bg_thresh);
+		if (thresh < 2 * wb_stat_error())
+			reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+		else
+			reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+		if (reclaimable > thresh)
 			return true;
 	}
 
Signature · 1 line
2.17.1
author
#4 lore raw
Quoted 20 lines · Andrew Morton <[email protected]> 于2021年5月10日周一 上午7:36写道:
Quoted 17 lines · On Thu, 29 Apr 2021 06:50:46 +0800 Chi Wu <[email protected]> wrote:

Fix performance when BDI's share of ratio is 0.

The issue is similar to commit 74d369443325 ("writeback: Fix performance regression in wb_over_bg_thresh()").

Balance_dirty_pages and the writeback worker will also disagree on whether writeback when a BDI uses BDI_CAP_STRICTLIMIT and BDI's share of the thresh ratio is zero.

For example, A thread on cpu0 writes 32 pages and then balance_dirty_pages, it will wake up background writeback and pauses because wb_dirty > wb->wb_thresh = 0 (share of thresh ratio is zero). A thread may runs on cpu0 again because scheduler prefers pre_cpu. Then writeback worker may runs on other cpus(1,2..) which causes the value of wb_stat(wb, WB_RECLAIMABLE) in wb_over_bg_thresh is 0 and does not writeback and returns.

Quoted 7 lines
Quoted 5 lines

Thus, balance_dirty_pages keeps looping, sleeping and then waking up the worker who will do nothing. It remains stuck in this state until the writeback worker hit the right dirty cpu or the dirty pages expire.

The fix that we should get the wb_stat_sum radically when thresh is low.

(optimistically Cc's various people who might remember how this code works)

Thanks, I'll correct it in the future.

Quoted 57 lines
Quoted 2 lines

Thanks. I'll add it for some testing and hopefully someone will find the time to review this.

Quoted 51 lines
---
 mm/page-writeback.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
mm/page-writeback.c +16 -5
index 0062d5c57d41..bd7052295246 100644
@@ -1945,6 +1945,8 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
      struct dirty_throttle_control * const gdtc = &gdtc_stor;
      struct dirty_throttle_control * const mdtc = mdtc_valid(&mdtc_stor) ?
                                                   &mdtc_stor : NULL;
+     unsigned long reclaimable;
+     unsigned long thresh;
 
      /*
       * Similar to balance_dirty_pages() but ignores pages being written
@@ -1957,8 +1959,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
      if (gdtc->dirty > gdtc->bg_thresh)
              return true;
 
-     if (wb_stat(wb, WB_RECLAIMABLE) >
-         wb_calc_thresh(gdtc->wb, gdtc->bg_thresh))
+     thresh = wb_calc_thresh(gdtc->wb, gdtc->bg_thresh);
+     if (thresh < 2 * wb_stat_error())
+             reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+     else
+             reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+     if (reclaimable > thresh)
              return true;
 
      if (mdtc) {
@@ -1972,8 +1979,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
              if (mdtc->dirty > mdtc->bg_thresh)
                      return true;
 
-             if (wb_stat(wb, WB_RECLAIMABLE) >
-                 wb_calc_thresh(mdtc->wb, mdtc->bg_thresh))
+             thresh = wb_calc_thresh(mdtc->wb, mdtc->bg_thresh);
+             if (thresh < 2 * wb_stat_error())
+                     reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+             else
+                     reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+             if (reclaimable > thresh)
                      return true;
      }
 
--

2.17.1

maintainer Reviewed-by
#5 lore raw
In reply to Andrew Morton
Quoted 19 lines · Andrew Morton
Quoted 17 lines · On Thu, 29 Apr 2021 06:50:46 +0800 Chi Wu <[email protected]> wrote:

Fix performance when BDI's share of ratio is 0.

The issue is similar to commit 74d369443325 ("writeback: Fix performance regression in wb_over_bg_thresh()").

Balance_dirty_pages and the writeback worker will also disagree on whether writeback when a BDI uses BDI_CAP_STRICTLIMIT and BDI's share of the thresh ratio is zero.

For example, A thread on cpu0 writes 32 pages and then balance_dirty_pages, it will wake up background writeback and pauses because wb_dirty > wb->wb_thresh = 0 (share of thresh ratio is zero). A thread may runs on cpu0 again because scheduler prefers pre_cpu. Then writeback worker may runs on other cpus(1,2..) which causes the value of wb_stat(wb, WB_RECLAIMABLE) in wb_over_bg_thresh is 0 and does not writeback and returns.

Quoted 7 lines
Quoted 5 lines

Thus, balance_dirty_pages keeps looping, sleeping and then waking up the worker who will do nothing. It remains stuck in this state until the writeback worker hit the right dirty cpu or the dirty pages expire.

The fix that we should get the wb_stat_sum radically when thresh is low.

(optimistically Cc's various people who might remember how this code works)

Thanks for forwarding Andrew!

Quoted 4 lines
Quoted 1 line

Thanks. I'll add it for some testing and hopefully someone will find the time to review this.

Thanks for the patch! It looks good to me, good catch! Feel free to add:

Honza

Quoted 51 lines
Quoted 51 lines
---
 mm/page-writeback.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
mm/page-writeback.c +16 -4
index 0062d5c57d41..bd7052295246 100644
@@ -1945,6 +1945,8 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 	struct dirty_throttle_control * const gdtc = &gdtc_stor;
 	struct dirty_throttle_control * const mdtc = mdtc_valid(&mdtc_stor) ?
 						     &mdtc_stor : NULL;
+	unsigned long reclaimable;
+	unsigned long thresh;
 
 	/*
 	 * Similar to balance_dirty_pages() but ignores pages being written
@@ -1957,8 +1959,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 	if (gdtc->dirty > gdtc->bg_thresh)
 		return true;
 
-	if (wb_stat(wb, WB_RECLAIMABLE) >
-	    wb_calc_thresh(gdtc->wb, gdtc->bg_thresh))
+	thresh = wb_calc_thresh(gdtc->wb, gdtc->bg_thresh);
+	if (thresh < 2 * wb_stat_error())
+		reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+	else
+		reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+	if (reclaimable > thresh)
 		return true;
 
 	if (mdtc) {
@@ -1972,8 +1979,13 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
 		if (mdtc->dirty > mdtc->bg_thresh)
 			return true;
 
-		if (wb_stat(wb, WB_RECLAIMABLE) >
-		    wb_calc_thresh(mdtc->wb, mdtc->bg_thresh))
+		thresh = wb_calc_thresh(mdtc->wb, mdtc->bg_thresh);
+		if (thresh < 2 * wb_stat_error())
+			reclaimable = wb_stat_sum(wb, WB_RECLAIMABLE);
+		else
+			reclaimable = wb_stat(wb, WB_RECLAIMABLE);
+
+		if (reclaimable > thresh)
 			return true;
 	}
 
Signature · 1 line
2.17.1
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR