mm, page_alloc: skip ->waternark_boost for atomic order-0 allocations

When boosting is enabled, it is observed that rate of atomic order-0 allocation failures are high due to the fact that free levels in the system are checked with ->watermark_boost offset. This is not a problem for sleepable allocations but for atomic allocations which looks like regression.

This problem is seen frequently on system setup of Android kernel running on Snapdragon hardware with 4GB RAM size. When no extfrag event occurred in the system, ->watermark_boost factor is zero, thus the watermark configurations in the system are:

   _watermark = (
          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB
   watermark_boost = 0

After launching some memory hungry applications in Android which can cause extfrag events in the system to an extent that ->watermark_boost can be set to max i.e. default boost factor makes it to 150% of high watermark.
_watermark = (

          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB

watermark_boost = 14077, -->~57MB

With default system configuration, for an atomic order-0 allocation to succeed, having free memory of ~2MB will suffice. But boosting makes the min_wmark to ~61MB thus for an atomic order-0 allocation to be successful system should have minimum of ~23MB of free memory(from calculations of zone_watermark_ok(), min = 3/4(min/2)). But failures are observed despite system is having ~20MB of free memory. In the testing, this is reproducible as early as first 300secs since boot and with furtherlowram configurations(<2GB) it is observed as early as first 150secs since boot.

These failures can be avoided by excluding the ->watermark_boost in watermark caluculations for atomic order-0 allocations.

mm/page_alloc.c +0 -0

 		if (!zone_watermark_fast(zone, order, mark,
 				       ac->highest_zoneidx, alloc_flags)) {
 			int ret;

Quoted 36 lines · On Tue, 19 May 2020 15:28:04 +0530 Charan Teja Reddy <[email protected]>…

When boosting is enabled, it is observed that rate of atomic order-0 allocation failures are high due to the fact that free levels in the system are checked with ->watermark_boost offset. This is not a problem for sleepable allocations but for atomic allocations which looks like regression.

This problem is seen frequently on system setup of Android kernel running on Snapdragon hardware with 4GB RAM size. When no extfrag event occurred in the system, ->watermark_boost factor is zero, thus the watermark configurations in the system are:

   _watermark = (
          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB
   watermark_boost = 0

After launching some memory hungry applications in Android which can cause extfrag events in the system to an extent that ->watermark_boost can be set to max i.e. default boost factor makes it to 150% of high watermark.
_watermark = (

          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB

watermark_boost = 14077, -->~57MB

With default system configuration, for an atomic order-0 allocation to succeed, having free memory of ~2MB will suffice. But boosting makes the min_wmark to ~61MB thus for an atomic order-0 allocation to be successful system should have minimum of ~23MB of free memory(from calculations of zone_watermark_ok(), min = 3/4(min/2)). But failures are observed despite system is having ~20MB of free memory. In the testing, this is reproducible as early as first 300secs since boot and with furtherlowram configurations(<2GB) it is observed as early as first 150secs since boot.

Quoted 2 lines

These failures can be avoided by excluding the ->watermark_boost in watermark caluculations for atomic order-0 allocations.

Seems sensible.

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3718

Quoted 13 lines

mm/page_alloc.c +7 -0

@@ -3709,6 +3709,18 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
 		}
 
 		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
+		/*
+		 * Allow GFP_ATOMIC order-0 allocations to exclude the
+		 * zone->watermark_boost in its watermark calculations.
+		 * We rely on the ALLOC_ flags set for GFP_ATOMIC
+		 * requests in gfp_to_alloc_flags() for this. Reason not to
+		 * use the GFP_ATOMIC directly is that we want to fall back
+		 * to slow path thus wake up kswapd.

Nice comment, but I don't understand it ;)

Why would testing gfp_mask prevent us from waking kswapd?

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3723

Quoted 6 lines

mm/page_alloc.c +5 -0

+		 */
+		if (unlikely(!order && !(alloc_flags & ALLOC_WMARK_MASK) &&
+		     (alloc_flags & (ALLOC_HARDER | ALLOC_HIGH)))) {
+			mark = zone->_watermark[WMARK_MIN];
+		}

Why is this not implemented for higher-order allocation attempts?

Quoted 4 lines

mm/page_alloc.c +0 -0

 		if (!zone_watermark_fast(zone, order, mark,
 				       ac->highest_zoneidx, alloc_flags)) {
 			int ret;

When boosting is enabled, it is observed that rate of atomic order-0 allocation failures are high due to the fact that free levels in the system are checked with ->watermark_boost offset. This is not a problem for sleepable allocations but for atomic allocations which looks like regression.

This problem is seen frequently on system setup of Android kernel running on Snapdragon hardware with 4GB RAM size. When no extfrag event occurred in the system, ->watermark_boost factor is zero, thus the watermark configurations in the system are:

   _watermark = (
          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB
   watermark_boost = 0

After launching some memory hungry applications in Android which can cause extfrag events in the system to an extent that ->watermark_boost can be set to max i.e. default boost factor makes it to 150% of high watermark.
_watermark = (

          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB

watermark_boost = 14077, -->~57MB

With default system configuration, for an atomic order-0 allocation to succeed, having free memory of ~2MB will suffice. But boosting makes the min_wmark to ~61MB thus for an atomic order-0 allocation to be successful system should have minimum of ~23MB of free memory(from calculations of zone_watermark_ok(), min = 3/4(min/2)). But failures are observed despite system is having ~20MB of free memory. In the testing, this is reproducible as early as first 300secs since boot and with furtherlowram configurations(<2GB) it is observed as early as first 150secs since boot.

These failures can be avoided by excluding the ->watermark_boost in watermark caluculations for atomic order-0 allocations.

mm/page_alloc.c +5 -0

@@ -3709,6 +3709,18 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
 		}
 
 		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
+		/*
+		 * Allow GFP_ATOMIC order-0 allocations to exclude the
+		 * zone->watermark_boost in its watermark calculations.
+		 * We rely on the ALLOC_ flags set for GFP_ATOMIC
+		 * requests in gfp_to_alloc_flags() for this. Reason not to

Quoted 36 lines · On Tue, 19 May 2020 15:28:04 +0530 Charan Teja Reddy <[email protected]>…

When boosting is enabled, it is observed that rate of atomic order-0 allocation failures are high due to the fact that free levels in the system are checked with ->watermark_boost offset. This is not a problem for sleepable allocations but for atomic allocations which looks like regression.

This problem is seen frequently on system setup of Android kernel running on Snapdragon hardware with 4GB RAM size. When no extfrag event occurred in the system, ->watermark_boost factor is zero, thus the watermark configurations in the system are:

   _watermark = (
          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB
   watermark_boost = 0

After launching some memory hungry applications in Android which can cause extfrag events in the system to an extent that ->watermark_boost can be set to max i.e. default boost factor makes it to 150% of high watermark.
_watermark = (

          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB

watermark_boost = 14077, -->~57MB

With default system configuration, for an atomic order-0 allocation to succeed, having free memory of ~2MB will suffice. But boosting makes the min_wmark to ~61MB thus for an atomic order-0 allocation to be successful system should have minimum of ~23MB of free memory(from calculations of zone_watermark_ok(), min = 3/4(min/2)). But failures are observed despite system is having ~20MB of free memory. In the testing, this is reproducible as early as first 300secs since boot and with furtherlowram configurations(<2GB) it is observed as early as first 150secs since boot.

Quoted 2 lines

These failures can be avoided by excluding the ->watermark_boost in watermark caluculations for atomic order-0 allocations.

Do we have any additional reviewer input on this one?

Quoted 11 lines

mm/page_alloc.c +5 -0

@@ -3709,6 +3709,18 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
 		}
 
 		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
+		/*
+		 * Allow GFP_ATOMIC order-0 allocations to exclude the
+		 * zone->watermark_boost in its watermark calculations.
+		 * We rely on the ALLOC_ flags set for GFP_ATOMIC
+		 * requests in gfp_to_alloc_flags() for this. Reason not to

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3726

Quoted 10 lines

mm/page_alloc.c +7 -0

+		 * use the GFP_ATOMIC directly is that we want to fall back
+		 * to slow path thus wake up kswapd.
+		 */
+		if (unlikely(!order && !(alloc_flags & ALLOC_WMARK_MASK) &&
+		     (alloc_flags & (ALLOC_HARDER | ALLOC_HIGH)))) {
+			mark = zone->_watermark[WMARK_MIN];
+		}
 		if (!zone_watermark_fast(zone, order, mark,
 				       ac->highest_zoneidx, alloc_flags)) {
 			int ret;

It would seem smart to do

mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-skip-waternark_boost-for-atomic-order-0-allocations-fix +2 -1

@@ -3745,7 +3745,6 @@ retry:
 			}
 		}
 
-		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
 		/*
 		 * Allow GFP_ATOMIC order-0 allocations to exclude the
 		 * zone->watermark_boost in their watermark calculations.

@@ -3757,6 +3756,8 @@ retry:
 		if (unlikely(!order && !(alloc_flags & ALLOC_WMARK_MASK) &&
 		     (alloc_flags & (ALLOC_HARDER | ALLOC_HIGH)))) {
 			mark = zone->_watermark[WMARK_MIN];
+		} else {
+			mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
 		}
 		if (!zone_watermark_fast(zone, order, mark,
 				       ac->highest_zoneidx, alloc_flags)) {

but that makes page_alloc.o 16 bytes larger, so I guess don't bother.

When boosting is enabled, it is observed that rate of atomic order-0 allocation failures are high due to the fact that free levels in the system are checked with ->watermark_boost offset. This is not a problem for sleepable allocations but for atomic allocations which looks like regression.

This problem is seen frequently on system setup of Android kernel running on Snapdragon hardware with 4GB RAM size. When no extfrag event occurred in the system, ->watermark_boost factor is zero, thus the watermark configurations in the system are:

   _watermark = (
          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB
   watermark_boost = 0

After launching some memory hungry applications in Android which can cause extfrag events in the system to an extent that ->watermark_boost can be set to max i.e. default boost factor makes it to 150% of high watermark.
_watermark = (

          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB

watermark_boost = 14077, -->~57MB

With default system configuration, for an atomic order-0 allocation to succeed, having free memory of ~2MB will suffice. But boosting makes the min_wmark to ~61MB thus for an atomic order-0 allocation to be successful system should have minimum of ~23MB of free memory(from calculations of zone_watermark_ok(), min = 3/4(min/2)). But failures are observed despite system is having ~20MB of free memory. In the testing, this is reproducible as early as first 300secs since boot and with furtherlowram configurations(<2GB) it is observed as early as first 150secs since boot.

These failures can be avoided by excluding the ->watermark_boost in watermark caluculations for atomic order-0 allocations.

Signed-off-by Charan Teja Reddy <[email protected]>

---
 mm/page_alloc.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

mm/page_alloc.c +0 -0

index d001d61..5193d7e 100644
@@ -3709,6 +3709,18 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
 		}

Quoted 6 lines · Charan Teja Reddy

When boosting is enabled, it is observed that rate of atomic order-0 allocation failures are high due to the fact that free levels in the system are checked with ->watermark_boost offset. This is not a problem for sleepable allocations but for atomic allocations which looks like regression.

Are high-order allocations in general of interest to this platform? If not then a potential option is to simply disable boosting. The patch is still relevant but it's worth thinking about.

Quoted 44 lines

This problem is seen frequently on system setup of Android kernel running on Snapdragon hardware with 4GB RAM size. When no extfrag event occurred in the system, ->watermark_boost factor is zero, thus the watermark configurations in the system are:

   _watermark = (
          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB
   watermark_boost = 0

After launching some memory hungry applications in Android which can cause extfrag events in the system to an extent that ->watermark_boost can be set to max i.e. default boost factor makes it to 150% of high watermark.
_watermark = (

          [WMARK_MIN] = 1272, --> ~5MB
          [WMARK_LOW] = 9067, --> ~36MB
          [WMARK_HIGH] = 9385), --> ~38MB

watermark_boost = 14077, -->~57MB

With default system configuration, for an atomic order-0 allocation to succeed, having free memory of ~2MB will suffice. But boosting makes the min_wmark to ~61MB thus for an atomic order-0 allocation to be successful system should have minimum of ~23MB of free memory(from calculations of zone_watermark_ok(), min = 3/4(min/2)). But failures are observed despite system is having ~20MB of free memory. In the testing, this is reproducible as early as first 300secs since boot and with furtherlowram configurations(<2GB) it is observed as early as first 150secs since boot.

These failures can be avoided by excluding the ->watermark_boost in watermark caluculations for atomic order-0 allocations.

Signed-off-by Charan Teja Reddy <[email protected]>

---
 mm/page_alloc.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

mm/page_alloc.c +0 -0

index d001d61..5193d7e 100644
@@ -3709,6 +3709,18 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
 		}

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3719

Quoted 10 lines

mm/page_alloc.c +8 -0

 
 		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
+		/*
+		 * Allow GFP_ATOMIC order-0 allocations to exclude the
+		 * zone->watermark_boost in its watermark calculations.
+		 * We rely on the ALLOC_ flags set for GFP_ATOMIC
+		 * requests in gfp_to_alloc_flags() for this. Reason not to
+		 * use the GFP_ATOMIC directly is that we want to fall back
+		 * to slow path thus wake up kswapd.
+		 */

The comment is a bit difficult to parse. Maybe this.

/*
* Ignore watermark boosting for GFP_ATOMIC order-0 allocations
* when checking the min watermark. The min watermark is the
* point where boosting is ignored so that kswapd is woken up
* when below the low watermark.
*/

I left out the ALLOC_ part for reasons that are explained blow.

mm/page_alloc.c static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) · line 3723

Quoted 4 lines

mm/page_alloc.c +4 -0

+		if (unlikely(!order && !(alloc_flags & ALLOC_WMARK_MASK) &&
+		     (alloc_flags & (ALLOC_HARDER | ALLOC_HIGH)))) {
+			mark = zone->_watermark[WMARK_MIN];
+		}

The second check is a bit more obscure than it needs to be and depends on WMARK_MIN == 0. That will probably be true forever but it's not obvious at a glance. I suggest something like
((alloc_flags & ALLOC_WMARK_MASK) == WMARK_MIN).

For detecting atomic alloctions, you rely on the either ALLOC_HARDER or ALLOC_HIGH being set. ALLOC_HIGH can be set for non-atomic allocations and ALLOC_HARDER can be set for RT tasks. You probably should just test the gfp_mask because as it stands non-atomic allocations can ignore the boost too.

Finally, the patch puts an unlikely check into a relatively fast path even though watermarks may be fine with or without boosting. Instead you could put the checks in zone_watermark_fast() if and only if the watermarks failed the first time. If the checks pass, the watermarks get checked a second time. This will be fractionally slower for requests failing watermark checks but there is no penalty for most allocation requests. It would need the gfp_mask to be passed into zone_watermark_fast but as it's an inlined function, there should be no cost to passing in the arguement i.e. do something like this at the end of zone_watermark_fast

	if (__zone_watermark_ok(z, order, mark, classzone_idx, alloc_flags, free_pages))
		return true;

	/* Ignore watermark boosting for .... */
	if (unlikely(!order .....) {
		mark = ...
		return __zone_watermark_ok(...);
	}

	return false;