Skip to contentSkip to conversation

mm/page_alloc: increase default min_free_kbytes bound

PATCH Applied Joel Savitz 5 participants 5 messages Feb 20, 2020 – Feb 27, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/page_alloc: increase default min_free_kbytes bound

Currently, the vm.min_free_kbytes sysctl value is capped at a hardcoded 64M in init_per_zone_wmark_min (unless it is overridden by khugepaged initialization).

This value has not been modified since 2005, and enterprise-grade systems now frequently have hundreds of GB of RAM and multiple 10, 40, or even 100 GB NICs. We have seen page allocation failures on heavily loaded systems related to NIC drivers. These issues were resolved by an increase to vm.min_free_kbytes.

This patch increases the hardcoded value by a factor of 4 as a temporary solution.

Further work to make the calculation of vm.min_free_kbytes more consistent throughout the kernel would be desirable.

As an example of the inconsistency of the current method, this value is recalculated by init_per_zone_wmark_min() in the case of memory hotplug which will override the value set by set_recommended_min_free_kbytes() called during khugepaged initialization even if khugepaged remains enabled, however an on/off toggle of khugepaged will then recalculate and set the value via set_recommended_min_free_kbytes().

---
 mm/page_alloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/page_alloc.c +2 -2
index 3c4eb750a199..32cbfb13e958 100644
@@ -7867,8 +7867,8 @@ int __meminit init_per_zone_wmark_min(void)
 		min_free_kbytes = new_min_free_kbytes;
 		if (min_free_kbytes < 128)
 			min_free_kbytes = 128;
-		if (min_free_kbytes > 65536)
-			min_free_kbytes = 65536;
+		if (min_free_kbytes > 262144)
+			min_free_kbytes = 262144;
 	} else {
 		pr_warn("min_free_kbytes is not updated to %d because user defined value %d is preferred\n",
 				new_min_free_kbytes, user_min_free_kbytes);
Signature · 1 line
2.23.0
maintainer
#1 lore raw
Quoted 5 lines · On Thu, 20 Feb 2020 10:01:03 -0500 Joel Savitz <[email protected]> wrote:

Currently, the vm.min_free_kbytes sysctl value is capped at a hardcoded 64M in init_per_zone_wmark_min (unless it is overridden by khugepaged initialization).

Quoted 8 lines

This value has not been modified since 2005, and enterprise-grade systems now frequently have hundreds of GB of RAM and multiple 10, 40, or even 100 GB NICs. We have seen page allocation failures on heavily loaded systems related to NIC drivers. These issues were resolved by an increase to vm.min_free_kbytes.

This patch increases the hardcoded value by a factor of 4 as a temporary solution.

OK, better than nothing I guess.

Quoted 10 lines

Further work to make the calculation of vm.min_free_kbytes more consistent throughout the kernel would be desirable.

As an example of the inconsistency of the current method, this value is recalculated by init_per_zone_wmark_min() in the case of memory hotplug which will override the value set by set_recommended_min_free_kbytes() called during khugepaged initialization even if khugepaged remains enabled, however an on/off toggle of khugepaged will then recalculate and set the value via set_recommended_min_free_kbytes().

Yup, these hardcoded numbers are lame.

#2 lore raw
In reply to Joel Savitz
Quoted 10 lines · Joel Savitz

Further work to make the calculation of vm.min_free_kbytes more consistent throughout the kernel would be desirable.

As an example of the inconsistency of the current method, this value is recalculated by init_per_zone_wmark_min() in the case of memory hotplug which will override the value set by set_recommended_min_free_kbytes() called during khugepaged initialization even if khugepaged remains enabled, however an on/off toggle of khugepaged will then recalculate and set the value via set_recommended_min_free_kbytes().

I took a shot at fixing some of those inconsistencies.

https://lore.kernel.org/linux-mm/[email protected]/

Signature · 1 line
Mike Kravetz
#3 lore raw
In reply to Joel Savitz
Quoted 34 lines · Joel Savitz

Currently, the vm.min_free_kbytes sysctl value is capped at a hardcoded 64M in init_per_zone_wmark_min (unless it is overridden by khugepaged initialization).

This value has not been modified since 2005, and enterprise-grade systems now frequently have hundreds of GB of RAM and multiple 10, 40, or even 100 GB NICs. We have seen page allocation failures on heavily loaded systems related to NIC drivers. These issues were resolved by an increase to vm.min_free_kbytes.

This patch increases the hardcoded value by a factor of 4 as a temporary solution.

Further work to make the calculation of vm.min_free_kbytes more consistent throughout the kernel would be desirable.

As an example of the inconsistency of the current method, this value is recalculated by init_per_zone_wmark_min() in the case of memory hotplug which will override the value set by set_recommended_min_free_kbytes() called during khugepaged initialization even if khugepaged remains enabled, however an on/off toggle of khugepaged will then recalculate and set the value via set_recommended_min_free_kbytes().

---
 mm/page_alloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/page_alloc.c +0 -0
index 3c4eb750a199..32cbfb13e958 100644

mm/page_alloc.c int __meminit init_per_zone_wmark_min(void) · line 7871

Quoted 7 lines
mm/page_alloc.c +2 -2
@@ -7867,8 +7867,8 @@ int __meminit init_per_zone_wmark_min(void)
 		min_free_kbytes = new_min_free_kbytes;
 		if (min_free_kbytes < 128)
 			min_free_kbytes = 128;
-		if (min_free_kbytes > 65536)
-			min_free_kbytes = 65536;
+		if (min_free_kbytes > 262144)
+			min_free_kbytes = 262144;

Would it be any better to at least use symbols, instead of numbers, in the routine? Like this:

mm/page_alloc.c +7 -4
index 3c4eb750a199..e705636bb644 100644
@@ -149,6 +149,9 @@ DEFINE_STATIC_KEY_FALSE(init_on_free);
 #endif
 EXPORT_SYMBOL(init_on_free);
 
+static const int MIN_FREE_KBYTES_LOWER_LIMIT = 128;
+static const int MIN_FREE_KBYTES_UPPER_LIMIT = 262144;
+
 static int __init early_init_on_alloc(char *buf)
 {
        int ret;
@@ -7865,10 +7868,10 @@ int __meminit init_per_zone_wmark_min(void)
 
        if (new_min_free_kbytes > user_min_free_kbytes) {
                min_free_kbytes = new_min_free_kbytes;
-               if (min_free_kbytes < 128)
-                       min_free_kbytes = 128;
-               if (min_free_kbytes > 65536)
-                       min_free_kbytes = 65536;
+               if (min_free_kbytes < MIN_FREE_KBYTES_LOWER_LIMIT)
+                       min_free_kbytes = MIN_FREE_KBYTES_LOWER_LIMIT;
+               if (min_free_kbytes > MIN_FREE_KBYTES_UPPER_LIMIT)
+                       min_free_kbytes = MIN_FREE_KBYTES_UPPER_LIMIT;
        } else {
                pr_warn("min_free_kbytes is not updated to %d because user defined value %d is preferred\n",
                                new_min_free_kbytes, user_min_free_kbytes);

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
Quoted 4 lines
mm/page_alloc.c +0 -0
 	} else {
 		pr_warn("min_free_kbytes is not updated to %d because user defined value %d is preferred\n",
 				new_min_free_kbytes, user_min_free_kbytes);
#4 lore raw
Quoted 30 lines · John Hubbard
Quoted 29 lines · Joel Savitz

Currently, the vm.min_free_kbytes sysctl value is capped at a hardcoded 64M in init_per_zone_wmark_min (unless it is overridden by khugepaged initialization).

This value has not been modified since 2005, and enterprise-grade systems now frequently have hundreds of GB of RAM and multiple 10, 40, or even 100 GB NICs. We have seen page allocation failures on heavily loaded systems related to NIC drivers. These issues were resolved by an increase to vm.min_free_kbytes.

This patch increases the hardcoded value by a factor of 4 as a temporary solution.

Further work to make the calculation of vm.min_free_kbytes more consistent throughout the kernel would be desirable.

As an example of the inconsistency of the current method, this value is recalculated by init_per_zone_wmark_min() in the case of memory hotplug which will override the value set by set_recommended_min_free_kbytes() called during khugepaged initialization even if khugepaged remains enabled, however an on/off toggle of khugepaged will then recalculate and set the value via set_recommended_min_free_kbytes().

---
 mm/page_alloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 16 lines

mm/page_alloc.c int __meminit init_per_zone_wmark_min(void) · line 7871

Quoted 12 lines
mm/page_alloc.c +2 -2
index 3c4eb750a199..32cbfb13e958 100644
@@ -7867,8 +7867,8 @@ int __meminit init_per_zone_wmark_min(void)
 		min_free_kbytes = new_min_free_kbytes;
 		if (min_free_kbytes < 128)
 			min_free_kbytes = 128;
-		if (min_free_kbytes > 65536)
-			min_free_kbytes = 65536;
+		if (min_free_kbytes > 262144)
+			min_free_kbytes = 262144;

Would it be any better to at least use symbols, instead of numbers, in the routine? Like this:

+1

Thanks