mm: override mTHP "enabled" defaults at kernel cmdline
Subsystems (4)
PATCH v5 1/1 mm: override mTHP "enabled" defaults at kernel cmdline
From: Ryan Roberts <[email protected]>
Add thp_anon= cmdline parameter to allow specifying the default enablement of each supported anon THP size. The parameter accepts the following format and can be provided multiple times to configure each size:
thp_anon=<size>,<size>[KMG]:<value>;<size>-<size>[KMG]:<value>
An example:
thp_anon=16K-64K:always;128K,512K:inherit;256K:madvise;1M-2M:never
See Documentation/admin-guide/mm/transhuge.rst for more details.
Configuring the defaults at boot time is useful to allow early user space to take advantage of mTHP before its been configured through sysfs.
---
-v5:
* collect Baolin's reviewed-by and tested-by tags, thanks!
* use get_order and check size is power 2, David, Baolin;
* use proper __initdata
.../admin-guide/kernel-parameters.txt | 9 ++
Documentation/admin-guide/mm/transhuge.rst | 38 +++++-- mm/huge_memory.c | 100 +++++++++++++++++- 3 files changed, 139 insertions(+), 8 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +9 -0
@@ -6629,6 +6629,15 @@ <deci-seconds>: poll all this frequency 0: no polling (default) + thp_anon= [KNL] + Format: <size>,<size>[KMG]:<state>;<size>-<size>[KMG]:<state> + state is one of "always", "madvise", "never" or "inherit". + Can be used to control the default behavior of the + system with respect to anonymous transparent hugepages. + Can be used multiple times for multiple anon THP sizes. + See Documentation/admin-guide/mm/transhuge.rst for more + details. + threadirqs [KNL,EARLY] Force threading of all interrupt handlers except those marked explicitly IRQF_NO_THREAD.
Documentation/admin-guide/mm/transhuge.rst +31 -7
@@ -284,13 +284,37 @@ that THP is shared. Exceeding the number would block the collapse:: A higher value may increase memory footprint for some workloads. -Boot parameter -============== - -You can change the sysfs boot time defaults of Transparent Hugepage -Support by passing the parameter ``transparent_hugepage=always`` or -``transparent_hugepage=madvise`` or ``transparent_hugepage=never`` -to the kernel command line. +Boot parameters +=============== + +You can change the sysfs boot time default for the top-level "enabled" +control by passing the parameter ``transparent_hugepage=always`` or +``transparent_hugepage=madvise`` or ``transparent_hugepage=never`` to the +kernel command line. + +Alternatively, each supported anonymous THP size can be controlled by +passing ``thp_anon=<size>,<size>[KMG]:<state>;<size>-<size>[KMG]:<state>``, +where ``<size>`` is the THP size (must be a power of 2 of PAGE_SIZE and +supported anonymous THP) and ``<state>`` is one of ``always``, ``madvise``, +``never`` or ``inherit``. + +For example, the following will set 16K, 32K, 64K THP to ``always``, +set 128K, 512K to ``inherit``, set 256K to ``madvise`` and 1M, 2M +to ``never``:: + + thp_anon=16K-64K:always;128K,512K:inherit;256K:madvise;1M-2M:never + +``thp_anon=`` may be specified multiple times to configure all THP sizes as +required. If ``thp_anon=`` is specified at least once, any anon THP sizes +not explicitly configured on the command line are implicitly set to +``never``. + +``transparent_hugepage`` setting only affects the global toggle. If +``thp_anon`` is not specified, PMD_ORDER THP will default to ``inherit``. +However, if a valid ``thp_anon`` setting is provided by the user, the +PMD_ORDER THP policy will be overridden. If the policy for PMD_ORDER +is not defined within a valid ``thp_anon``, its policy will default to +``never``. Hugepages in tmpfs/shmem ========================
mm/huge_memory.c +99 -1
118 lines not on this page · raw message
Signature · 1 line
2.39.3 (Apple Git-146)
Quoted 18 lines · Barry Song
From: Ryan Roberts <[email protected]>
Add thp_anon= cmdline parameter to allow specifying the default enablement of each supported anon THP size. The parameter accepts the following format and can be provided multiple times to configure each size:
thp_anon=<size>,<size>[KMG]:<value>;<size>-<size>[KMG]:<value>
An example:
thp_anon=16K-64K:always;128K,512K:inherit;256K:madvise;1M-2M:never
See
Documentation/admin-guide/mm/transhuge.rstfor more details.Configuring the defaults at boot time is useful to allow early user space to take advantage of mTHP before its been configured through sysfs.
Quoted 8 lines
Signed-off-by Ryan Roberts <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>Reviewed-by Baolin Wang <[email protected]>Tested-by Baolin Wang <[email protected]>Cc David Hildenbrand <[email protected]>Cc Jonathan Corbet <[email protected]>Cc Lance Yang <[email protected]>
Some nits:
Quoted 16 lines
---
-v5:
* collect Baolin's reviewed-by and tested-by tags, thanks!
* useget_orderand check size is power 2, David, Baolin;
* use proper__initdata
.../admin-guide/kernel-parameters.txt| 9 ++Documentation/admin-guide/mm/transhuge.rst | 38 +++++-- mm/huge_memory.c | 100 +++++++++++++++++- 3 files changed, 139 insertions(+), 8 deletions(-)Documentation/admin-guide/kernel-parameters.txt +0
-0
Documentation/admin-guide/kernel-parameters.txt line 6635
Quoted 7 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0@@ -6629,6 +6629,15 @@ <deci-seconds>: poll all this frequency 0: no polling (default) + thp_anon= [KNL] + Format: <size>,<size>[KMG]:<state>;<size>-<size>[KMG]:<state> + state is one of "always", "madvise", "never" or "inherit". + Can be used to control the default behavior of the
s/Can be used to control/Control/
Quoted 15 lines
Documentation/admin-guide/kernel-parameters.txt +5
-0+ system with respect to anonymous transparent hugepages. + Can be used multiple times for multiple anon THP sizes. + See Documentation/admin-guide/mm/transhuge.rst for more + details. + threadirqs [KNL,EARLY] Force threading of all interrupt handlers except those marked explicitly IRQF_NO_THREAD.Documentation/admin-guide/mm/transhuge.rst +0
-0@@ -284,13 +284,37 @@ that THP is shared. Exceeding the number would block the collapse:: A higher value may increase memory footprint for some workloads.
[...]
Quoted 20 lines
unsigned long __thp_vma_allowable_orders(struct
vm_area_struct*vma,
unsigned longvm_flags,Documentation/admin-guide/mm/transhuge.rst +6
-1@@ -756,7 +757,10 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) * disable all other sizes. powerpc's PMD_ORDER isn't a compile-time * constant so we have to do this here. */ - huge_anon_orders_inherit = BIT(PMD_ORDER); + if (!anon_orders_configured) { + huge_anon_orders_inherit = BIT(PMD_ORDER); + anon_orders_configured = true; + } *hugepage_kobj = kobject_create_and_add("transparent_hugepage", mm_kobj); if (unlikely(!*hugepage_kobj)) {@@ -941,6 +945,100 @@ static int __init setup_transparent_hugepage(char *str) } __setup("transparent_hugepage=", setup_transparent_hugepage); +static inline int get_order_from_str(const char *size_str) +{
Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 959
Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +10
-0+ unsigned long size; + char *endptr; + int order; + + size = memparse(size_str, &endptr); + + if (!is_power_of_2(size)) + goto err; + order = get_order(size); + if ((1 << order) & ~THP_ORDERS_ALL_ANON)
Could do
"if (BIT(order) & ~THP_ORDERS_ALL_ANON)"
Quoted 14 lines
Documentation/admin-guide/mm/transhuge.rst +14
-0+ goto err; + + return order; +err: + pr_err("invalid size %s in thp_anon boot parameter\n", size_str); + return -EINVAL; +} + +static char str_dup[PAGE_SIZE] __initdata; +static int __init setup_thp_anon(char *str) +{ + char *token, *range, *policy, *subtoken; + unsigned long always, inherit, madvise; + char *start_size, *end_size;
Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 983
Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +10
-0+ int start, end, nr; + char *p; + + if (!str || strlen(str) + 1 > PAGE_SIZE) + goto err; + strcpy(str_dup, str); + + always = huge_anon_orders_always; + madvise = huge_anon_orders_madvise; + inherit = huge_anon_orders_inherit;
Should we only pickup these values if "anon_orders_configured", otherwise start with 0? Likely that's implicit right now.
Quoted 13 lines
Documentation/admin-guide/mm/transhuge.rst +13
-0+ p = str_dup; + while ((token = strsep(&p, ";")) != NULL) { + range = strsep(&token, ":"); + policy = token; + + if (!policy) + goto err; + + while ((subtoken = strsep(&range, ",")) != NULL) { + if (strchr(subtoken, '-')) { + start_size = strsep(&subtoken, "-"); + end_size = subtoken; +
Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 1006
Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +10
-0+ start = get_order_from_str(start_size); + end = get_order_from_str(end_size); + } else { + start = end = get_order_from_str(subtoken); + } + + if (start < 0 || end < 0 || start > end) + goto err; + + nr = end - start + 1;
This only works as long as we don't have any "Holes", which is the case right now.
Quoted 21 lines
Documentation/admin-guide/mm/transhuge.rst +21
-0+ if (!strcmp(policy, "always")) { + bitmap_set(&always, start, nr); + bitmap_clear(&inherit, start, nr); + bitmap_clear(&madvise, start, nr); + } else if (!strcmp(policy, "madvise")) { + bitmap_set(&madvise, start, nr); + bitmap_clear(&inherit, start, nr); + bitmap_clear(&always, start, nr); + } else if (!strcmp(policy, "inherit")) { + bitmap_set(&inherit, start, nr); + bitmap_clear(&madvise, start, nr); + bitmap_clear(&always, start, nr); + } else if (!strcmp(policy, "never")) { + bitmap_clear(&inherit, start, nr); + bitmap_clear(&madvise, start, nr); + bitmap_clear(&always, start, nr); + } else { + pr_err("invalid policy %s in thp_anon boot parameter\n", policy); + goto err; + } + }
Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 1037
Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +10
-0+ } + + huge_anon_orders_always = always; + huge_anon_orders_madvise = madvise; + huge_anon_orders_inherit = inherit; + anon_orders_configured = true; + return 1; + +err: + pr_warn("thp_anon=%s: cannot parse, ignored\n", str);
"cannot parse, ignored" -> "error parsing string, ignoring setting" ?
Quoted 7 lines
Documentation/admin-guide/mm/transhuge.rst +4
-0+ return 0; +} +__setup("thp_anon=", setup_thp_anon); + pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { if (likely(vma->vm_flags & VM_WRITE))
Signature · 3 lines
Cheers, David / dhildenb
Quoted 30 lines · David Hildenbrand
Quoted 18 lines · Barry Song
From: Ryan Roberts <[email protected]>
Add thp_anon= cmdline parameter to allow specifying the default enablement of each supported anon THP size. The parameter accepts the following format and can be provided multiple times to configure each size:
thp_anon=<size>,<size>[KMG]:<value>;<size>-<size>[KMG]:<value>
An example:
thp_anon=16K-64K:always;128K,512K:inherit;256K:madvise;1M-2M:never
See
Documentation/admin-guide/mm/transhuge.rstfor more details.Configuring the defaults at boot time is useful to allow early user space to take advantage of mTHP before its been configured through sysfs.
Quoted 8 lines
Signed-off-by Ryan Roberts <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>Reviewed-by Baolin Wang <[email protected]>Tested-by Baolin Wang <[email protected]>Cc David Hildenbrand <[email protected]>Cc Jonathan Corbet <[email protected]>Cc Lance Yang <[email protected]>
Quoted 2 lines
thanks!
Quoted 21 lines
Some nits:
Quoted 19 lines
---
-v5:
* collect Baolin's reviewed-by and tested-by tags, thanks!
* useget_orderand check size is power 2, David, Baolin;
* use proper__initdata
.../admin-guide/kernel-parameters.txt| 9 ++Documentation/admin-guide/mm/transhuge.rst | 38 +++++-- mm/huge_memory.c | 100 +++++++++++++++++- 3 files changed, 139 insertions(+), 8 deletions(-)Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -6629,6 +6629,15 @@ <deci-seconds>: poll all this frequency 0: no polling (default)
Quoted 6 lines
Documentation/admin-guide/kernel-parameters.txt line 6635
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ thp_anon= [KNL] + Format: <size>,<size>[KMG]:<state>;<size>-<size>[KMG]:<state> + state is one of "always", "madvise", "never" or "inherit". + Can be used to control the default behavior of thes/Can be used to control/Control/
ack
Quoted 19 lines
Quoted 16 lines
Documentation/admin-guide/kernel-parameters.txt +5
-0+ system with respect to anonymous transparent hugepages. + Can be used multiple times for multiple anon THP sizes. + See Documentation/admin-guide/mm/transhuge.rst for more + details. + threadirqs [KNL,EARLY] Force threading of all interrupt handlers except those marked explicitly IRQF_NO_THREAD.Documentation/admin-guide/mm/transhuge.rst +0
-0@@ -284,13 +284,37 @@ that THP is shared. Exceeding the number would block the collapse:: A higher value may increase memory footprint for some workloads.[...]
Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) · line 762
Quoted 10 lines
unsigned long __thp_vma_allowable_orders(struct
vm_area_struct*vma,
unsigned longvm_flags,Documentation/admin-guide/mm/transhuge.rst +3
-1@@ -756,7 +757,10 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) * disable all other sizes. powerpc's PMD_ORDER isn't a compile-time * constant so we have to do this here. */ - huge_anon_orders_inherit = BIT(PMD_ORDER); + if (!anon_orders_configured) { + huge_anon_orders_inherit = BIT(PMD_ORDER); + anon_orders_configured = true;
I realized this is redundant since anon_orders_configured won't be accessed later.
so i would like to also drop "anon_orders_configured = true" in v6.
Quoted 23 lines
Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +3
-0+ } *hugepage_kobj = kobject_create_and_add("transparent_hugepage", mm_kobj); if (unlikely(!*hugepage_kobj)) {@@ -941,6 +945,100 @@ static int __init setup_transparent_hugepage(char *str) } __setup("transparent_hugepage=", setup_transparent_hugepage); +static inline int get_order_from_str(const char *size_str) +{Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 959
Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +10
-0+ unsigned long size; + char *endptr; + int order; + + size = memparse(size_str, &endptr); + + if (!is_power_of_2(size)) + goto err; + order = get_order(size); + if ((1 << order) & ~THP_ORDERS_ALL_ANON)Could do
Quoted 1 line
"if (BIT(order) & ~THP_ORDERS_ALL_ANON)"
ack
Quoted 20 lines
Quoted 20 lines
Documentation/admin-guide/mm/transhuge.rst +19
-0+ goto err; + + return order; +err: + pr_err("invalid size %s in thp_anon boot parameter\n", size_str); + return -EINVAL; +} + +static char str_dup[PAGE_SIZE] __initdata; +static int __init setup_thp_anon(char *str) +{ + char *token, *range, *policy, *subtoken; + unsigned long always, inherit, madvise; + char *start_size, *end_size; + int start, end, nr; + char *p; + + if (!str || strlen(str) + 1 > PAGE_SIZE) + goto err;
Quoted 8 lines
Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 983
Quoted 5 lines
Documentation/admin-guide/mm/transhuge.rst +5
-0+ strcpy(str_dup, str); + + always = huge_anon_orders_always; + madvise = huge_anon_orders_madvise; + inherit = huge_anon_orders_inherit;Should we only pickup these values if "
anon_orders_configured", otherwise start with 0? Likely that's implicit right now.
My point is that, initially, those values are always 0, so copying them won't cause any issues.
Of course, we could add a check like
if (anon_orders_configured)
copy...
but it doesn't seem to be a hot path by any means? in that case, i will have to initialize:
unsigned long always = 0, inherit = 0, madvise = 0;
Quoted 19 lines
Quoted 19 lines
Documentation/admin-guide/mm/transhuge.rst +18
-0+ p = str_dup; + while ((token = strsep(&p, ";")) != NULL) { + range = strsep(&token, ":"); + policy = token; + + if (!policy) + goto err; + + while ((subtoken = strsep(&range, ",")) != NULL) { + if (strchr(subtoken, '-')) { + start_size = strsep(&subtoken, "-"); + end_size = subtoken; + + start = get_order_from_str(start_size); + end = get_order_from_str(end_size); + } else { + start = end = get_order_from_str(subtoken); + }
Quoted 8 lines
Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 1006
Quoted 5 lines
Documentation/admin-guide/mm/transhuge.rst +5
-0+ + if (start < 0 || end < 0 || start > end) + goto err; + + nr = end - start + 1;This only works as long as we don't have any "Holes", which is the case right now.
Right. If a gap appears in the future, I can verify that all bits from start to end are valid against THP_ORDERS_ALL_ANON.
Quoted 26 lines
Quoted 26 lines
Documentation/admin-guide/mm/transhuge.rst +25
-0+ if (!strcmp(policy, "always")) { + bitmap_set(&always, start, nr); + bitmap_clear(&inherit, start, nr); + bitmap_clear(&madvise, start, nr); + } else if (!strcmp(policy, "madvise")) { + bitmap_set(&madvise, start, nr); + bitmap_clear(&inherit, start, nr); + bitmap_clear(&always, start, nr); + } else if (!strcmp(policy, "inherit")) { + bitmap_set(&inherit, start, nr); + bitmap_clear(&madvise, start, nr); + bitmap_clear(&always, start, nr); + } else if (!strcmp(policy, "never")) { + bitmap_clear(&inherit, start, nr); + bitmap_clear(&madvise, start, nr); + bitmap_clear(&always, start, nr); + } else { + pr_err("invalid policy %s in thp_anon boot parameter\n", policy); + goto err; + } + } + } + + huge_anon_orders_always = always; + huge_anon_orders_madvise = madvise;
Quoted 8 lines
Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 1037
Quoted 6 lines
Documentation/admin-guide/mm/transhuge.rst +6
-0+ huge_anon_orders_inherit = inherit; + anon_orders_configured = true; + return 1; + +err: + pr_warn("thp_anon=%s: cannot parse, ignored\n", str);"cannot parse, ignored" -> "error parsing string, ignoring setting" ?
Ack.
Quoted 12 lines
Documentation/admin-guide/mm/transhuge.rst static int __init setup_transparent_hugepage(char *str) · line 1044
Quoted 8 lines
Documentation/admin-guide/mm/transhuge.rst +4
-0+ return 0; +} +__setup("thp_anon=", setup_thp_anon); + pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { if (likely(vma->vm_flags & VM_WRITE))--
Cheers,
Quoted 2 lines
David / dhildenb
Thanks
Barry
long __thp_vma_allowable_orders(struct vm_area_struct *vma,>>>
unsigned long vm_flags,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mm/huge_memory.c +0
-0@@ -756,7 +757,10 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj)* disable all other sizes. powerpc's PMD_ORDER isn't a compile-time * constant so we have to do this here. */
Quoted 8 lines
Quoted 4 lines
mm/huge_memory.c static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) · line 762
Quoted 4 lines
mm/huge_memory.c +3
-1- huge_anon_orders_inherit =BIT (PMD_ORDER ); +if (!anon_orders_configured) { + huge_anon_orders_inherit =BIT (PMD_ORDER ); + anon_orders_configured = true;I realized this is redundant since
anon_orders_configuredwon't be accessed later.
so i would like to also drop "anon_orders_configured= true" in v6.
Makes sense.
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
mm/huge_memory.c +11
-0+static char str_dup[PAGE_SIZE ]__initdata ; +static int __init setup_thp_anon (char *str) +{ +char *token, *range, *policy, *subtoken; +unsigned long always, inherit, madvise; +char *start_size, *end_size; +int start, end, nr; +char *p; + +if (!str ||strlen (str) +1 >PAGE_SIZE ) + goto err;Quoted 8 lines
mm/huge_memory.c static int __init setup_transparent_hugepage(char *str) · line 983
Quoted 5 lines
mm/huge_memory.c +5
-0+strcpy (str_dup, str); + + always = huge_anon_orders_always; + madvise = huge_anon_orders_madvise; + inherit = huge_anon_orders_inherit;Should we only pickup these values if "
anon_orders_configured", otherwise start with 0? Likely that's implicit right now.
Quoted 2 lines
My point is that, initially, those values are always 0, so copying them won't cause any issues.
Right, it's more a conceptual thing: on the first cmdline configuration, we start from scratch. Afterwards we start with the state that the previous configuration left behind.
I'm fine with leaving it as is as well, whatever you prefer.
Signature · 3 lines
Cheers, David / dhildenb