mm: thp: always enable mTHP support

Shmem performs two kinds of has_transparent_hugepage() usage:

1. shmem_parse_one() and shmem_init(): since the calls to has_transparent_hugepage() are protected by #ifdef CONFIG_TRANSPARENT_HUGEPAGE, this actually checks if the CPU supports PMD-sized pages. This is irrelevant for shmem as it supports mTHP

2. shmem_parse_huge(): This is checking if THP is enabled and on architectures that implement has_transparent_hugepage(), this also checks if the CPU supports PMD-sized pages

While it's necessary to check if CONFIG_TRANSPARENT_HUGEPAGE is enabled, shmem can determine mTHP size support at folio allocation time. Therefore, drop has_transparent_hugepage() usage while keeping the CONFIG_TRANSPARENT_HUGEPAGE checks.

Signed-off-by Luiz Capitulino <[email protected]>

---

  mm/shmem.c | 7 +++----
  1 file changed, 3 insertions(+), 4 deletions(-)

mm/shmem.c +3 -4

index 79af5f9f8b90..32529586cd78 100644
@@ -689,7 +689,7 @@ static int shmem_parse_huge(const char *str)
  	else
  		return -EINVAL;
 
-	if (!has_transparent_hugepage() &&
+	if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
  	    huge != SHMEM_HUGE_NEVER && huge != SHMEM_HUGE_DENY)
  		return -EINVAL;
 

@@ -4678,8 +4678,7 @@ static int shmem_parse_one(struct fs_context *fc, struct fs_parameter *param)
  	case Opt_huge:
  		ctx->huge = result.uint_32;
  		if (ctx->huge != SHMEM_HUGE_NEVER &&
-		    !(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
-		      has_transparent_hugepage()))
+		    !IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
  			goto unsupported_parameter;
  		ctx->seen |= SHMEM_SEEN_HUGE;
  		break;

@@ -5463,7 +5462,7 @@ void __init shmem_init(void)
  #endif
 
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (has_transparent_hugepage() && shmem_huge > SHMEM_HUGE_DENY)
+	if (shmem_huge > SHMEM_HUGE_DENY)
  		SHMEM_SB(shm_mnt->mnt_sb)->huge = shmem_huge;
  	else
  		shmem_huge = SHMEM_HUGE_NEVER; /* just in case it was patched */

Quoted 16 lines · Luiz Capitulino

Shmem performs two kinds of has_transparent_hugepage() usage:

1. shmem_parse_one() and shmem_init(): since the calls to has_transparent_hugepage() are protected by #ifdef CONFIG_TRANSPARENT_HUGEPAGE, this actually checks if the CPU supports PMD-sized pages. This is irrelevant for shmem as it supports mTHP

2. shmem_parse_huge(): This is checking if THP is enabled and on architectures that implement has_transparent_hugepage(), this also checks if the CPU supports PMD-sized pages

While it's necessary to check if CONFIG_TRANSPARENT_HUGEPAGE is enabled, shmem can determine mTHP size support at folio allocation time. Therefore, drop has_transparent_hugepage() usage while keeping the CONFIG_TRANSPARENT_HUGEPAGE checks.

Quoted 2 lines

Signed-off-by Luiz Capitulino <[email protected]>

---

Looks reasonable to me. Thanks.

Reviewed-by Baolin Wang <[email protected]>

Quoted 33 lines

  mm/shmem.c | 7 +++----
  1 file changed, 3 insertions(+), 4 deletions(-)

mm/shmem.c +3 -4

index 79af5f9f8b90..32529586cd78 100644
@@ -689,7 +689,7 @@ static int shmem_parse_huge(const char *str)
      else
          return -EINVAL;
-    if (!has_transparent_hugepage() &&
+    if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
          huge != SHMEM_HUGE_NEVER && huge != SHMEM_HUGE_DENY)
          return -EINVAL;

@@ -4678,8 +4678,7 @@ static int shmem_parse_one(struct fs_context *fc, struct fs_parameter *param)
      case Opt_huge:
          ctx->huge = result.uint_32;
          if (ctx->huge != SHMEM_HUGE_NEVER &&
-            !(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
-              has_transparent_hugepage()))
+            !IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
              goto unsupported_parameter;
          ctx->seen |= SHMEM_SEEN_HUGE;
          break;

@@ -5463,7 +5462,7 @@ void __init shmem_init(void)
  #endif
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-    if (has_transparent_hugepage() && shmem_huge > SHMEM_HUGE_DENY)
+    if (shmem_huge > SHMEM_HUGE_DENY)
          SHMEM_SB(shm_mnt->mnt_sb)->huge = shmem_huge;
      else
          shmem_huge = SHMEM_HUGE_NEVER; /* just in case it was patched */

If PMD-sized pages are not supported on an architecture (ie. the arch implements arch_has_pmd_leaves() and it returns false) then the current code disables all THP, including mTHP.

This commit fixes this by allowing mTHP to be always enabled for all archs. When PMD-sized pages are not supported, its sysfs entry won't be created and their mapping will be disallowed at page-fault time.

Similarly, this commit implements the following changes for shmem:

- In shmem_allowable_huge_orders(): drop the pgtable_has_pmd_leaves() check so that mTHP sizes are considered
- In shmem_alloc_and_add_folio(): don't consider PMD and PUD orders when PMD-sized pages are not supported by the CPU

Signed-off-by Luiz Capitulino <[email protected]>

---
  mm/huge_memory.c | 11 +++++++----
  mm/shmem.c       |  4 +++-
  2 files changed, 10 insertions(+), 5 deletions(-)

mm/huge_memory.c +4 -1

index 1e5ea2e47f79..882331592928 100644
@@ -115,6 +115,9 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
  	else
  		supported_orders = THP_ORDERS_ALL_FILE_DEFAULT;
 
+	if (!pgtable_has_pmd_leaves())
+		supported_orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER));
+
  	orders &= supported_orders;
  	if (!orders)
  		return 0;

@@ -122,7 +125,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
  	if (!vma->vm_mm)		/* vdso */
  		return 0;
 
-	if (!pgtable_has_pmd_leaves() || vma_thp_disabled(vma, vm_flags, forced_collapse))
+	if (vma_thp_disabled(vma, vm_flags, forced_collapse))
  		return 0;
 
  	/* khugepaged doesn't collapse DAX vma, but page fault is fine. */

mm/huge_memory.c +1 -3

+
  	order = highest_order(orders);
  	while (orders) {
  		thpsize = thpsize_create(order, *hugepage_kobj);

@@ -905,9 +911,6 @@ static int __init hugepage_init(void)
  	int err;
  	struct kobject *hugepage_kobj;
 
-	if (!pgtable_has_pmd_leaves())
-		return -EINVAL;
-
  	/*
  	 * hugepages can't be allocated by the buddy allocator
  	 */

mm/shmem.c +1 -1

index 1c98e84667a4..cb325d1e2d1e 100644
@@ -1827,7 +1827,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
  	vm_flags_t vm_flags = vma ? vma->vm_flags : 0;
  	unsigned int global_orders;
 
-	if (!pgtable_has_pmd_leaves() || (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force)))
+	if (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force))
  		return 0;
 
  	global_orders = shmem_huge_global_enabled(inode, index, write_end,

mm/shmem.c +0 -0

  	if (orders > 0) {
  		suitable_orders = shmem_suitable_orders(inode, vmf,

Quoted 42 lines · Luiz Capitulino

If PMD-sized pages are not supported on an architecture (ie. the arch implements arch_has_pmd_leaves() and it returns false) then the current code disables all THP, including mTHP.

This commit fixes this by allowing mTHP to be always enabled for all archs. When PMD-sized pages are not supported, its sysfs entry won't be created and their mapping will be disallowed at page-fault time.

Similarly, this commit implements the following changes for shmem:

- In shmem_allowable_huge_orders(): drop the pgtable_has_pmd_leaves() check so that mTHP sizes are considered
- In shmem_alloc_and_add_folio(): don't consider PMD and PUD orders when PMD-sized pages are not supported by the CPU

Signed-off-by Luiz Capitulino <[email protected]>

---
  mm/huge_memory.c | 11 +++++++----
  mm/shmem.c       |  4 +++-
  2 files changed, 10 insertions(+), 5 deletions(-)

mm/huge_memory.c +4 -1

index 1e5ea2e47f79..882331592928 100644
@@ -115,6 +115,9 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
      else
          supported_orders = THP_ORDERS_ALL_FILE_DEFAULT;
+    if (!pgtable_has_pmd_leaves())
+        supported_orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER));
+
      orders &= supported_orders;
      if (!orders)
          return 0;

@@ -122,7 +125,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
      if (!vma->vm_mm)        /* vdso */
          return 0;
-    if (!pgtable_has_pmd_leaves() || vma_thp_disabled(vma, vm_flags, forced_collapse))
+    if (vma_thp_disabled(vma, vm_flags, forced_collapse))
          return 0;
      /* khugepaged doesn't collapse DAX vma, but page fault is fine. */

mm/huge_memory.c static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) · line 812

Quoted 4 lines

mm/huge_memory.c +2 -0

@@ -806,6 +809,9 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj)
      }
      orders = THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DEFAULT;
+    if (!pgtable_has_pmd_leaves())
+        orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER));

I think you should also handle the 'huge_anon_orders_inherit' setting in this function if pgtable_has_pmd_leaves() returns false. Shmem as well.

if (!anon_orders_configured)

     huge_anon_orders_inherit = BIT(PMD_ORDER);

Quoted 24 lines

mm/huge_memory.c +1 -3

+
      order = highest_order(orders);
      while (orders) {
          thpsize = thpsize_create(order, *hugepage_kobj);

@@ -905,9 +911,6 @@ static int __init hugepage_init(void)
      int err;
      struct kobject *hugepage_kobj;
-    if (!pgtable_has_pmd_leaves())
-        return -EINVAL;
-
      /*
       * hugepages can't be allocated by the buddy allocator
       */

mm/shmem.c +1 -1

index 1c98e84667a4..cb325d1e2d1e 100644
@@ -1827,7 +1827,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
      vm_flags_t vm_flags = vma ? vma->vm_flags : 0;
      unsigned int global_orders;
-    if (!pgtable_has_pmd_leaves() || (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force)))
+    if (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force))
          return 0;

mm/shmem.c static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, · line 1938

Quoted 6 lines

mm/shmem.c +2 -0

      global_orders = shmem_huge_global_enabled(inode, index, write_end,

@@ -1935,6 +1935,8 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf,
      if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
          orders = 0;
+    else if (!pgtable_has_pmd_leaves())
+        orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER));

Moving this check into shmem_allowable_huge_orders() would be more appropriate.

Quoted 3 lines

mm/shmem.c +0 -0

      if (orders > 0) {
          suitable_orders = shmem_suitable_orders(inode, vmf,

Quoted 56 lines · Baolin Wang

Quoted 45 lines · Luiz Capitulino

If PMD-sized pages are not supported on an architecture (ie. the arch implements arch_has_pmd_leaves() and it returns false) then the current code disables all THP, including mTHP.

This commit fixes this by allowing mTHP to be always enabled for all archs. When PMD-sized pages are not supported, its sysfs entry won't be created and their mapping will be disallowed at page-fault time.

Similarly, this commit implements the following changes for shmem:

  - In shmem_allowable_huge_orders(): drop the pgtable_has_pmd_leaves() check so that mTHP sizes are considered
  - In shmem_alloc_and_add_folio(): don't consider PMD and PUD orders when PMD-sized pages are not supported by the CPU

Signed-off-by Luiz Capitulino <[email protected]>

---
  mm/huge_memory.c | 11 +++++++----
  mm/shmem.c       |  4 +++-

  2 files changed, 10 insertions(+), 5 deletions(-)

mm/huge_memory.c +0 -0

index 1e5ea2e47f79..882331592928 100644
@@ -115,6 +115,9 @@ unsigned long __thp_vma_allowable_orders(struct 
 vm_area_struct *vma,

      else
          supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; +    if (!pgtable_has_pmd_leaves())
+        supported_orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER)); +
      orders &= supported_orders;
      if (!orders)
          return 0;

mm/huge_memory.c +0 -0

@@ -122,7 +125,7 @@ unsigned long __thp_vma_allowable_orders(struct 
 vm_area_struct *vma,

      if (!vma->vm_mm)        /* vdso */
          return 0;
-    if (!pgtable_has_pmd_leaves() || vma_thp_disabled(vma, vm_flags, forced_collapse))
+    if (vma_thp_disabled(vma, vm_flags, forced_collapse)) return 0;
      /* khugepaged doesn't collapse DAX vma, but page fault is fine. */

mm/huge_memory.c static int __init hugepage_init_sysfs(struct · line 813

Quoted 5 lines

mm/huge_memory.c +2 -0

@@ -806,6 +809,9 @@ static int __init hugepage_init_sysfs(struct 
 kobject **hugepage_kobj)
       }
       orders = THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DEFAULT;
+    if (!pgtable_has_pmd_leaves())
+        orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER));

I think you should also handle the 'huge_anon_orders_inherit' setting in this function if pgtable_has_pmd_leaves() returns false. Shmem as well.

Quoted 2 lines

if (!anon_orders_configured)
     huge_anon_orders_inherit = BIT(PMD_ORDER);

Good catch. So, would you agree that should set it to BIT(PMD_ORDER - 1) in this case?

Quoted 42 lines · Luiz Capitulino

If PMD-sized pages are not supported on an architecture (ie. the arch implements arch_has_pmd_leaves() and it returns false) then the current code disables all THP, including mTHP.

This commit fixes this by allowing mTHP to be always enabled for all archs. When PMD-sized pages are not supported, its sysfs entry won't be created and their mapping will be disallowed at page-fault time.

Similarly, this commit implements the following changes for shmem:

- In shmem_allowable_huge_orders(): drop the pgtable_has_pmd_leaves() check so that mTHP sizes are considered
- In shmem_alloc_and_add_folio(): don't consider PMD and PUD orders when PMD-sized pages are not supported by the CPU

Signed-off-by Luiz Capitulino <[email protected]>

---
  mm/huge_memory.c | 11 +++++++----
  mm/shmem.c       |  4 +++-
  2 files changed, 10 insertions(+), 5 deletions(-)

mm/huge_memory.c +4 -1

index 1e5ea2e47f79..882331592928 100644
@@ -115,6 +115,9 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
      else
          supported_orders = THP_ORDERS_ALL_FILE_DEFAULT;
+    if (!pgtable_has_pmd_leaves())
+        supported_orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER));
+
      orders &= supported_orders;
      if (!orders)
          return 0;

@@ -122,7 +125,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
      if (!vma->vm_mm)        /* vdso */
          return 0;
-    if (!pgtable_has_pmd_leaves() || vma_thp_disabled(vma, vm_flags, forced_collapse))
+    if (vma_thp_disabled(vma, vm_flags, forced_collapse))
          return 0;
      /* khugepaged doesn't collapse DAX vma, but page fault is fine. */

mm/huge_memory.c static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) · line 812

Quoted 4 lines

mm/huge_memory.c +2 -0

@@ -806,6 +809,9 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj)
      }
      orders = THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DEFAULT;
+    if (!pgtable_has_pmd_leaves())
+        orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER));

I think you should also handle the 'huge_anon_orders_inherit' setting in this function if pgtable_has_pmd_leaves() returns false. Shmem as well.

if (!anon_orders_configured)

     huge_anon_orders_inherit = BIT(PMD_ORDER);

Quoted 24 lines

mm/huge_memory.c +1 -3

+
      order = highest_order(orders);
      while (orders) {
          thpsize = thpsize_create(order, *hugepage_kobj);

@@ -905,9 +911,6 @@ static int __init hugepage_init(void)
      int err;
      struct kobject *hugepage_kobj;
-    if (!pgtable_has_pmd_leaves())
-        return -EINVAL;
-
      /*
       * hugepages can't be allocated by the buddy allocator
       */

mm/shmem.c +1 -1

index 1c98e84667a4..cb325d1e2d1e 100644
@@ -1827,7 +1827,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
      vm_flags_t vm_flags = vma ? vma->vm_flags : 0;
      unsigned int global_orders;
-    if (!pgtable_has_pmd_leaves() || (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force)))
+    if (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force))
          return 0;

mm/shmem.c static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, · line 1938

Quoted 6 lines

mm/shmem.c +2 -0

      global_orders = shmem_huge_global_enabled(inode, index, write_end,

@@ -1935,6 +1935,8 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf,
      if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
          orders = 0;
+    else if (!pgtable_has_pmd_leaves())
+        orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER));

Moving this check into shmem_allowable_huge_orders() would be more appropriate.

Quoted 3 lines

mm/shmem.c +0 -0

      if (orders > 0) {
          suitable_orders = shmem_suitable_orders(inode, vmf,

Today, if an architecture implements has_transparent_hugepage() and the CPU lacks support for PMD-sized pages, the THP code disables all THP, including mTHP. In addition, the kernel lacks a well defined API to check for PMD-sized page support. It currently relies on has_transparent_hugepage() and thp_disabled_by_hw(), but they are not well defined and are tied to THP support.

This series addresses both issues by introducing a new well defined API to query PMD-sized page support: pgtable_has_pmd_leaves(). Using this new helper, we ensure that mTHP remains enabled even when the architecture or CPU doesn't support PMD-sized pages.

Thanks to David Hildenbrand for suggesting this improvement and for providing guidance (all bugs and misconceptions are mine).

This applies to v6.19, but I tested it on v6.19-rc8+.

Quoted 12 lines · Luiz Capitulino

Today, if an architecture implements has_transparent_hugepage() and the CPU lacks support for PMD-sized pages, the THP code disables all THP, including mTHP. In addition, the kernel lacks a well defined API to check for PMD-sized page support. It currently relies on has_transparent_hugepage() and thp_disabled_by_hw(), but they are not well defined and are tied to THP support.

This series addresses both issues by introducing a new well defined API to query PMD-sized page support: pgtable_has_pmd_leaves(). Using this new helper, we ensure that mTHP remains enabled even when the architecture or CPU doesn't support PMD-sized pages.

Quoted 4 lines

Thanks to David Hildenbrand for suggesting this improvement and for providing guidance (all bugs and misconceptions are mine).

This applies to v6.19, but I tested it on v6.19-rc8+.

Apologies for the delay Luiz (as told you off-list :) ), I'll try going through this soon so we can start queuing this after -rc1.

Quoted 21 lines · David Hildenbrand (Arm)

Quoted 19 lines · Luiz Capitulino

Today, if an architecture implements has_transparent_hugepage() and the CPU
lacks support for PMD-sized pages, the THP code disables all THP, including
mTHP. In addition, the kernel lacks a well defined API to check for PMD-sized page support. It currently relies on has_transparent_hugepage()
and thp_disabled_by_hw(), but they are not well defined and are tied to THP support.

This series addresses both issues by introducing a new well defined API to query PMD-sized page support: pgtable_has_pmd_leaves(). Using this new helper, we ensure that mTHP remains enabled even when the architecture or CPU doesn't support PMD-sized pages.

Thanks to David Hildenbrand for suggesting this improvement and for providing guidance (all bugs and misconceptions are mine).

This applies to v6.19, but I tested it on v6.19-rc8+.

Quoted 2 lines

Apologies for the delay Luiz (as told you off-list :) ), I'll try going through this soon so we can start queuing this after -rc1.

Oh, wow, thanks for the support!

I intended to get back to this series this week to address Baolin's comments, but I got busy with other things (which is something I may bring up on upstream but it's not related to this series at all).