Skip to contentSkip to conversation

mm: huge_memory: don't start_stop_khugepaged for non-PMD THP

PATCH In review Barry Song 3 participants 8 messages Jul 26, 2024

Subsystems (2)
lore
Share & tools
lore

Barry Song

author
#0 lore raw

PATCH 1/1 mm: huge_memory: don't start_stop_khugepaged for non-PMD THP

From: Barry Song <[email protected]>

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

It seems unnecessary to call start_stop_khugepaged() for non-PMD THP, as it would only waste CPU time.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yang Shi <[email protected]>
Cc Zi Yan <[email protected]>
---
 mm/huge_memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
index 41460847988c..bd365e35acf7 100644
@@ -514,7 +514,7 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj,
 	} else
 		ret = -EINVAL;
 
-	if (ret > 0) {
+	if (ret > 0 && order == HPAGE_PMD_ORDER) {
 		int err;
 
 		err = start_stop_khugepaged();
Signature · 1 line
2.34.1
reviewer
#1 lore raw
Quoted 27 lines · Barry Song

From: Barry Song <[email protected]>

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

It seems unnecessary to call start_stop_khugepaged() for non-PMD THP, as it would only waste CPU time.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yang Shi <[email protected]>
Cc Zi Yan <[email protected]>
---
 mm/huge_memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +0 -0
index 41460847988c..bd365e35acf7 100644

mm/huge_memory.c static ssize_t thpsize_enabled_store(struct kobject *kobj, · line 520

Quoted 8 lines
mm/huge_memory.c +1 -1
@@ -514,7 +514,7 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj,
 	} else
 		ret = -EINVAL;
 
-	if (ret > 0) {
+	if (ret > 0 && order == HPAGE_PMD_ORDER) {
 		int err;
 
 		err = start_stop_khugepaged();

Personally I see this as a bit of a layering violation; its start_stop_khugepaged() that should decide the policy for when to start and stop the daemon. thpsize_enabled_store() should just be calling start_stop_khugepaged() to notify that something potentially pertinent to the a policy decision has changed.

And I don't think this should be a hot path; I'd expect the global thp size configuration to be set at boot and remain pretty static. Do you have evidence to the contrary?

#2 lore raw
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Barry Song

From: Barry Song <[email protected]>

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

It seems unnecessary to call start_stop_khugepaged() for non-PMD THP, as it would only waste CPU time.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yang Shi <[email protected]>
Cc Zi Yan <[email protected]>
---
  mm/huge_memory.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
index 41460847988c..bd365e35acf7 100644
@@ -514,7 +514,7 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj,
  	} else
  		ret = -EINVAL;
 
-	if (ret > 0) {
+	if (ret > 0 && order == HPAGE_PMD_ORDER) {
  		int err;
 
Quoted 7 lines

mm/huge_memory.c static ssize_t thpsize_enabled_store(struct kobject *kobj, · line 520

Quoted 1 line
mm/huge_memory.c +0 -0
  		err = start_stop_khugepaged();

Personally I see this as a bit of a layering violation; its start_stop_khugepaged() that should decide the policy for when to start and stop the daemon. thpsize_enabled_store() should just be calling start_stop_khugepaged() to notify that something potentially pertinent to the a policy decision has changed.

Agreed, skimming the subject I was under the impression that we would be fixing something here.

Signature · 3 lines
Cheers,

David / dhildenb
author
#3 lore raw
Quoted 37 lines · David Hildenbrand
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Barry Song

From: Barry Song <[email protected]>

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

It seems unnecessary to call start_stop_khugepaged() for non-PMD THP, as it would only waste CPU time.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yang Shi <[email protected]>
Cc Zi Yan <[email protected]>
---
  mm/huge_memory.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
index 41460847988c..bd365e35acf7 100644
@@ -514,7 +514,7 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj,
     } else
             ret = -EINVAL;
 
-    if (ret > 0) {
+    if (ret > 0 && order == HPAGE_PMD_ORDER) {
             int err;
 
Quoted 8 lines
Quoted 7 lines

mm/huge_memory.c static ssize_t thpsize_enabled_store(struct kobject *kobj, · line 520

Quoted 1 line
mm/huge_memory.c +0 -0
             err = start_stop_khugepaged();

Personally I see this as a bit of a layering violation; its start_stop_khugepaged() that should decide the policy for when to start and stop the daemon. thpsize_enabled_store() should just be calling start_stop_khugepaged() to notify that something potentially pertinent to the a policy decision has changed.

My impression is that it slightly deviates from the huge page documentation in Documentation/admin-guide/mm/transhuge.rst.

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

non-PMD size is not involved in khugepaged, but I agree the policy might change in the future.

Quoted 2 lines

Agreed, skimming the subject I was under the impression that we would be fixing something here.

working on another swapin_enabled and reviewing the enabled source code. I don't need this startstop for all sizes in that case, so I made a quick adjustment
to this part as well. If neither of you likes it, that's fine with me :-)

Quoted 6 lines

--
Cheers,

David / dhildenb

Thanks
Barry

reviewer
#4 lore raw
Quoted 47 lines · Barry Song
Quoted 37 lines · David Hildenbrand
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Barry Song

From: Barry Song <[email protected]>

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

It seems unnecessary to call start_stop_khugepaged() for non-PMD THP, as it would only waste CPU time.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yang Shi <[email protected]>
Cc Zi Yan <[email protected]>
---
  mm/huge_memory.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
index 41460847988c..bd365e35acf7 100644
@@ -514,7 +514,7 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj,
     } else
             ret = -EINVAL;
 
-    if (ret > 0) {
+    if (ret > 0 && order == HPAGE_PMD_ORDER) {
             int err;
 
Quoted 8 lines
Quoted 7 lines

mm/huge_memory.c static ssize_t thpsize_enabled_store(struct kobject *kobj, · line 520

Quoted 1 line
mm/huge_memory.c +0 -0
             err = start_stop_khugepaged();

Personally I see this as a bit of a layering violation; its start_stop_khugepaged() that should decide the policy for when to start and stop the daemon. thpsize_enabled_store() should just be calling start_stop_khugepaged() to notify that something potentially pertinent to the a policy decision has changed.

Quoted 8 lines

My impression is that it slightly deviates from the huge page documentation in Documentation/admin-guide/mm/transhuge.rst.

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

But start_stop_khugepaged() doesn't unconditionally start khugepaged, it takes action based on hugepage_pmd_enabled() which only returns true if there are any pmd sized THP enabled (currently looking at anon and file, but should also look at shmem in future; that's a known bug that's been there forever). So I don't think it is inconsistent with the documentation?

Quoted 21 lines

non-PMD size is not involved in khugepaged, but I agree the policy might change in the future.

Quoted 2 lines

Agreed, skimming the subject I was under the impression that we would be fixing something here.

working on another swapin_enabled and reviewing the enabled source code. I don't need this startstop for all sizes in that case, so I made a quick adjustment
to this part as well. If neither of you likes it, that's fine with me :-)

Quoted 6 lines

--
Cheers,

David / dhildenb

Thanks
Barry

author
#5 lore raw
Quoted 58 lines · Ryan Roberts
Quoted 47 lines · Barry Song
Quoted 37 lines · David Hildenbrand
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Barry Song

From: Barry Song <[email protected]>

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

It seems unnecessary to call start_stop_khugepaged() for non-PMD THP, as it would only waste CPU time.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yang Shi <[email protected]>
Cc Zi Yan <[email protected]>
---
  mm/huge_memory.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
index 41460847988c..bd365e35acf7 100644
@@ -514,7 +514,7 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj,
     } else
             ret = -EINVAL;
 
-    if (ret > 0) {
+    if (ret > 0 && order == HPAGE_PMD_ORDER) {
             int err;
 
Quoted 8 lines
Quoted 7 lines

mm/huge_memory.c static ssize_t thpsize_enabled_store(struct kobject *kobj, · line 520

Quoted 1 line
mm/huge_memory.c +0 -0
             err = start_stop_khugepaged();

Personally I see this as a bit of a layering violation; its start_stop_khugepaged() that should decide the policy for when to start and stop the daemon. thpsize_enabled_store() should just be calling start_stop_khugepaged() to notify that something potentially pertinent to the a policy decision has changed.

Quoted 8 lines

My impression is that it slightly deviates from the huge page documentation in Documentation/admin-guide/mm/transhuge.rst.

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

Quoted 5 lines

But start_stop_khugepaged() doesn't unconditionally start khugepaged, it takes action based on hugepage_pmd_enabled() which only returns true if there are any pmd sized THP enabled (currently looking at anon and file, but should also look at shmem in future; that's a known bug that's been there forever). So I don't think it is inconsistent with the documentation?

My point is that, as a code reader, if non-PMD sizes are never involved with khugepaged,
we can proceed without needing to check the lower-layer code. otherwise it is a bit
confusing, especially since start_stop_khugepaged() unconditionally calls set_recommended_min_free_kbytes() for all sizes, regardless of whether they are PMD.

However, I agree with your point that if non-PMD sizes are eventually managed by khugepaged, then this approach is flawed. This is the downside of this change.

Quoted 14 lines
Quoted 14 lines

non-PMD size is not involved in khugepaged, but I agree the policy might change in the future.

Quoted 2 lines

Agreed, skimming the subject I was under the impression that we would be fixing something here.

working on another swapin_enabled and reviewing the enabled source code. I don't need this startstop for all sizes in that case, so I made a quick adjustment
to this part as well. If neither of you likes it, that's fine with me :-)

Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

--
Cheers,

David / dhildenb

Thanks
Barry

reviewer
#6 lore raw
Quoted 73 lines · Barry Song
Quoted 58 lines · Ryan Roberts
Quoted 47 lines · Barry Song
Quoted 37 lines · David Hildenbrand
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Barry Song

From: Barry Song <[email protected]>

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

It seems unnecessary to call start_stop_khugepaged() for non-PMD THP, as it would only waste CPU time.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yang Shi <[email protected]>
Cc Zi Yan <[email protected]>
---
  mm/huge_memory.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
index 41460847988c..bd365e35acf7 100644
@@ -514,7 +514,7 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj,
     } else
             ret = -EINVAL;
 
-    if (ret > 0) {
+    if (ret > 0 && order == HPAGE_PMD_ORDER) {
             int err;
 
Quoted 8 lines
Quoted 7 lines

mm/huge_memory.c static ssize_t thpsize_enabled_store(struct kobject *kobj, · line 520

Quoted 1 line
mm/huge_memory.c +0 -0
             err = start_stop_khugepaged();

Personally I see this as a bit of a layering violation; its start_stop_khugepaged() that should decide the policy for when to start and stop the daemon. thpsize_enabled_store() should just be calling start_stop_khugepaged() to notify that something potentially pertinent to the a policy decision has changed.

Quoted 8 lines

My impression is that it slightly deviates from the huge page documentation in Documentation/admin-guide/mm/transhuge.rst.

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

Quoted 5 lines

But start_stop_khugepaged() doesn't unconditionally start khugepaged, it takes action based on hugepage_pmd_enabled() which only returns true if there are any pmd sized THP enabled (currently looking at anon and file, but should also look at shmem in future; that's a known bug that's been there forever). So I don't think it is inconsistent with the documentation?

My point is that, as a code reader, if non-PMD sizes are never involved with khugepaged,
we can proceed without needing to check the lower-layer code. otherwise it is a bit
confusing, especially since start_stop_khugepaged() unconditionally calls set_recommended_min_free_kbytes() for all sizes, regardless of whether they are PMD.

Quoted 2 lines

However, I agree with your point that if non-PMD sizes are eventually managed by khugepaged, then this approach is flawed. This is the downside of this change.

Yes, I'd hope that eventually khugepaged will support non-PMD sizes. That's something we are hoping to look at later this year. So my personal preference is to leave as is.

Quoted 23 lines
Quoted 15 lines
Quoted 15 lines

non-PMD size is not involved in khugepaged, but I agree the policy might change in the future.

Quoted 2 lines

Agreed, skimming the subject I was under the impression that we would be fixing something here.

working on another swapin_enabled and reviewing the enabled source code. I don't need this startstop for all sizes in that case, so I made a quick adjustment
to this part as well. If neither of you likes it, that's fine with me :-)

Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

--
Cheers,

David / dhildenb

Thanks
Barry

author
#7 lore raw
In reply to Ryan Roberts
Quoted 44 lines · Ryan Roberts
Quoted 27 lines · Barry Song

From: Barry Song <[email protected]>

khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

It seems unnecessary to call start_stop_khugepaged() for non-PMD THP, as it would only waste CPU time.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Baolin Wang <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yang Shi <[email protected]>
Cc Zi Yan <[email protected]>
---
 mm/huge_memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +0 -0
index 41460847988c..bd365e35acf7 100644

mm/huge_memory.c static ssize_t thpsize_enabled_store(struct kobject *kobj, · line 520

Quoted 8 lines
mm/huge_memory.c +1 -1
@@ -514,7 +514,7 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj,
      } else
              ret = -EINVAL;
 
-     if (ret > 0) {
+     if (ret > 0 && order == HPAGE_PMD_ORDER) {
              int err;
 
              err = start_stop_khugepaged();

Personally I see this as a bit of a layering violation; its start_stop_khugepaged() that should decide the policy for when to start and stop the daemon. thpsize_enabled_store() should just be calling start_stop_khugepaged() to notify that something potentially pertinent to the a policy decision has changed.

Quoted 3 lines

And I don't think this should be a hot path; I'd expect the global thp size configuration to be set at boot and remain pretty static. Do you have evidence to the contrary?

Absolutely no, I was reading and patching the doc "Documentation/admin-guide/mm/transhuge.rst"
and was also copying-paste your "enabled" code to "swapin_enabled" with dropping start_stop_khugepaged() in that case. I believe I don't need it for enabling/disabling
swapin as swapin is a subset of mTHP not a main entry.

just feel a bit inconsistency between the doc and the code at least for this moment:
khugepaged will be automatically started when PMD-sized THP is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when PMD-sized THP is disabled (when both the per-size anon control and the top-level control are "never").

which triggered me to make this change.

Thanks
Barry