Skip to contentSkip to conversation

mm, bpf: BPF based THP adjustment

RFC PATCH Superseded Yafang Shao 11 participants 53 messages May 20, 2025 – May 28, 2025

Subsystems (5)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover mm, bpf: BPF based THP adjustment

Background
----------

At my current employer, PDD, we have consistently configured THP to "never" on our production servers due to past incidents caused by its behavior:

- Increased memory consumption
THP significantly raises overall memory usage.

- Latency spikes
Random latency spikes occur due to more frequent memory compaction activity triggered by THP.

These issues have made sysadmins hesitant to switch to "madvise" or "always" modes.

New Motivation
--------------

We have now identified that certain AI workloads achieve substantial performance gains with THP enabled. However, we’ve also verified that some workloads see little to no benefit—or are even negatively impacted—by THP.

In our Kubernetes environment, we deploy mixed workloads on a single server to maximize resource utilization. Our goal is to selectively enable THP for services that benefit from it while keeping it disabled for others. This approach allows us to incrementally enable THP for additional services and assess how to make it more viable in production.

Proposed Solution
-----------------

For this use case, Johannes suggested introducing a dedicated mode [0]. In this new mode, we could implement BPF-based THP adjustment for fine-grained control over tasks or cgroups. If no BPF program is attached, THP remains in "never" mode. This solution elegantly meets our needs while avoiding the complexity of managing BPF alongside other THP modes.

A selftest example demonstrates how to enable THP for the current task while keeping it disabled for others.

Alternative Proposals
---------------------

- Gutierrez’s cgroup-based approach [1]
- Proposed adding a new cgroup file to control THP policy.
- However, as Johannes noted, cgroups are designed for hierarchical resource allocation, not arbitrary policy settings [2].

- Usama’s per-task THP proposal based on prctl() [3]:
- Enabling THP per task via prctl().
- As David pointed out, neither madvise() nor prctl() works in "never" mode [4], making this solution insufficient for our needs.

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

This is currently a PoC implementation. Feedback of any kind is welcome.

RFC v1->v2:
The main changes are as follows,
- Use struct_ops instead of fmod_ret (Alexei)
- Introduce a new THP mode (Johannes)
- Introduce new helpers for BPF hook (Zi)
- Refine the commit log

RFC v1: https://lwn.net/Articles/1019290/

Yafang Shao (5):
mm: thp: Add a new mode "bpf"
mm: thp: Add hook for BPF based THP adjustment
mm: thp: add struct ops for BPF based THP adjustment
bpf: Add get_current_comm to bpf_base_func_proto selftests/bpf: Add selftest for THP adjustment

 include/linux/huge_mm.h                       |  15 +-
 kernel/bpf/cgroup.c                           |   2 -
 kernel/bpf/helpers.c                          |   2 +
 mm/Makefile                                   |   3 +
 mm/bpf_thp.c                                  | 120 ++++++++++++
 mm/huge_memory.c                              |  65 ++++++-
 mm/khugepaged.c                               |   3 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 175 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  39 ++++
 10 files changed, 414 insertions(+), 11 deletions(-)
 create mode 100644 mm/bpf_thp.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
Signature · 1 line
2.43.5
#1 lore raw

RFC PATCH v2 1/5 mm: thp: Add a new mode "bpf"

Background
----------

Historically, our production environment has always configured THP to never due to past incidents. This has made system administrators hesitant to switch to madvise.

New Motivation
--------------

We’ve now identified that AI workloads can achieve significant performance gains with THP enabled. To balance safety and performance, we aim to allow THP only for AI services while keeping the global system setting at never.

Proposed Solution
-----------------

Johannes suggested introducing a dedicated mode for this use case [0]. This approach elegantly solves our problem while avoiding the complexity of managing BPF alongside other THP modes.

---
 include/linux/huge_mm.h |  2 ++
 mm/huge_memory.c        | 65 ++++++++++++++++++++++++++++++++++++-----
 2 files changed, 59 insertions(+), 8 deletions(-)
include/linux/huge_mm.h +2 -0
index e893d546a49f..3b5429f73e6e 100644
@@ -54,6 +54,7 @@ enum transparent_hugepage_flag {
 	TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
 	TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG,
 	TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG,
+	TRANSPARENT_HUGEPAGE_REQ_BPF_FLAG,	/* "bpf" mode */
 };
 
 struct kobject;
@@ -174,6 +175,7 @@ static inline void count_mthp_stat(int order, enum mthp_stat_item item)
 
 extern unsigned long transparent_hugepage_flags;
 extern unsigned long huge_anon_orders_always;
+extern unsigned long huge_anon_orders_bpf;
 extern unsigned long huge_anon_orders_madvise;
 extern unsigned long huge_anon_orders_inherit;
 
mm/huge_memory.c +57 -8

180 lines not on this page · raw message

Signature · 1 line
2.43.5
#2 lore raw

RFC PATCH v2 2/5 mm: thp: Add hook for BPF based THP adjustment

In reply to Yafang Shao

This patch introduces new hooks for BPF program attachment and adds a flag to indicate when a BPF program is attached. The program only functions when "bpf" mode is enabled.

Per task THP policy based on BPF will be added in the followup patch.

---
 include/linux/huge_mm.h | 24 +++++++++++++++++++++++-
 mm/khugepaged.c         |  3 +++
 2 files changed, 26 insertions(+), 1 deletion(-)
include/linux/huge_mm.h +23 -1
index 3b5429f73e6e..fedb5b014d9a 100644
@@ -55,6 +55,7 @@ enum transparent_hugepage_flag {
 	TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG,
 	TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG,
 	TRANSPARENT_HUGEPAGE_REQ_BPF_FLAG,	/* "bpf" mode */
+	TRANSPARENT_HUGEPAGE_BPF_ATTACHED,	/* BPF program is attached */
 };
 
 struct kobject;
@@ -192,6 +193,26 @@ static inline bool hugepage_global_always(void)
 			(1<<TRANSPARENT_HUGEPAGE_FLAG);
 }
 
+static inline bool hugepage_bpf_allowable(void)
+{
+	/* Works only for BPF mode */
+	if (!(transparent_hugepage_flags & (1<<TRANSPARENT_HUGEPAGE_REQ_BPF_FLAG)))
+		return 0;
+
+	/* No BPF program is attached */
+	if (!(transparent_hugepage_flags & (1<<TRANSPARENT_HUGEPAGE_BPF_ATTACHED)))
+		return 0;
+	/* We will add struct ops in the future */
+	return 1;
+}
+
+static inline bool hugepaged_bpf_allowable(void)
+{
+	if (!(transparent_hugepage_flags & (1<<TRANSPARENT_HUGEPAGE_BPF_ATTACHED)))
+		return 0;
+	return 1;
+}
+
 static inline int highest_order(unsigned long orders)
 {
 	return fls_long(orders) - 1;
@@ -295,7 +316,8 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
 		if (vm_flags & VM_HUGEPAGE)
 			mask |= READ_ONCE(huge_anon_orders_madvise);
 		if (hugepage_global_always() ||
-		    ((vm_flags & VM_HUGEPAGE) && hugepage_global_enabled()))
+		    ((vm_flags & VM_HUGEPAGE) && hugepage_global_enabled()) ||
+		    hugepage_bpf_allowable())
 			mask |= READ_ONCE(huge_anon_orders_inherit);
 
 		orders &= mask;
mm/khugepaged.c +3 -0
index cc945c6ab3bd..762e03b50bca 100644
@@ -432,6 +432,9 @@ static bool hugepage_pmd_enabled(void)
 	if (test_bit(PMD_ORDER, &huge_anon_orders_inherit) &&
 	    hugepage_global_enabled())
 		return true;
+	if (test_bit(PMD_ORDER, &huge_anon_orders_bpf) &&
+	    hugepaged_bpf_allowable())
+		return true;
 	if (IS_ENABLED(CONFIG_SHMEM) && shmem_hpage_pmd_enabled())
 		return true;
 	return false;
Signature · 1 line
2.43.5
#3 lore raw

RFC PATCH v2 3/5 mm: thp: add struct ops for BPF based THP adjustment

In reply to Yafang Shao

This patch introduces a minimal struct_ops for BPF-based THP adjustment.

Currently, only a single BPF program can be attached. Support for multiple attachments will be added in the future.

---
 include/linux/huge_mm.h |  13 +----
 mm/Makefile             |   3 +
 mm/bpf_thp.c            | 120 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 124 insertions(+), 12 deletions(-)
 create mode 100644 mm/bpf_thp.c
include/linux/huge_mm.h +1 -12
index fedb5b014d9a..a75f5f902af0 100644
@@ -193,18 +193,7 @@ static inline bool hugepage_global_always(void)
 			(1<<TRANSPARENT_HUGEPAGE_FLAG);
 }
 
-static inline bool hugepage_bpf_allowable(void)
-{
-	/* Works only for BPF mode */
-	if (!(transparent_hugepage_flags & (1<<TRANSPARENT_HUGEPAGE_REQ_BPF_FLAG)))
-		return 0;
-
-	/* No BPF program is attached */
-	if (!(transparent_hugepage_flags & (1<<TRANSPARENT_HUGEPAGE_BPF_ATTACHED)))
-		return 0;
-	/* We will add struct ops in the future */
-	return 1;
-}
+bool hugepage_bpf_allowable(void);
 
 static inline bool hugepaged_bpf_allowable(void)
 {
mm/Makefile +3 -0
index e7f6bbf8ae5f..c355f9426c93 100644
@@ -99,6 +99,9 @@ obj-$(CONFIG_MIGRATION) += migrate.o
 obj-$(CONFIG_NUMA) += memory-tiers.o
 obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
 obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
+ifdef CONFIG_BPF_SYSCALL
+obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += bpf_thp.o
+endif
 obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
 obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
 obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
mm/bpf_thp.c new file +120 -0

120 lines not on this page · raw message

Signature · 1 line
2.43.5
#4 lore raw

Re: [RFC PATCH v2 3/5] mm: thp: add struct ops for BPF based THP adjustment

kernel test robot noticed the following build errors:

Hi Yafang,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Yafang-Shao/mm-thp-Add-a-new-mode-bpf/20250520-140834
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20250520060504.20251-4-laoar.shao%40gmail.com
patch subject: [RFC PATCH v2 3/5] mm: thp: add struct ops for BPF based THP adjustment
config: s390-randconfig-002-20250520 (https://download.01.org/0day-ci/archive/20250521/[email protected]/config)
compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project f819f46284f2a79790038e1f6649172789734ae8)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250521/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

s390x-linux-ld: mm/memory.o: in function `thp_vma_allowable_orders':

>> include/linux/huge_mm.h:309:(.text+0x43bc0): undefined reference to `hugepage_bpf_allowable'

s390x-linux-ld: mm/khugepaged.o: in function `thp_vma_allowable_orders':
include/linux/huge_mm.h:309:(.text+0x1628): undefined reference to `hugepage_bpf_allowable'
s390x-linux-ld: fs/proc/task_mmu.o: in function `thp_vma_allowable_orders':
include/linux/huge_mm.h:309:(.text+0xb00e): undefined reference to `hugepage_bpf_allowable'

vim +309 include/linux/huge_mm.h

3485b88390b0af9 Ryan Roberts   2023-12-07  274  
3485b88390b0af9 Ryan Roberts   2023-12-07  275  unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
e0ffb29bc54d86b Matthew Wilcox 2024-04-25  276  					 unsigned long vm_flags,
e0ffb29bc54d86b Matthew Wilcox 2024-04-25  277  					 unsigned long tva_flags,
3485b88390b0af9 Ryan Roberts   2023-12-07  278  					 unsigned long orders);
3485b88390b0af9 Ryan Roberts   2023-12-07  279  
3485b88390b0af9 Ryan Roberts   2023-12-07  280  /**
3485b88390b0af9 Ryan Roberts   2023-12-07  281   * thp_vma_allowable_orders - determine hugepage orders that are allowed for vma
3485b88390b0af9 Ryan Roberts   2023-12-07  282   * @vma:  the vm area to check
3485b88390b0af9 Ryan Roberts   2023-12-07  283   * @vm_flags: use these vm_flags instead of vma->vm_flags
e0ffb29bc54d86b Matthew Wilcox 2024-04-25  284   * @tva_flags: Which TVA flags to honour
3485b88390b0af9 Ryan Roberts   2023-12-07  285   * @orders: bitfield of all orders to consider
3485b88390b0af9 Ryan Roberts   2023-12-07  286   *
3485b88390b0af9 Ryan Roberts   2023-12-07  287   * Calculates the intersection of the requested hugepage orders and the allowed
3485b88390b0af9 Ryan Roberts   2023-12-07  288   * hugepage orders for the provided vma. Permitted orders are encoded as a set
3485b88390b0af9 Ryan Roberts   2023-12-07  289   * bit at the corresponding bit position (bit-2 corresponds to order-2, bit-3
3485b88390b0af9 Ryan Roberts   2023-12-07  290   * corresponds to order-3, etc). Order-0 is never considered a hugepage order.
3485b88390b0af9 Ryan Roberts   2023-12-07  291   *
3485b88390b0af9 Ryan Roberts   2023-12-07  292   * Return: bitfield of orders allowed for hugepage in the vma. 0 if no hugepage
3485b88390b0af9 Ryan Roberts   2023-12-07  293   * orders are allowed.
3485b88390b0af9 Ryan Roberts   2023-12-07  294   */
3485b88390b0af9 Ryan Roberts   2023-12-07  295  static inline
3485b88390b0af9 Ryan Roberts   2023-12-07  296  unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
e0ffb29bc54d86b Matthew Wilcox 2024-04-25  297  				       unsigned long vm_flags,
e0ffb29bc54d86b Matthew Wilcox 2024-04-25  298  				       unsigned long tva_flags,
3485b88390b0af9 Ryan Roberts   2023-12-07  299  				       unsigned long orders)
3485b88390b0af9 Ryan Roberts   2023-12-07  300  {
3485b88390b0af9 Ryan Roberts   2023-12-07  301  	/* Optimization to check if required orders are enabled early. */
e0ffb29bc54d86b Matthew Wilcox 2024-04-25  302  	if ((tva_flags & TVA_ENFORCE_SYSFS) && vma_is_anonymous(vma)) {
3485b88390b0af9 Ryan Roberts   2023-12-07  303  		unsigned long mask = READ_ONCE(huge_anon_orders_always);
3485b88390b0af9 Ryan Roberts   2023-12-07  304  
3485b88390b0af9 Ryan Roberts   2023-12-07  305  		if (vm_flags & VM_HUGEPAGE)
3485b88390b0af9 Ryan Roberts   2023-12-07  306  			mask |= READ_ONCE(huge_anon_orders_madvise);
3485b88390b0af9 Ryan Roberts   2023-12-07  307  		if (hugepage_global_always() ||
31f9439b0bbc383 Yafang Shao    2025-05-20  308  		    ((vm_flags & VM_HUGEPAGE) && hugepage_global_enabled()) ||
31f9439b0bbc383 Yafang Shao    2025-05-20 @309  		    hugepage_bpf_allowable())
3485b88390b0af9 Ryan Roberts   2023-12-07  310  			mask |= READ_ONCE(huge_anon_orders_inherit);
3485b88390b0af9 Ryan Roberts   2023-12-07  311  
3485b88390b0af9 Ryan Roberts   2023-12-07  312  		orders &= mask;
3485b88390b0af9 Ryan Roberts   2023-12-07  313  		if (!orders)
3485b88390b0af9 Ryan Roberts   2023-12-07  314  			return 0;
3485b88390b0af9 Ryan Roberts   2023-12-07  315  	}
3485b88390b0af9 Ryan Roberts   2023-12-07  316  
e0ffb29bc54d86b Matthew Wilcox 2024-04-25  317  	return __thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders);
3485b88390b0af9 Ryan Roberts   2023-12-07  318  }
43675e6fbbeadca Yang Shi       2019-07-18  319  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#5 lore raw

RFC PATCH v2 4/5 bpf: Add get_current_comm to bpf_base_func_proto

In reply to Yafang Shao

While testing the BPF based THP adjustment feature, I noticed bpf_get_current_comm() isn't available in bpf_base_func_proto. As this is a commonly used helper, we should add it to bpf_base_func_proto.

---
 kernel/bpf/cgroup.c  | 2 --
 kernel/bpf/helpers.c | 2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)
kernel/bpf/cgroup.c +0 -2
index 84f58f3d028a..22cd4f54d023 100644
@@ -2609,8 +2609,6 @@ cgroup_current_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 	switch (func_id) {
 	case BPF_FUNC_get_current_uid_gid:
 		return &bpf_get_current_uid_gid_proto;
-	case BPF_FUNC_get_current_comm:
-		return &bpf_get_current_comm_proto;
 #ifdef CONFIG_CGROUP_NET_CLASSID
 	case BPF_FUNC_get_cgroup_classid:
 		return &bpf_get_cgroup_classid_curr_proto;
kernel/bpf/helpers.c +2 -0
index e3a2662f4e33..2a60522cd66f 100644
@@ -1965,6 +1965,8 @@ bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 		return &bpf_get_current_pid_tgid_proto;
 	case BPF_FUNC_get_ns_current_pid_tgid:
 		return &bpf_get_ns_current_pid_tgid_proto;
+	case BPF_FUNC_get_current_comm:
+		return &bpf_get_current_comm_proto;
 	default:
 		break;
 	}
Signature · 1 line
2.43.5
#6 lore raw

Re: [RFC PATCH v2 4/5] bpf: Add get_current_comm to bpf_base_func_proto

Quoted 11 lines · Yafang Shao

While testing the BPF based THP adjustment feature, I noticed bpf_get_current_comm() isn't available in bpf_base_func_proto. As this is a commonly used helper, we should add it to bpf_base_func_proto.

---
 kernel/bpf/cgroup.c  | 2 --
 kernel/bpf/helpers.c | 2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)

please rebase, there were changes in this area and bpf_get_current_comm is already in bpf_base_func_proto (and cgroup_current_func_proto is gone)

Quoted 29 lines
kernel/bpf/cgroup.c +0 -2
index 84f58f3d028a..22cd4f54d023 100644
@@ -2609,8 +2609,6 @@ cgroup_current_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
        switch (func_id) {
        case BPF_FUNC_get_current_uid_gid:
                return &bpf_get_current_uid_gid_proto;
-       case BPF_FUNC_get_current_comm:
-               return &bpf_get_current_comm_proto;
 #ifdef CONFIG_CGROUP_NET_CLASSID
        case BPF_FUNC_get_cgroup_classid:
                return &bpf_get_cgroup_classid_curr_proto;
kernel/bpf/helpers.c +2 -1
index e3a2662f4e33..2a60522cd66f 100644
@@ -1965,6 +1965,8 @@ bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
                return &bpf_get_current_pid_tgid_proto;
        case BPF_FUNC_get_ns_current_pid_tgid:
                return &bpf_get_ns_current_pid_tgid_proto;
+       case BPF_FUNC_get_current_comm:
+               return &bpf_get_current_comm_proto;
        default:
                break;
        }
--

2.43.5

#7 lore raw

RFC PATCH v2 5/5 selftests/bpf: Add selftest for THP adjustment

In reply to Yafang Shao

This test case uses a BPF program to enforce the following THP allocation policy:
- Only the current task is permitted to allocate THP.
- All other tasks are denied.

The expected behavior:
- Before the BPF prog is attached
No tasks can allocate THP.
- After the BPF prog is attached
Only the current task can allocate THP.
- Switch to "never" mode after the BPF prog is attached THP allocation is not allowed even for the current task.

The result is as follows,

  $ ./test_progs --name="thp_adjust"
  #437     thp_adjust:OK
  Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED

CONFIG_TRANSPARENT_HUGEPAGE=y is required for this test.

---
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 175 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  39 ++++
 3 files changed, 215 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
tools/testing/selftests/bpf/config +1 -0
index c378d5d07e02..bb8a8a9d77a2 100644
@@ -113,3 +113,4 @@ CONFIG_XDP_SOCKETS=y
 CONFIG_XFRM_INTERFACE=y
 CONFIG_TCP_CONG_DCTCP=y
 CONFIG_TCP_CONG_BBR=y
+CONFIG_TRANSPARENT_HUGEPAGE=y
tools/testing/selftests/bpf/prog_tests/thp_adjust.c new file +175 -0

175 lines not on this page · raw message

tools/testing/selftests/bpf/progs/test_thp_adjust.c new file +39 -0
new file mode 100644
index 000000000000..69135380853c
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+int target_pid;
+int thp_calls;
+int thp_wrong_calls;
+
+SEC("fentry/do_huge_pmd_anonymous_page")
+int BPF_PROG(thp_run)
+{
+	struct task_struct *current = bpf_get_current_task_btf();
+
+	if (current->pid == target_pid)
+		thp_calls++;
+	else
+		thp_wrong_calls++;
+	return 0;
+}
+
+SEC("struct_ops/thp_bpf_allowable")
+bool BPF_PROG(thp_bpf_allowable)
+{
+	struct task_struct *current = bpf_get_current_task_btf();
+
+	/* Permit the current task to allocate memory using THP. */
+	if (current->pid == target_pid)
+		return true;
+	return false;
+}
+
+SEC(".struct_ops.link")
+struct bpf_thp_ops thp = {
+	.thp_bpf_allowable = (void *)thp_bpf_allowable,
+};
Signature · 1 line
2.43.5
#8 lore raw
In reply to Yafang Shao
Quoted 50 lines · Yafang Shao

Background
----------

At my current employer, PDD, we have consistently configured THP to "never" on our production servers due to past incidents caused by its behavior:

- Increased memory consumption
THP significantly raises overall memory usage.

- Latency spikes
Random latency spikes occur due to more frequent memory compaction activity triggered by THP.

These issues have made sysadmins hesitant to switch to "madvise" or "always" modes.

New Motivation
--------------

We have now identified that certain AI workloads achieve substantial performance gains with THP enabled. However, we’ve also verified that some workloads see little to no benefit—or are even negatively impacted—by THP.

In our Kubernetes environment, we deploy mixed workloads on a single server to maximize resource utilization. Our goal is to selectively enable THP for services that benefit from it while keeping it disabled for others. This approach allows us to incrementally enable THP for additional services and assess how to make it more viable in production.

Proposed Solution
-----------------

For this use case, Johannes suggested introducing a dedicated mode [0]. In this new mode, we could implement BPF-based THP adjustment for fine-grained control over tasks or cgroups. If no BPF program is attached, THP remains in "never" mode. This solution elegantly meets our needs while avoiding the complexity of managing BPF alongside other THP modes.

A selftest example demonstrates how to enable THP for the current task while keeping it disabled for others.

Alternative Proposals
---------------------

- Gutierrez’s cgroup-based approach [1]
- Proposed adding a new cgroup file to control THP policy.
- However, as Johannes noted, cgroups are designed for hierarchical resource allocation, not arbitrary policy settings [2].

Quoted 4 lines

- Usama’s per-task THP proposal based on prctl() [3]:
- Enabling THP per task via prctl().
- As David pointed out, neither madvise() nor prctl() works in "never" mode [4], making this solution insufficient for our needs.

Hi Yafang Shao,

I believe you would have to invert your logic and disable the processes you dont want using THPs, and have THP="madvise"|"always". I have yet to look over Usama's solution in detail but I believe this is possible based on his cover letter.

I also have an alternative solution proposed here!
https://lore.kernel.org/lkml/[email protected]/

It's different in the sense it doesn't give you granular control per process, cgroup, or BPF programmability, but it "may" suit your needs by taming the THP waste and removing the latency spikes of PF time THP compactions/allocations.

Cheers,
-- Nico

Quoted 51 lines

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

This is currently a PoC implementation. Feedback of any kind is welcome.

RFC v1->v2:
The main changes are as follows,
- Use struct_ops instead of fmod_ret (Alexei)
- Introduce a new THP mode (Johannes)
- Introduce new helpers for BPF hook (Zi)
- Refine the commit log

RFC v1: https://lwn.net/Articles/1019290/

Yafang Shao (5):
mm: thp: Add a new mode "bpf"
mm: thp: Add hook for BPF based THP adjustment
mm: thp: add struct ops for BPF based THP adjustment
bpf: Add get_current_comm to bpf_base_func_proto selftests/bpf: Add selftest for THP adjustment

 include/linux/huge_mm.h                       |  15 +-
 kernel/bpf/cgroup.c                           |   2 -
 kernel/bpf/helpers.c                          |   2 +
 mm/Makefile                                   |   3 +
 mm/bpf_thp.c                                  | 120 ++++++++++++
 mm/huge_memory.c                              |  65 ++++++-
 mm/khugepaged.c                               |   3 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 175 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  39 ++++
 10 files changed, 414 insertions(+), 11 deletions(-)
 create mode 100644 mm/bpf_thp.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c

--
2.43.5

#9 lore raw
Quoted 63 lines · Nico Pache
Quoted 50 lines · Yafang Shao

Background
----------

At my current employer, PDD, we have consistently configured THP to "never" on our production servers due to past incidents caused by its behavior:

- Increased memory consumption
THP significantly raises overall memory usage.

- Latency spikes
Random latency spikes occur due to more frequent memory compaction activity triggered by THP.

These issues have made sysadmins hesitant to switch to "madvise" or "always" modes.

New Motivation
--------------

We have now identified that certain AI workloads achieve substantial performance gains with THP enabled. However, we’ve also verified that some workloads see little to no benefit—or are even negatively impacted—by THP.

In our Kubernetes environment, we deploy mixed workloads on a single server to maximize resource utilization. Our goal is to selectively enable THP for services that benefit from it while keeping it disabled for others. This approach allows us to incrementally enable THP for additional services and assess how to make it more viable in production.

Proposed Solution
-----------------

For this use case, Johannes suggested introducing a dedicated mode [0]. In this new mode, we could implement BPF-based THP adjustment for fine-grained control over tasks or cgroups. If no BPF program is attached, THP remains in "never" mode. This solution elegantly meets our needs while avoiding the complexity of managing BPF alongside other THP modes.

A selftest example demonstrates how to enable THP for the current task while keeping it disabled for others.

Alternative Proposals
---------------------

- Gutierrez’s cgroup-based approach [1]
- Proposed adding a new cgroup file to control THP policy.
- However, as Johannes noted, cgroups are designed for hierarchical resource allocation, not arbitrary policy settings [2].

Quoted 4 lines

- Usama’s per-task THP proposal based on prctl() [3]:
- Enabling THP per task via prctl().
- As David pointed out, neither madvise() nor prctl() works in "never" mode [4], making this solution insufficient for our needs.

Hi Yafang Shao,

I believe you would have to invert your logic and disable the processes you dont want using THPs, and have THP="madvise"|"always". I have yet to look over Usama's solution in detail but I believe this is possible based on his cover letter.

Quoted 7 lines

I also have an alternative solution proposed here!
https://lore.kernel.org/lkml/[email protected]/

It's different in the sense it doesn't give you granular control per process, cgroup, or BPF programmability, but it "may" suit your needs by taming the THP waste and removing the latency spikes of PF time THP compactions/allocations.

Thank you for developing this feature. I'll review it carefully.

The challenge we face is that our system administration team doesn't permit enabling THP globally in production by setting it to "madvise" or "always". As a result, we can only experiment with your feature on our test servers at this stage.

Therefore, our immediate priority isn't THP optimization, but rather finding a way to safely enable THP in production first. The kernel team needs a solution that addresses this fundamental deployment hurdle before we can consider performance improvements.

Signature · 2 lines
Regards
Yafang
#10 lore raw
Quoted 4 lines · Yafang Shao

The challenge we face is that our system administration team doesn't permit enabling THP globally in production by setting it to "madvise" or "always". As a result, we can only experiment with your feature on our test servers at this stage.

That's a you problem. You need to figure out how to influence your sysadmin team to change their mind; whether it's by talking to their superiors or persuading them directly. It's not a justification for why upstream should take this patch.

#11 lore raw
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Yafang Shao

The challenge we face is that our system administration team doesn't permit enabling THP globally in production by setting it to "madvise" or "always". As a result, we can only experiment with your feature on our test servers at this stage.

That's a you problem.

perhaps.

Quoted 3 lines

You need to figure out how to influence your
sysadmin team to change their mind; whether it's by talking to their superiors or persuading them directly.

I believe that "practicing" matters more than "talking" or "persuading". I’m surprised your suggestion relies on "talking" ;-)
If I understand correctly, we all agree that "talk is cheap", right?

Quoted 2 lines

It's not a justification for why
upstream should take this patch.

I believe Johannes has clearly explained the challenges the community is currently facing [0].

[0]. https://lore.kernel.org/linux-mm/[email protected]/

--
Regards

Yafang

#12 lore raw
Quoted 20 lines · Yafang Shao
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Yafang Shao

The challenge we face is that our system administration team doesn't permit enabling THP globally in production by setting it to "madvise" or "always". As a result, we can only experiment with your feature on our test servers at this stage.

That's a you problem.

perhaps.

Quoted 3 lines

You need to figure out how to influence your
sysadmin team to change their mind; whether it's by talking to their superiors or persuading them directly.

I believe that "practicing" matters more than "talking" or "persuading". I’m surprised your suggestion relies on "talking" ;-)
If I understand correctly, we all agree that "talk is cheap", right?

Quoted 7 lines
Quoted 2 lines

It's not a justification for why
upstream should take this patch.

I believe Johannes has clearly explained the challenges the community is currently facing [0].

[0]. https://lore.kernel.org/linux-mm/[email protected]/

(Sorry to interject on your conversation, but :)

I don't think anybody denies we have issues in configuring this stuff sensibly. A global-only control isn't going to cut it in the real world it seems.

To me as you say yourself, definining the ABI/API here is what really matters, and we're right now inundated with several series all at once (you wait for one bus then 3 come at once... :).

So this I think, should be the question.

I like the idea of just exposing something like madvise(), which is something we're going to maintain indefinitely.

Though any such exposure would in my view would need to be opt-in i.e. have a list of MADV_... options that are accepted, as we'd need to very cautiously determine which are safe from this context.

Of course then this leads to the whole thing (and I really know very little about BPF internals - obviously happy to understand more) of whether we can just use the madvise() code direct or what locking we can do or how all that works.

At any rate, a custom thing that is specific as 'switch mode for mTHP pages of size X to Y' is just something I'd rather us not tie ourselves to.

Quoted 6 lines

--
Regards

Yafang

What do you think re: bpf vs. something like my proposed process_madvise() extensions or Usama's proposed prctl()?

Simpler, but really just using madvise functionality and having a means of defaulting across fork/exec (notwithstanding Jann's concerns in this area).

#13 lore raw
Quoted 60 lines · Lorenzo Stoakes
Quoted 20 lines · Yafang Shao
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Yafang Shao

The challenge we face is that our system administration team doesn't permit enabling THP globally in production by setting it to "madvise" or "always". As a result, we can only experiment with your feature on our test servers at this stage.

That's a you problem.

perhaps.

Quoted 3 lines

You need to figure out how to influence your
sysadmin team to change their mind; whether it's by talking to their superiors or persuading them directly.

I believe that "practicing" matters more than "talking" or "persuading". I’m surprised your suggestion relies on "talking" ;-)
If I understand correctly, we all agree that "talk is cheap", right?

Quoted 7 lines
Quoted 2 lines

It's not a justification for why
upstream should take this patch.

I believe Johannes has clearly explained the challenges the community is currently facing [0].

[0]. https://lore.kernel.org/linux-mm/[email protected]/

(Sorry to interject on your conversation, but :)

I don't think anybody denies we have issues in configuring this stuff sensibly. A global-only control isn't going to cut it in the real world it seems.

To me as you say yourself, definining the ABI/API here is what really matters, and we're right now inundated with several series all at once (you wait for one bus then 3 come at once... :).

So this I think, should be the question.

I like the idea of just exposing something like madvise(), which is something we're going to maintain indefinitely.

Though any such exposure would in my view would need to be opt-in i.e. have a list of MADV_... options that are accepted, as we'd need to very cautiously determine which are safe from this context.

Of course then this leads to the whole thing (and I really know very little about BPF internals - obviously happy to understand more) of whether we can just use the madvise() code direct or what locking we can do or how all that works.

At any rate, a custom thing that is specific as 'switch mode for mTHP pages of size X to Y' is just something I'd rather us not tie ourselves to.

Quoted 5 lines

--
Regards

Quoted 7 lines
Quoted 1 line

Yafang

What do you think re: bpf vs. something like my proposed process_madvise() extensions or Usama's proposed prctl()?

Simpler, but really just using madvise functionality and having a means of defaulting across fork/exec (notwithstanding Jann's concerns in this area).

Unfortunately I think the issue is that neither prctl or process_madvise would work for Yafangs usecase? Its usecase 3 mentioned in [1], i.e.
global system policy=never, process wants "madvise" policy for itself. Will let Yafang confirm.

[1] https://lore.kernel.org/all/[email protected]/

#14 lore raw
Quoted 71 lines · Usama Arif
Quoted 60 lines · Lorenzo Stoakes
Quoted 20 lines · Yafang Shao
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Yafang Shao

The challenge we face is that our system administration team doesn't permit enabling THP globally in production by setting it to "madvise" or "always". As a result, we can only experiment with your feature on our test servers at this stage.

That's a you problem.

perhaps.

Quoted 3 lines

You need to figure out how to influence your
sysadmin team to change their mind; whether it's by talking to their superiors or persuading them directly.

I believe that "practicing" matters more than "talking" or "persuading". I’m surprised your suggestion relies on "talking" ;-)
If I understand correctly, we all agree that "talk is cheap", right?

Quoted 7 lines
Quoted 2 lines

It's not a justification for why
upstream should take this patch.

I believe Johannes has clearly explained the challenges the community is currently facing [0].

[0]. https://lore.kernel.org/linux-mm/[email protected]/

(Sorry to interject on your conversation, but :)

I don't think anybody denies we have issues in configuring this stuff sensibly. A global-only control isn't going to cut it in the real world it seems.

To me as you say yourself, definining the ABI/API here is what really matters, and we're right now inundated with several series all at once (you wait for one bus then 3 come at once... :).

So this I think, should be the question.

I like the idea of just exposing something like madvise(), which is something we're going to maintain indefinitely.

Though any such exposure would in my view would need to be opt-in i.e. have a list of MADV_... options that are accepted, as we'd need to very cautiously determine which are safe from this context.

Of course then this leads to the whole thing (and I really know very little about BPF internals - obviously happy to understand more) of whether we can just use the madvise() code direct or what locking we can do or how all that works.

At any rate, a custom thing that is specific as 'switch mode for mTHP pages of size X to Y' is just something I'd rather us not tie ourselves to.

Quoted 5 lines

--
Regards

Quoted 7 lines
Quoted 1 line

Yafang

What do you think re: bpf vs. something like my proposed process_madvise() extensions or Usama's proposed prctl()?

Simpler, but really just using madvise functionality and having a means of defaulting across fork/exec (notwithstanding Jann's concerns in this area).

Quoted 7 lines

Unfortunately I think the issue is that neither prctl or process_madvise would work for Yafangs usecase? Its usecase 3 mentioned in [1], i.e.
global system policy=never, process wants "madvise" policy for itself. Will let Yafang confirm.

[1] https://lore.kernel.org/all/[email protected]/

Yeah I really object to that case. I explicitly said on your series I object to it, I believe David did too.

Never should mean never.

It's a NACK if that's what this is about unless I'm missing something here.

I agree global settings are not fine-grained enough, but 'sys admins refuse to do X so we want to ignore what they do' is... really not right at all.

#15 lore raw
Quoted 2 lines · Lorenzo Stoakes

I agree global settings are not fine-grained enough, but 'sys admins refuse to do X so we want to ignore what they do' is... really not right at all.

Oh, we do that all the time, Leave the interface around but document it's now a no-op. For example, file-backed memory ignores the THP settings completely. And mounting an NFS filesystem as "intr" has been a no-op for over a decade.

#16 lore raw
Quoted 7 lines · Matthew Wilcox
Quoted 2 lines · Lorenzo Stoakes

I agree global settings are not fine-grained enough, but 'sys admins refuse to do X so we want to ignore what they do' is... really not right at all.

Oh, we do that all the time, Leave the interface around but document it's now a no-op. For example, file-backed memory ignores the THP settings completely.

IIRC, it never honored it. Like shmem it also never honored it an instead used it's own toggle (not sure what to think about that ...).

Signature · 3 lines
Cheers,

David / dhildenb
#17 lore raw
In reply to Matthew Wilcox
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Lorenzo Stoakes

I agree global settings are not fine-grained enough, but 'sys admins refuse to do X so we want to ignore what they do' is... really not right at all.

Oh, we do that all the time, Leave the interface around but document it's now a no-op. For example, file-backed memory ignores the THP settings completely.

This essentially invites downstream kernel developers to implement their own "file-enabled" solutions ;-)

If you haven't yet encountered reports of file-backed THP causing performance regressions for specific workloads, you may be missing something. Our testing has confirmed performance degradation with certain HDFS workloads, even on the 6.12.y kernel - though I've prioritized discussing BPF-based THP control with you over investigating those specific cases.

Quoted 2 lines

And mounting an NFS filesystem as "intr" has
been a no-op for over a decade.

Signature · 2 lines
Regards
Yafang
#18 lore raw
In reply to Lorenzo Stoakes
Quoted 77 lines · Lorenzo Stoakes
Quoted 76 lines · Usama Arif
Quoted 60 lines · Lorenzo Stoakes
Quoted 20 lines · Yafang Shao
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Yafang Shao

The challenge we face is that our system administration team doesn't permit enabling THP globally in production by setting it to "madvise" or "always". As a result, we can only experiment with your feature on our test servers at this stage.

That's a you problem.

perhaps.

Quoted 3 lines

You need to figure out how to influence your
sysadmin team to change their mind; whether it's by talking to their superiors or persuading them directly.

I believe that "practicing" matters more than "talking" or "persuading". I’m surprised your suggestion relies on "talking" ;-)
If I understand correctly, we all agree that "talk is cheap", right?

Quoted 7 lines
Quoted 2 lines

It's not a justification for why
upstream should take this patch.

I believe Johannes has clearly explained the challenges the community is currently facing [0].

[0]. https://lore.kernel.org/linux-mm/[email protected]/

(Sorry to interject on your conversation, but :)

I don't think anybody denies we have issues in configuring this stuff sensibly. A global-only control isn't going to cut it in the real world it seems.

To me as you say yourself, definining the ABI/API here is what really matters, and we're right now inundated with several series all at once (you wait for one bus then 3 come at once... :).

So this I think, should be the question.

I like the idea of just exposing something like madvise(), which is something we're going to maintain indefinitely.

Though any such exposure would in my view would need to be opt-in i.e. have a list of MADV_... options that are accepted, as we'd need to very cautiously determine which are safe from this context.

Of course then this leads to the whole thing (and I really know very little about BPF internals - obviously happy to understand more) of whether we can just use the madvise() code direct or what locking we can do or how all that works.

At any rate, a custom thing that is specific as 'switch mode for mTHP pages of size X to Y' is just something I'd rather us not tie ourselves to.

Quoted 5 lines

--
Regards

Quoted 7 lines
Quoted 1 line

Yafang

What do you think re: bpf vs. something like my proposed process_madvise() extensions or Usama's proposed prctl()?

Simpler, but really just using madvise functionality and having a means of defaulting across fork/exec (notwithstanding Jann's concerns in this area).

Unfortunately I think the issue is that neither prctl or process_madvise would work for Yafangs usecase? Its usecase 3 mentioned in [1], i.e.
global system policy=never, process wants "madvise" policy for itself. Will let Yafang confirm.

Quoted 5 lines
Quoted 2 lines

[1] https://lore.kernel.org/all/[email protected]/

Yeah I really object to that case. I explicitly said on your series I object to it, I believe David did too.

Yes, I am not for it as well, which is why my series never tried to do it :) As I mentioned in my series several times (unfortunately too many to count) hugepage_global_enabled always evaluated to false when THP is never.

Quoted 7 lines

Never should mean never.

It's a NACK if that's what this is about unless I'm missing something here.

I agree global settings are not fine-grained enough, but 'sys admins refuse to do X so we want to ignore what they do' is... really not right at all.

#19 lore raw
In reply to Usama Arif
Quoted 68 lines · Usama Arif
Quoted 65 lines · Lorenzo Stoakes
Quoted 20 lines · Yafang Shao
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Yafang Shao

The challenge we face is that our system administration team doesn't permit enabling THP globally in production by setting it to "madvise" or "always". As a result, we can only experiment with your feature on our test servers at this stage.

That's a you problem.

perhaps.

Quoted 3 lines

You need to figure out how to influence your
sysadmin team to change their mind; whether it's by talking to their superiors or persuading them directly.

I believe that "practicing" matters more than "talking" or "persuading". I’m surprised your suggestion relies on "talking" ;-)
If I understand correctly, we all agree that "talk is cheap", right?

Quoted 7 lines
Quoted 2 lines

It's not a justification for why
upstream should take this patch.

I believe Johannes has clearly explained the challenges the community is currently facing [0].

[0]. https://lore.kernel.org/linux-mm/[email protected]/

(Sorry to interject on your conversation, but :)

I don't think anybody denies we have issues in configuring this stuff sensibly. A global-only control isn't going to cut it in the real world it seems.

To me as you say yourself, definining the ABI/API here is what really matters, and we're right now inundated with several series all at once (you wait for one bus then 3 come at once... :).

So this I think, should be the question.

I like the idea of just exposing something like madvise(), which is something we're going to maintain indefinitely.

Though any such exposure would in my view would need to be opt-in i.e. have a list of MADV_... options that are accepted, as we'd need to very cautiously determine which are safe from this context.

Of course then this leads to the whole thing (and I really know very little about BPF internals - obviously happy to understand more) of whether we can just use the madvise() code direct or what locking we can do or how all that works.

At any rate, a custom thing that is specific as 'switch mode for mTHP pages of size X to Y' is just something I'd rather us not tie ourselves to.

Quoted 6 lines

--
Regards

Yafang

What do you think re: bpf vs. something like my proposed process_madvise() extensions or Usama's proposed prctl()?

Quoted 6 lines
Quoted 2 lines

Simpler, but really just using madvise functionality and having a means of defaulting across fork/exec (notwithstanding Jann's concerns in this area).

Unfortunately I think the issue is that neither prctl or process_madvise would work for Yafangs usecase? Its usecase 3 mentioned in [1], i.e.
global system policy=never, process wants "madvise" policy for itself.

If the global system policy would be "madvise", you'd need a way to just disable it for processes where you wouldn't ever want them.

Signature · 3 lines
Cheers,

David / dhildenb
#20 lore raw
In reply to Yafang Shao
Quoted 7 lines

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

A new "bpf" mode sounds way too special.

We currently have:

never -> never
madvise -> MADV_HUGEPAGE, except PR_SET_THP_DISABLE always -> always, except PR_SET_THP_DISABLE and MADV_NOHUGEPAGE

Whatever new mode we add, it should honor PR_SET_THP_DISABLE + MADV_NOHUGEPAGE.

So, if we want another way to enable things, it would live between "never" and "madvise".

I'm wondering how we could make that generic: likely we want this new mechanism to *not* be triggerable by the process itself (madvise).

I am not convinced bpf is the answer here ...

Signature · 3 lines
Cheers,

David / dhildenb
#21 lore raw
Quoted 18 lines · David Hildenbrand
Quoted 7 lines

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

A new "bpf" mode sounds way too special.

We currently have:

never -> never
madvise -> MADV_HUGEPAGE, except PR_SET_THP_DISABLE always -> always, except PR_SET_THP_DISABLE and MADV_NOHUGEPAGE

Whatever new mode we add, it should honor PR_SET_THP_DISABLE + MADV_NOHUGEPAGE.

Quoted 7 lines

So, if we want another way to enable things, it would live between "never" and "madvise".

I'm wondering how we could make that generic: likely we want this new mechanism to *not* be triggerable by the process itself (madvise).

I am not convinced bpf is the answer here ...

Agreed.

I am also very concerned with us inserting BPF bits here - are we not then ensuring that we cannot in any way move towards a future where we 'automagically' determine what to do?

I don't know what is claimed about BPF, but it strikes me that we're establishing a permanent uABI (uAPI?) if we do that and essentially promising that THP will continue to operate in a fashion similar to how it does now.

While BPF is a wonderful technology, I thik we have to be very very careful about inserting it in places that consist of -implementation details- that we in mm already are planning to move away from.

It's one thing adding BPF in the oomk (simple interface, unlikely to change, doesn't really constrain us) or the scheduler (again the hooks are by nature reasonably stable), it's quite another sticking it in the heart of a part of mm that is undergoing _constant_ change, partly as evidenced by the sheer number of series related to THP that are currently on-list.

So while BPF may be the best solution for your needs _right now_, we need be concerned with how things affect the kernel in the future.

I think we really do have to tread very carefully here.

Quoted 6 lines

--
Cheers,

David / dhildenb

#22 lore raw
Quoted 49 lines · Lorenzo Stoakes
Quoted 18 lines · David Hildenbrand
Quoted 7 lines

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

A new "bpf" mode sounds way too special.

We currently have:

never -> never
madvise -> MADV_HUGEPAGE, except PR_SET_THP_DISABLE always -> always, except PR_SET_THP_DISABLE and MADV_NOHUGEPAGE

Whatever new mode we add, it should honor PR_SET_THP_DISABLE + MADV_NOHUGEPAGE.

Quoted 7 lines

So, if we want another way to enable things, it would live between "never" and "madvise".

I'm wondering how we could make that generic: likely we want this new mechanism to *not* be triggerable by the process itself (madvise).

I am not convinced bpf is the answer here ...

Agreed.

I am also very concerned with us inserting BPF bits here - are we not then ensuring that we cannot in any way move towards a future where we 'automagically' determine what to do?

I don't know what is claimed about BPF, but it strikes me that we're establishing a permanent uABI (uAPI?) if we do that and essentially promising that THP will continue to operate in a fashion similar to how it does now.

While BPF is a wonderful technology, I thik we have to be very very careful about inserting it in places that consist of -implementation details- that we in mm already are planning to move away from.

It's one thing adding BPF in the oomk (simple interface, unlikely to change, doesn't really constrain us) or the scheduler (again the hooks are by nature reasonably stable), it's quite another sticking it in the heart of a part of mm that is undergoing _constant_ change, partly as evidenced by the sheer number of series related to THP that are currently on-list.

Quoted 4 lines

So while BPF may be the best solution for your needs _right now_, we need be concerned with how things affect the kernel in the future.

I think we really do have to tread very carefully here.

I totally agree with you that the key point here is how to define the API. As I replied to David, I believe we have two fundamental principles to adjust the THP policies:
1. Selective Benefit: Some tasks benefit from THP, while others do not.
2. Conditional Safety: THP allocation is safe under certain conditions but not others.

Therefore, I believe we can define these APIs based on the established principles - everything else constitutes implementation details, even if core MM internals need to change.

Signature · 2 lines
Regards
Yafang
#23 lore raw
Quoted 63 lines · Yafang Shao
Quoted 49 lines · Lorenzo Stoakes
Quoted 18 lines · David Hildenbrand
Quoted 7 lines

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

A new "bpf" mode sounds way too special.

We currently have:

never -> never
madvise -> MADV_HUGEPAGE, except PR_SET_THP_DISABLE always -> always, except PR_SET_THP_DISABLE and MADV_NOHUGEPAGE

Whatever new mode we add, it should honor PR_SET_THP_DISABLE + MADV_NOHUGEPAGE.

Quoted 7 lines

So, if we want another way to enable things, it would live between "never" and "madvise".

I'm wondering how we could make that generic: likely we want this new mechanism to *not* be triggerable by the process itself (madvise).

I am not convinced bpf is the answer here ...

Agreed.

I am also very concerned with us inserting BPF bits here - are we not then ensuring that we cannot in any way move towards a future where we 'automagically' determine what to do?

I don't know what is claimed about BPF, but it strikes me that we're establishing a permanent uABI (uAPI?) if we do that and essentially promising that THP will continue to operate in a fashion similar to how it does now.

While BPF is a wonderful technology, I thik we have to be very very careful about inserting it in places that consist of -implementation details- that we in mm already are planning to move away from.

It's one thing adding BPF in the oomk (simple interface, unlikely to change, doesn't really constrain us) or the scheduler (again the hooks are by nature reasonably stable), it's quite another sticking it in the heart of a part of mm that is undergoing _constant_ change, partly as evidenced by the sheer number of series related to THP that are currently on-list.

Quoted 4 lines

So while BPF may be the best solution for your needs _right now_, we need be concerned with how things affect the kernel in the future.

I think we really do have to tread very carefully here.

I totally agree with you that the key point here is how to define the API. As I replied to David, I believe we have two fundamental principles to adjust the THP policies:
1. Selective Benefit: Some tasks benefit from THP, while others do not.
2. Conditional Safety: THP allocation is safe under certain conditions but not others.

Quoted 3 lines

Therefore, I believe we can define these APIs based on the established principles - everything else constitutes implementation details, even if core MM internals need to change.

But if we're looking to make the concept of THP go away, we really need to go further than this.

The second we have 'bpf program that figures out whether THP should be used' we are permanently tied to the idea of THP on/off being a thing.

I mean any future stuff that makes THP more automagic will probably involve having new modes for the legacy THP /sys/kernel/mm/transparent_hugepage/enabled and /sys/kernel/mm/transparent_hugepage/hugepages-xxkB/enabled

But if people are super reliant on this stuff it's potentially really limiting.

I think you said in another post here that you were toying with the notion of exposing somehow the madvise() interface and having that be the 'stable API' of sorts?

That definitely sounds more sensible than something that very explicitly interacts with THP.

Of course we have Usama's series and my proposed series for extending process_madvise() along those lines also.

Quoted 4 lines

--
Regards
Yafang

#24 lore raw
Quoted 7 lines
Quoted 7 lines

I totally agree with you that the key point here is how to define the API. As I replied to David, I believe we have two fundamental principles to adjust the THP policies:
1. Selective Benefit: Some tasks benefit from THP, while others do not.
2. Conditional Safety: THP allocation is safe under certain conditions but not others.

Quoted 6 lines
Quoted 3 lines

Therefore, I believe we can define these APIs based on the established principles - everything else constitutes implementation details, even if core MM internals need to change.

But if we're looking to make the concept of THP go away, we really need to go further than this.

Yeah. I might be wrong, but I also don't think doing control on a per-process level etc would be the right solution long-term.

In a world where we do stuff automatically ("auto" mode), we would be much smarter about where to place a (m)THP, and which size we would use.

One might use bpf to control the allocation policy. But I don't think this would be per-process or even per-VMA etc. Sure, we might give hints, but placement decisions should happen on another level (e.g., during page faults, during khugepaged etc).

Quoted 8 lines

The second we have 'bpf program that figures out whether THP should be used' we are permanently tied to the idea of THP on/off being a thing.

I mean any future stuff that makes THP more automagic will probably involve having new modes for the legacy THP /sys/kernel/mm/transparent_hugepage/enabled and /sys/kernel/mm/transparent_hugepage/hugepages-xxkB/enabled

Yeah, the plan is to have "auto" in /sys/kernel/mm/transparent_hugepage/enabled and just have all other sizes "inherit" that option. And have a Kconfig that just enables that as default. Once we're there, just phase out the interface long-term.

That's the plan. Now we "only" have to figure out how to make the placement actually better ;)

Quoted 8 lines

But if people are super reliant on this stuff it's potentially really limiting.

I think you said in another post here that you were toying with the notion of exposing somehow the madvise() interface and having that be the 'stable API' of sorts?

Quoted 5 lines

That definitely sounds more sensible than something that very explicitly interacts with THP.

Of course we have Usama's series and my proposed series for extending process_madvise() along those lines also.

Yes.

Signature · 3 lines
Cheers,

David / dhildenb
#25 lore raw
Quoted 12 lines · David Hildenbrand
Quoted 12 lines
Quoted 11 lines

I totally agree with you that the key point here is how to define the API. As I replied to David, I believe we have two fundamental principles to adjust the THP policies:
1. Selective Benefit: Some tasks benefit from THP, while others do not.
2. Conditional Safety: THP allocation is safe under certain conditions but not others.

Therefore, I believe we can define these APIs based on the established principles - everything else constitutes implementation details, even if core MM internals need to change.

Quoted 5 lines
Quoted 2 lines

But if we're looking to make the concept of THP go away, we really need to go further than this.

Yeah. I might be wrong, but I also don't think doing control on a per-process level etc would be the right solution long-term.

The reality is that achieving truly 'automatic' THP behavior requires process-level control. Given that THP provides no benefit for certain workloads, there's no justification for incurring the overhead of allocating higher-order pages in those cases.

Quoted 3 lines

In a world where we do stuff automatically ("auto" mode), we would be much smarter about where to place a (m)THP, and which size we would use.

We still have considerable ground to cover before reaching this goal.

Quoted 5 lines

One might use bpf to control the allocation policy. But I don't think this would be per-process or even per-VMA etc. Sure, we might give hints, but placement decisions should happen on another level (e.g., during page faults, during khugepaged etc).

Nico has proposed introducing a new 'defer' mode to address this. However, I argue that we could achieve the same functionality through BPF instead of adding a dedicated policy mode. [0]

[0]. https://lore.kernel.org/linux-mm/CALOAHbAa7DY6+hO4RJtjg-MS+cnUmsiPXX8KS1MKSfgy6HLYAQ@mail.gmail.com/

Quoted 34 lines
Quoted 9 lines

The second we have 'bpf program that figures out whether THP should be used' we are permanently tied to the idea of THP on/off being a thing.

I mean any future stuff that makes THP more automagic will probably involve having new modes for the legacy THP /sys/kernel/mm/transparent_hugepage/enabled and /sys/kernel/mm/transparent_hugepage/hugepages-xxkB/enabled

Yeah, the plan is to have "auto" in /sys/kernel/mm/transparent_hugepage/enabled and just have all other sizes "inherit" that option. And have a Kconfig that just enables that as default. Once we're there, just phase out the interface long-term.

That's the plan. Now we "only" have to figure out how to make the placement actually better ;)

Quoted 8 lines

But if people are super reliant on this stuff it's potentially really limiting.

I think you said in another post here that you were toying with the notion of exposing somehow the madvise() interface and having that be the 'stable API' of sorts?

Quoted 5 lines

That definitely sounds more sensible than something that very explicitly interacts with THP.

Of course we have Usama's series and my proposed series for extending process_madvise() along those lines also.

Yes.

Signature · 2 lines
Regards
Yafang
#26 lore raw
In reply to Lorenzo Stoakes
Quoted 80 lines · Lorenzo Stoakes
Quoted 63 lines · Yafang Shao
Quoted 49 lines · Lorenzo Stoakes
Quoted 18 lines · David Hildenbrand
Quoted 7 lines

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

A new "bpf" mode sounds way too special.

We currently have:

never -> never
madvise -> MADV_HUGEPAGE, except PR_SET_THP_DISABLE always -> always, except PR_SET_THP_DISABLE and MADV_NOHUGEPAGE

Whatever new mode we add, it should honor PR_SET_THP_DISABLE + MADV_NOHUGEPAGE.

Quoted 7 lines

So, if we want another way to enable things, it would live between "never" and "madvise".

I'm wondering how we could make that generic: likely we want this new mechanism to *not* be triggerable by the process itself (madvise).

I am not convinced bpf is the answer here ...

Agreed.

I am also very concerned with us inserting BPF bits here - are we not then ensuring that we cannot in any way move towards a future where we 'automagically' determine what to do?

I don't know what is claimed about BPF, but it strikes me that we're establishing a permanent uABI (uAPI?) if we do that and essentially promising that THP will continue to operate in a fashion similar to how it does now.

While BPF is a wonderful technology, I thik we have to be very very careful about inserting it in places that consist of -implementation details- that we in mm already are planning to move away from.

It's one thing adding BPF in the oomk (simple interface, unlikely to change, doesn't really constrain us) or the scheduler (again the hooks are by nature reasonably stable), it's quite another sticking it in the heart of a part of mm that is undergoing _constant_ change, partly as evidenced by the sheer number of series related to THP that are currently on-list.

Quoted 4 lines

So while BPF may be the best solution for your needs _right now_, we need be concerned with how things affect the kernel in the future.

I think we really do have to tread very carefully here.

I totally agree with you that the key point here is how to define the API. As I replied to David, I believe we have two fundamental principles to adjust the THP policies:
1. Selective Benefit: Some tasks benefit from THP, while others do not.
2. Conditional Safety: THP allocation is safe under certain conditions but not others.

Quoted 3 lines

Therefore, I believe we can define these APIs based on the established principles - everything else constitutes implementation details, even if core MM internals need to change.

But if we're looking to make the concept of THP go away, we really need to go further than this.

The second we have 'bpf program that figures out whether THP should be used' we are permanently tied to the idea of THP on/off being a thing.

I mean any future stuff that makes THP more automagic will probably involve having new modes for the legacy THP /sys/kernel/mm/transparent_hugepage/enabled and /sys/kernel/mm/transparent_hugepage/hugepages-xxkB/enabled

Quoted 6 lines

But if people are super reliant on this stuff it's potentially really limiting.

I think you said in another post here that you were toying with the notion of exposing somehow the madvise() interface and having that be the 'stable API' of sorts?

Yes, I have a BPF program that hooks into madvise() to selectively enforce THP policies—allowing it for certain tasks while blocking it for others. However, this violates the semantic guarantee of madvise(). For instance, if a user sees THP configured in madvise mode, they’d expect madvise() to reliably enable it. But with this BPF logic, such calls might silently fail, creating inconsistency. This is why we propose introducing a dedicated BPF-controlled mode, or alternatively extending the semantics of the existing "never" mode.

Signature · 2 lines
Regards
Yafang
#27 lore raw
In reply to David Hildenbrand
Quoted 9 lines · David Hildenbrand
Quoted 8 lines

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

A new "bpf" mode sounds way too special.

Alternatively, we could simply hook 'madvise' to define a BPF-based policy.

Quoted 6 lines

We currently have:

never -> never
madvise -> MADV_HUGEPAGE, except PR_SET_THP_DISABLE always -> always, except PR_SET_THP_DISABLE and MADV_NOHUGEPAGE

If BPF had been invented before THP, we likely would have only three modes—without PR_SET_THP_DISABLE, MADV_NOHUGEPAGE, or MADV_HUGEPAGE;-)

never -> never
user -> user defined per task or per vma THP mode selector, based on BPF
We can select "never" or "always" for a specific task or vma The API is as follows, bpf->per_task_mode_selector(task); bpf->per_vma_mode_selecor(vma);
always -> always

However, it’s not too late to introduce a new BPF-based mode for THP, especially since future adjustments to THP policies are still expected. Regardless of the specific policy, two fundamental principles apply:
1. Selective Benefit: Some tasks benefit from THP, while others do not.
2. Conditional Safety: THP allocation is safe under certain conditions but not others.

Given these constraints, we could abstract stable APIs that allow users to define custom THP policies tailored to their needs.

Quoted 3 lines

Whatever new mode we add, it should honor PR_SET_THP_DISABLE + MADV_NOHUGEPAGE.

Yes, the BPF only selects different THP modes for different tasks, nothing else won't be changed.

Quoted 3 lines

So, if we want another way to enable things, it would live between "never" and "madvise".

Yes, BPF only selects the appropriate THP mode for each task—nothing else is modified.

Quoted 5 lines

I'm wondering how we could make that generic: likely we want this new mechanism to *not* be triggerable by the process itself (madvise).

I am not convinced bpf is the answer here ...

I believe the key insight is that we should define a generic, stable API for BPF-based THP mode selection.

--
Regards
Yafang

#28 lore raw
In reply to Yafang Shao
Quoted 102 lines · Yafang Shao

Background
----------

At my current employer, PDD, we have consistently configured THP to "never" on our production servers due to past incidents caused by its behavior:

- Increased memory consumption
THP significantly raises overall memory usage.

- Latency spikes
Random latency spikes occur due to more frequent memory compaction activity triggered by THP.

These issues have made sysadmins hesitant to switch to "madvise" or "always" modes.

New Motivation
--------------

We have now identified that certain AI workloads achieve substantial performance gains with THP enabled. However, we’ve also verified that some workloads see little to no benefit—or are even negatively impacted—by THP.

In our Kubernetes environment, we deploy mixed workloads on a single server to maximize resource utilization. Our goal is to selectively enable THP for services that benefit from it while keeping it disabled for others. This approach allows us to incrementally enable THP for additional services and assess how to make it more viable in production.

Proposed Solution
-----------------

For this use case, Johannes suggested introducing a dedicated mode [0]. In this new mode, we could implement BPF-based THP adjustment for fine-grained control over tasks or cgroups. If no BPF program is attached, THP remains in "never" mode. This solution elegantly meets our needs while avoiding the complexity of managing BPF alongside other THP modes.

A selftest example demonstrates how to enable THP for the current task while keeping it disabled for others.

Alternative Proposals
---------------------

- Gutierrez’s cgroup-based approach [1]
- Proposed adding a new cgroup file to control THP policy.
- However, as Johannes noted, cgroups are designed for hierarchical resource allocation, not arbitrary policy settings [2].

- Usama’s per-task THP proposal based on prctl() [3]:
- Enabling THP per task via prctl().
- As David pointed out, neither madvise() nor prctl() works in "never" mode [4], making this solution insufficient for our needs.

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

This is currently a PoC implementation. Feedback of any kind is welcome.

RFC v1->v2:
The main changes are as follows,
- Use struct_ops instead of fmod_ret (Alexei)
- Introduce a new THP mode (Johannes)
- Introduce new helpers for BPF hook (Zi)
- Refine the commit log

RFC v1: https://lwn.net/Articles/1019290/

Yafang Shao (5):
mm: thp: Add a new mode "bpf"
mm: thp: Add hook for BPF based THP adjustment
mm: thp: add struct ops for BPF based THP adjustment
bpf: Add get_current_comm to bpf_base_func_proto selftests/bpf: Add selftest for THP adjustment

 include/linux/huge_mm.h                       |  15 +-
 kernel/bpf/cgroup.c                           |   2 -
 kernel/bpf/helpers.c                          |   2 +
 mm/Makefile                                   |   3 +
 mm/bpf_thp.c                                  | 120 ++++++++++++
 mm/huge_memory.c                              |  65 ++++++-
 mm/khugepaged.c                               |   3 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 175 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  39 ++++
 10 files changed, 414 insertions(+), 11 deletions(-)
 create mode 100644 mm/bpf_thp.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
Quoted 3 lines

--
2.43.5

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;) Both Lorenzo and David propose relying on the madvise mode. However, since madvise is an unprivileged userspace mechanism, any user can freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.
Given these limitations, the community should consider introducing a new "admin" mode for privileged THP policy management.

- Can the Kernel Automatically Manage THP Without User Input? In practice, users define their own success metrics—such as latency (RT), queries per second (QPS), or throughput—to evaluate a feature’s usefulness. If a feature fails to improve these metrics, it provides no practical value.
Currently, the kernel lacks visibility into user-defined metrics, making fully automated optimization impossible (at least without user input). More importantly, automatic management offers no benefit if it doesn’t align with user needs.
Exception: For kernel-enforced changes (e.g., the page-to-folio transition), users must adapt regardless. But THP tuning requires flexibility—forcing automation without measurable gains is counterproductive.
(Please correct me if I’ve overlooked anything.)

Signature · 2 lines
Regards
Yafang
#29 lore raw
Quoted 127 lines · Yafang Shao
Quoted 102 lines · Yafang Shao

Background
----------

At my current employer, PDD, we have consistently configured THP to "never" on our production servers due to past incidents caused by its behavior:

- Increased memory consumption
THP significantly raises overall memory usage.

- Latency spikes
Random latency spikes occur due to more frequent memory compaction activity triggered by THP.

These issues have made sysadmins hesitant to switch to "madvise" or "always" modes.

New Motivation
--------------

We have now identified that certain AI workloads achieve substantial performance gains with THP enabled. However, we’ve also verified that some workloads see little to no benefit—or are even negatively impacted—by THP.

In our Kubernetes environment, we deploy mixed workloads on a single server to maximize resource utilization. Our goal is to selectively enable THP for services that benefit from it while keeping it disabled for others. This approach allows us to incrementally enable THP for additional services and assess how to make it more viable in production.

Proposed Solution
-----------------

For this use case, Johannes suggested introducing a dedicated mode [0]. In this new mode, we could implement BPF-based THP adjustment for fine-grained control over tasks or cgroups. If no BPF program is attached, THP remains in "never" mode. This solution elegantly meets our needs while avoiding the complexity of managing BPF alongside other THP modes.

A selftest example demonstrates how to enable THP for the current task while keeping it disabled for others.

Alternative Proposals
---------------------

- Gutierrez’s cgroup-based approach [1]
- Proposed adding a new cgroup file to control THP policy.
- However, as Johannes noted, cgroups are designed for hierarchical resource allocation, not arbitrary policy settings [2].

- Usama’s per-task THP proposal based on prctl() [3]:
- Enabling THP per task via prctl().
- As David pointed out, neither madvise() nor prctl() works in "never" mode [4], making this solution insufficient for our needs.

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

This is currently a PoC implementation. Feedback of any kind is welcome.

RFC v1->v2:
The main changes are as follows,
- Use struct_ops instead of fmod_ret (Alexei)
- Introduce a new THP mode (Johannes)
- Introduce new helpers for BPF hook (Zi)
- Refine the commit log

RFC v1: https://lwn.net/Articles/1019290/

Yafang Shao (5):
mm: thp: Add a new mode "bpf"
mm: thp: Add hook for BPF based THP adjustment
mm: thp: add struct ops for BPF based THP adjustment
bpf: Add get_current_comm to bpf_base_func_proto selftests/bpf: Add selftest for THP adjustment

 include/linux/huge_mm.h                       |  15 +-
 kernel/bpf/cgroup.c                           |   2 -
 kernel/bpf/helpers.c                          |   2 +
 mm/Makefile                                   |   3 +
 mm/bpf_thp.c                                  | 120 ++++++++++++
 mm/huge_memory.c                              |  65 ++++++-
 mm/khugepaged.c                               |   3 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 175 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  39 ++++
 10 files changed, 414 insertions(+), 11 deletions(-)
 create mode 100644 mm/bpf_thp.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
Quoted 3 lines

--
2.43.5

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;) Both Lorenzo and David propose relying on the madvise mode. However, since madvise is an unprivileged userspace mechanism, any user can freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.
Given these limitations, the community should consider introducing a new "admin" mode for privileged THP policy management.

Quoted 9 lines

- Can the Kernel Automatically Manage THP Without User Input? In practice, users define their own success metrics—such as latency (RT), queries per second (QPS), or throughput—to evaluate a feature’s usefulness. If a feature fails to improve these metrics, it provides no practical value.
Currently, the kernel lacks visibility into user-defined metrics, making fully automated optimization impossible (at least without user input). More importantly, automatic management offers no benefit if it doesn’t align with user needs.

I don't think that using things like RPS or QPS is the right way. These metrics can be affected by many factors like network issues, garbage collectors in the user space (JVM, golang, etc.) and many other things out of our control. Even noisy neighbors can slow down a service.

Quoted 6 lines

Exception: For kernel-enforced changes (e.g., the page-to-folio transition), users must adapt regardless. But THP tuning requires flexibility—forcing automation without measurable gains is counterproductive.
(Please correct me if I’ve overlooked anything.)

Signature · 2 lines
Asier Gutierrez
Huawei
#30 lore raw
Quoted 141 lines · Gutierrez Asier
Quoted 127 lines · Yafang Shao
Quoted 102 lines · Yafang Shao

Background
----------

At my current employer, PDD, we have consistently configured THP to "never" on our production servers due to past incidents caused by its behavior:

- Increased memory consumption
THP significantly raises overall memory usage.

- Latency spikes
Random latency spikes occur due to more frequent memory compaction activity triggered by THP.

These issues have made sysadmins hesitant to switch to "madvise" or "always" modes.

New Motivation
--------------

We have now identified that certain AI workloads achieve substantial performance gains with THP enabled. However, we’ve also verified that some workloads see little to no benefit—or are even negatively impacted—by THP.

In our Kubernetes environment, we deploy mixed workloads on a single server to maximize resource utilization. Our goal is to selectively enable THP for services that benefit from it while keeping it disabled for others. This approach allows us to incrementally enable THP for additional services and assess how to make it more viable in production.

Proposed Solution
-----------------

For this use case, Johannes suggested introducing a dedicated mode [0]. In this new mode, we could implement BPF-based THP adjustment for fine-grained control over tasks or cgroups. If no BPF program is attached, THP remains in "never" mode. This solution elegantly meets our needs while avoiding the complexity of managing BPF alongside other THP modes.

A selftest example demonstrates how to enable THP for the current task while keeping it disabled for others.

Alternative Proposals
---------------------

- Gutierrez’s cgroup-based approach [1]
- Proposed adding a new cgroup file to control THP policy.
- However, as Johannes noted, cgroups are designed for hierarchical resource allocation, not arbitrary policy settings [2].

- Usama’s per-task THP proposal based on prctl() [3]:
- Enabling THP per task via prctl().
- As David pointed out, neither madvise() nor prctl() works in "never" mode [4], making this solution insufficient for our needs.

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

This is currently a PoC implementation. Feedback of any kind is welcome.

RFC v1->v2:
The main changes are as follows,
- Use struct_ops instead of fmod_ret (Alexei)
- Introduce a new THP mode (Johannes)
- Introduce new helpers for BPF hook (Zi)
- Refine the commit log

RFC v1: https://lwn.net/Articles/1019290/

Yafang Shao (5):
mm: thp: Add a new mode "bpf"
mm: thp: Add hook for BPF based THP adjustment
mm: thp: add struct ops for BPF based THP adjustment
bpf: Add get_current_comm to bpf_base_func_proto selftests/bpf: Add selftest for THP adjustment

 include/linux/huge_mm.h                       |  15 +-
 kernel/bpf/cgroup.c                           |   2 -
 kernel/bpf/helpers.c                          |   2 +
 mm/Makefile                                   |   3 +
 mm/bpf_thp.c                                  | 120 ++++++++++++
 mm/huge_memory.c                              |  65 ++++++-
 mm/khugepaged.c                               |   3 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 175 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  39 ++++
 10 files changed, 414 insertions(+), 11 deletions(-)
 create mode 100644 mm/bpf_thp.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
Quoted 3 lines

--
2.43.5

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;) Both Lorenzo and David propose relying on the madvise mode. However, since madvise is an unprivileged userspace mechanism, any user can freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.
Given these limitations, the community should consider introducing a new "admin" mode for privileged THP policy management.

Quoted 9 lines

- Can the Kernel Automatically Manage THP Without User Input? In practice, users define their own success metrics—such as latency (RT), queries per second (QPS), or throughput—to evaluate a feature’s usefulness. If a feature fails to improve these metrics, it provides no practical value.
Currently, the kernel lacks visibility into user-defined metrics, making fully automated optimization impossible (at least without user input). More importantly, automatic management offers no benefit if it doesn’t align with user needs.

Quoted 4 lines

I don't think that using things like RPS or QPS is the right way. These metrics can be affected by many factors like network issues, garbage collectors in the user space (JVM, golang, etc.) and many other things out of our control. Even noisy neighbors can slow down a service.

This is an example of how to measure whether apps can benefit from a new feature.
Please review the A/B test details here:
https://en.wikipedia.org/wiki/A/B_testing

Signature · 2 lines
Regards
Yafang
#31 lore raw
In reply to Yafang Shao
Quoted 5 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

Quoted 7 lines

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

I guess it really depends on how different the workloads are that you are running on the same machine.

Quoted 4 lines

Both Lorenzo and David propose relying on the madvise mode. However,> since madvise is an unprivileged userspace mechanism, any user can freely adjust their THP policy. This makes fine-grained control
impossible without breaking userspace compatibility—an undesirable
tradeoff.

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

The could be added on top of the current proposals that are flying around, and could be done e.g., per-process.

Signature · 3 lines
Cheers,

David / dhildenb
#32 lore raw
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 10 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

Quoted 3 lines

The could be added on top of the current proposals that are flying around, and could be done e.g., per-process.

How about introducing a dedicated "process" mode? This would allow each process to use different THP modes—some in "always," others in "madvise," and the rest in "never." Future THP modes could also be added to this framework.

Signature · 2 lines
Regards
Yafang
#33 lore raw
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

Quoted 8 lines
Quoted 3 lines

The could be added on top of the current proposals that are flying around, and could be done e.g., per-process.

How about introducing a dedicated "process" mode? This would allow each process to use different THP modes—some in "always," others in "madvise," and the rest in "never." Future THP modes could also be added to this framework.

We have to be really careful about not creating even more mess with more modes.

How would that design look like in detail (how would we set it per process etc?)?

Signature · 3 lines
Cheers,

David / dhildenb
#34 lore raw
Quoted 46 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Quoted 7 lines

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

I think this works. Take the example from a previous thread where containers are differentiated by allowing or not allowing THP. If you set a container MADV_HOHUGEPAGE (or whatever flag we used for the same meaning), then if a library uses that call and it fails do we want to report it as a failure? I would reason that the library shouldn't hard fail if its unable to use THP, so it's okay to return the failure.

Alternatively, if it is a hard requirement, then that container shouldn't be allowed to continue in such a state and should verify the return. (If this is even a possibility?)

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

The could be added on top of the current proposals that are flying around, and could be done e.g., per-process.

Quoted 7 lines
Quoted 4 lines

How about introducing a dedicated "process" mode? This would allow each process to use different THP modes—some in "always," others in "madvise," and the rest in "never." Future THP modes could also be added to this framework.

We have to be really careful about not creating even more mess with more modes.

Yes, and clarity would depend on the mode name, imo. Never meaning never, for example.

So we'd need an answer to David's question below before agreeing on "process". If it survives across fork and exec calls, is it really a "process" setting?

I believe you are seeing it as "setting default" really doesn't mean setting a default if you cannot overwrite it, and if you can overwrite the "default" then it's not going to work for all use cases.

Quoted 9 lines

How would that design look like in detail (how would we set it per process etc?)?

Signature · 3 lines
Cheers,

David / dhildenb
#35 lore raw
Quoted 47 lines · Liam R. Howlett
Quoted 46 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Quoted 7 lines
Quoted 7 lines

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

I am also not entirely sure how sealing a non-existing vma would work. We'd have to seal the default flags, but sealing is one way and this surely shouldn't be one way?

Quoted 45 lines

I think this works. Take the example from a previous thread where containers are differentiated by allowing or not allowing THP. If you set a container MADV_HOHUGEPAGE (or whatever flag we used for the same meaning), then if a library uses that call and it fails do we want to report it as a failure? I would reason that the library shouldn't hard fail if its unable to use THP, so it's okay to return the failure.

Alternatively, if it is a hard requirement, then that container shouldn't be allowed to continue in such a state and should verify the return. (If this is even a possibility?)

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

The could be added on top of the current proposals that are flying around, and could be done e.g., per-process.

Quoted 7 lines
Quoted 4 lines

How about introducing a dedicated "process" mode? This would allow each process to use different THP modes—some in "always," others in "madvise," and the rest in "never." Future THP modes could also be added to this framework.

We have to be really careful about not creating even more mess with more modes.

Yes, and clarity would depend on the mode name, imo. Never meaning never, for example.

So we'd need an answer to David's question below before agreeing on "process". If it survives across fork and exec calls, is it really a "process" setting?

I believe you are seeing it as "setting default" really doesn't mean setting a default if you cannot overwrite it, and if you can overwrite the "default" then it's not going to work for all use cases.

Quoted 9 lines

How would that design look like in detail (how would we set it per process etc?)?

Signature · 3 lines
Cheers,

David / dhildenb
#36 lore raw
Quoted 51 lines · Liam R. Howlett
Quoted 50 lines · Liam R. Howlett
Quoted 49 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

I am also not entirely sure how sealing a non-existing vma would work. We'd have to seal the default flags, but sealing is one way and this surely shouldn't be one way?

You probably have mseal() in mind. Just like we wouldn't be using madvise(), we also wouldn't be using mseal().

It could be a simple mctrl()/whatever option/flag to set the default and no longer allow changing the default and per-VMA flags, unless CAP_SYS_ADMIN or sth like that.

Signature · 3 lines
Cheers,

David / dhildenb
#37 lore raw
Quoted 57 lines · David Hildenbrand
Quoted 56 lines · Liam R. Howlett
Quoted 54 lines · Liam R. Howlett
Quoted 53 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

Quoted 6 lines
Quoted 3 lines

I am also not entirely sure how sealing a non-existing vma would work. We'd have to seal the default flags, but sealing is one way and this surely shouldn't be one way?

You probably have mseal() in mind. Just like we wouldn't be using madvise(), we also wouldn't be using mseal().

yes, I do - but mostly in terms of the language used and not the code as that can't be used here.

Do we use the term seal somewhere that allows undoing the sealed thing? I'm _really_ hoping we don't, but am almost sure you're going to say we do.

Quoted 4 lines

It could be a simple mctrl()/whatever option/flag to set the default and no longer allow changing the default and per-VMA flags, unless CAP_SYS_ADMIN or sth like that.

Ah... okay, as long as we have testing I guess. I can see that getting confusing.

Cheers,
Liam

#38 lore raw
Quoted 65 lines · Liam R. Howlett
Quoted 57 lines · David Hildenbrand
Quoted 56 lines · Liam R. Howlett
Quoted 54 lines · Liam R. Howlett
Quoted 53 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

Quoted 6 lines
Quoted 3 lines

I am also not entirely sure how sealing a non-existing vma would work. We'd have to seal the default flags, but sealing is one way and this surely shouldn't be one way?

You probably have mseal() in mind. Just like we wouldn't be using madvise(), we also wouldn't be using mseal().

Quoted 6 lines

yes, I do - but mostly in terms of the language used and not the code as that can't be used here.

Do we use the term seal somewhere that allows undoing the sealed thing? I'm _really_ hoping we don't, but am almost sure you're going to say we do.

The other place that comes to mind is memfd_create() with MFD_ALLOW_SEALING and fcntl() with F_ADD_SEALS.

And yes, there is no "F_DEL_SEALS" :)

Signature · 3 lines
Cheers,

David / dhildenb
#39 lore raw
In reply to David Hildenbrand
Quoted 57 lines · David Hildenbrand
Quoted 56 lines · Liam R. Howlett
Quoted 54 lines · Liam R. Howlett
Quoted 53 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines

   > Both Lorenzo and David propose relying on the madvise mode. However,> since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

Quoted 8 lines
Quoted 3 lines

I am also not entirely sure how sealing a non-existing vma would work. We'd have to seal the default flags, but sealing is one way and this surely shouldn't be one way?

You probably have  mseal() in mind. Just like we wouldn't be using madvise(), we also wouldn't be using mseal().

It could be a simple mctrl()/whatever option/flag to set the default and no longer allow changing the default and per-VMA flags, unless CAP_SYS_ADMIN or sth like that.

This isn't really TRANSPARENT Huge Pages, since we will require the application to determine which memory range will be mapped with huge pages.

Signature · 2 lines
Asier Gutierrez
Huawei
#40 lore raw
Quoted 64 lines · Gutierrez Asier
Quoted 61 lines · David Hildenbrand
Quoted 59 lines · Liam R. Howlett
Quoted 47 lines · Liam R. Howlett
Quoted 46 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines

   > Both Lorenzo and David propose relying on the madvise mode. However,>
since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Quoted 7 lines
Quoted 7 lines

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

I am also not entirely sure how sealing a non-existing vma would work. We'd have to seal the default flags, but sealing is one way and this surely shouldn't be one way?

Quoted 8 lines
Quoted 4 lines

You probably have  mseal() in mind. Just like we wouldn't be using madvise(), we also wouldn't be using mseal().

It could be a simple mctrl()/whatever option/flag to set the default and no longer allow changing the default and per-VMA flags, unless CAP_SYS_ADMIN or sth like that.

This isn't really TRANSPARENT Huge Pages, since we will require the application to determine which memory range will be mapped with huge pages.

Huh? No idea how you concluded that. Can you elaborate?

Signature · 3 lines
Cheers,

David / dhildenb
#41 lore raw
In reply to David Hildenbrand
Quoted 64 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

Quoted 8 lines
Quoted 3 lines

The could be added on top of the current proposals that are flying around, and could be done e.g., per-process.

How about introducing a dedicated "process" mode? This would allow each process to use different THP modes—some in "always," others in "madvise," and the rest in "never." Future THP modes could also be added to this framework.

Quoted 5 lines

We have to be really careful about not creating even more mess with more modes.

How would that design look like in detail (how would we set it per process etc?)?

I have a preliminary idea to implement this using BPF. We could define the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.
This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

As Liam pointed out in another thread, naming is challenging here - "process" might not be the most accurate term for this context.

Signature · 2 lines
Regards
Yafang
#42 lore raw
Quoted 65 lines · Yafang Shao
Quoted 64 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

Quoted 8 lines
Quoted 3 lines

The could be added on top of the current proposals that are flying around, and could be done e.g., per-process.

How about introducing a dedicated "process" mode? This would allow each process to use different THP modes—some in "always," others in "madvise," and the rest in "never." Future THP modes could also be added to this framework.

Quoted 7 lines
Quoted 5 lines

We have to be really careful about not creating even more mess with more modes.

How would that design look like in detail (how would we set it per process etc?)?

I have a preliminary idea to implement this using BPF.

I don't think we want to add such a mechanism (new mode) where the primary configuration mechanism is through bpf.

Maybe bpf could be used as an alternative, but we should look into a reasonable alternative first, like the discussed mctrl()/.../ raised in the process_madvise() series.

No "bpf" mode in disguise, please :)

Quoted 26 lines

We could define
the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never
Quoted 5 lines

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

Quoted 5 lines

This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

I am not sure that is the proper way to handle these scenarios: I never heard that people would be adjusting the system-wide policy dynamically in that way either.

Whatever we do, we have to make sure that what we add won't over-complicate things in the future. Having tooling dynamically adjust the THP policy of processes that coarsely sounds ... very wrong long-term.

Quoted 2 lines
Quoted 1 line

As Liam pointed out in another thread, naming is challenging here -

"process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be used by multiple processes ...

Signature · 3 lines
Cheers,

David / dhildenb
#43 lore raw
Quoted 75 lines · David Hildenbrand
Quoted 65 lines · Yafang Shao
Quoted 64 lines · David Hildenbrand
Quoted 35 lines · Yafang Shao
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 4 lines
Quoted 2 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;)

IIRC, RHEL9 has the default set to "always" for a long time.

good to know.

Quoted 3 lines

I guess it really depends on how different the workloads are that you are running on the same machine.

Correct. If we want to enable THP for specific workloads without modifying the kernel, we must isolate them on dedicated servers. However, this approach wastes resources and is not an acceptable solution.

Quoted 7 lines
Quoted 2 lines

Both Lorenzo and David propose relying on the madvise mode. However,>

since madvise is an unprivileged userspace mechanism, any user can

Quoted 3 lines

freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.

Quoted 7 lines
Quoted 3 lines

If required, we could look into a "sealing" mechanism, that would essentially lock modification attempts performed by the process (i.e., MADV_HUGEPAGE).

If we don’t introduce a new THP mode and instead rely solely on madvise, the "sealing" mechanism could either violate the intended semantics of madvise(), or simply break madvise() entirely, right?

We would have to be a bit careful, yes.

Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because these options also fail with -EINVAL on kernels without THP support.

Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.

What you likely really want to do is seal when you configured MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.

Quoted 8 lines
Quoted 3 lines

The could be added on top of the current proposals that are flying around, and could be done e.g., per-process.

How about introducing a dedicated "process" mode? This would allow each process to use different THP modes—some in "always," others in "madvise," and the rest in "never." Future THP modes could also be added to this framework.

Quoted 7 lines
Quoted 5 lines

We have to be really careful about not creating even more mess with more modes.

How would that design look like in detail (how would we set it per process etc?)?

I have a preliminary idea to implement this using BPF.

Quoted 8 lines

I don't think we want to add such a mechanism (new mode) where the primary configuration mechanism is through bpf.

Maybe bpf could be used as an alternative, but we should look into a reasonable alternative first, like the discussed mctrl()/.../ raised in the process_madvise() series.

No "bpf" mode in disguise, please :)

This goal can be readily achieved using a BPF program. In any case, it is a feasible solution.

Quoted 41 lines
Quoted 27 lines

We could define
the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never
Quoted 5 lines

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

Quoted 5 lines

This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

Quoted 7 lines

I am not sure that is the proper way to handle these scenarios: I never heard that people would be adjusting the system-wide policy dynamically in that way either.

Whatever we do, we have to make sure that what we add won't over-complicate things in the future. Having tooling dynamically adjust the THP policy of processes that coarsely sounds ... very wrong long-term.

This is just an example demonstrating how BPF can be used to adjust its flexibility. Notably, all these policies can be implemented without modifying the kernel.

Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

As Liam pointed out in another thread, naming is challenging here -

"process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be used by multiple processes ...

I consistently use 'thread' for the latter case. Additionally, this can be implemented per-MM without kernel code modifications. With a well-designed API, users can even implement custom THP policies—all without altering kernel code.

Signature · 2 lines
Regards
Yafang
#44 lore raw
Quoted 11 lines
Quoted 8 lines

I don't think we want to add such a mechanism (new mode) where the primary configuration mechanism is through bpf.

Maybe bpf could be used as an alternative, but we should look into a reasonable alternative first, like the discussed mctrl()/.../ raised in the process_madvise() series.

No "bpf" mode in disguise, please :)

This goal can be readily achieved using a BPF program. In any case, it is a feasible solution.

No BPF-only solution.

Quoted 46 lines
Quoted 46 lines
Quoted 28 lines

We could define
the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never
Quoted 5 lines

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

Quoted 5 lines

This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

I am not sure that is the proper way to handle these scenarios: I never heard that people would be adjusting the system-wide policy dynamically in that way either.

Quoted 7 lines
Quoted 3 lines

Whatever we do, we have to make sure that what we add won't over-complicate things in the future. Having tooling dynamically adjust the THP policy of processes that coarsely sounds ... very wrong long-term.

This is just an example demonstrating how BPF can be used to adjust its flexibility. Notably, all these policies can be implemented without modifying the kernel.

See below on "policy".

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

As Liam pointed out in another thread, naming is challenging here -

"process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be used by multiple processes ...

I consistently use 'thread' for the latter case.

You can use CLONE_VM without CLONE_THREAD ...

Additionally, this

Quoted 3 lines

can be implemented per-MM without kernel code modifications. With a well-designed API, users can even implement custom THP policies—all without altering kernel code.

You can switch between modes, that' all you can do. I wouldn't really call that "custom policy" as it is extremely limited.

And that's exactly my point: it's basic switching between modes ... a reasonable policy in the future will make placement decisions and not just state "always/never/madvise".

Signature · 3 lines
Cheers,

David / dhildenb
#45 lore raw
Quoted 74 lines · David Hildenbrand
Quoted 4 lines
Quoted 4 lines

I don't think we want to add such a mechanism (new mode) where the primary configuration mechanism is through bpf.

Quoted 8 lines
Quoted 5 lines

Maybe bpf could be used as an alternative, but we should look into a reasonable alternative first, like the discussed mctrl()/.../ raised in the process_madvise() series.

No "bpf" mode in disguise, please :)

This goal can be readily achieved using a BPF program. In any case, it is a feasible solution.

No BPF-only solution.

Quoted 46 lines
Quoted 46 lines
Quoted 28 lines

We could define
the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never
Quoted 5 lines

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

Quoted 5 lines

This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

I am not sure that is the proper way to handle these scenarios: I never heard that people would be adjusting the system-wide policy dynamically in that way either.

Quoted 7 lines
Quoted 3 lines

Whatever we do, we have to make sure that what we add won't over-complicate things in the future. Having tooling dynamically adjust the THP policy of processes that coarsely sounds ... very wrong long-term.

This is just an example demonstrating how BPF can be used to adjust its flexibility. Notably, all these policies can be implemented without modifying the kernel.

See below on "policy".

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

As Liam pointed out in another thread, naming is challenging here -

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

"process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be used by multiple processes ...

I consistently use 'thread' for the latter case.

You can use CLONE_VM without CLONE_THREAD ...

If I understand correctly, this can only occur for shared THP but not anonymous THP. For instance, if either process allocates an anonymous THP, it would trigger the creation of a new MM. Please correct me if I'm mistaken.

Quoted 6 lines

Additionally, this

Quoted 3 lines

can be implemented per-MM without kernel code modifications. With a well-designed API, users can even implement custom THP policies—all without altering kernel code.

Quoted 6 lines

You can switch between modes, that' all you can do. I wouldn't really call that "custom policy" as it is extremely limited.

And that's exactly my point: it's basic switching between modes ... a reasonable policy in the future will make placement decisions and not just state "always/never/madvise".

Could you please elaborate further on 'make placement decisions'? As previously mentioned, we (including the broader community) really need the user input to determine whether THP allocation is appropriate in a given case.

Signature · 2 lines
Regards
Yafang
#46 lore raw
Quoted 80 lines · Yafang Shao
Quoted 79 lines · David Hildenbrand
Quoted 4 lines
Quoted 4 lines

I don't think we want to add such a mechanism (new mode) where the primary configuration mechanism is through bpf.

Quoted 8 lines
Quoted 5 lines

Maybe bpf could be used as an alternative, but we should look into a reasonable alternative first, like the discussed mctrl()/.../ raised in the process_madvise() series.

No "bpf" mode in disguise, please :)

This goal can be readily achieved using a BPF program. In any case, it is a feasible solution.

No BPF-only solution.

Quoted 46 lines
Quoted 46 lines
Quoted 28 lines

We could define
the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never
Quoted 5 lines

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

Quoted 5 lines

This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

I am not sure that is the proper way to handle these scenarios: I never heard that people would be adjusting the system-wide policy dynamically in that way either.

Quoted 7 lines
Quoted 3 lines

Whatever we do, we have to make sure that what we add won't over-complicate things in the future. Having tooling dynamically adjust the THP policy of processes that coarsely sounds ... very wrong long-term.

This is just an example demonstrating how BPF can be used to adjust its flexibility. Notably, all these policies can be implemented without modifying the kernel.

See below on "policy".

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

As Liam pointed out in another thread, naming is challenging here -

"process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be used by multiple processes ...

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I consistently use 'thread' for the latter case.

You can use CLONE_VM without CLONE_THREAD ...

If I understand correctly, this can only occur for shared THP but not anonymous THP. For instance, if either process allocates an anonymous THP, it would trigger the creation of a new MM. Please correct me if I'm mistaken.

What clone(CLONE_VM) will do is essentially create a new process, that shares the MM with the original process. Similar to a thread, just that the new process will show up in /proc/ as ... a new process, not as a thread under /prod/$pid/tasks of the original process.

Both processes will operate on the shared MM struct as if they were ordinary threads. No Copy-on-Write involved.

One example use case I've been involved in is async teardown in QEMU [1].

[1] https://kvm-forum.qemu.org/2022/ibm_async_destroy.pdf

Quoted 10 lines
Quoted 10 lines

Additionally, this

Quoted 3 lines

can be implemented per-MM without kernel code modifications. With a well-designed API, users can even implement custom THP policies—all without altering kernel code.

You can switch between modes, that' all you can do. I wouldn't really call that "custom policy" as it is extremely limited.

Quoted 8 lines
Quoted 3 lines

And that's exactly my point: it's basic switching between modes ... a reasonable policy in the future will make placement decisions and not just state "always/never/madvise".

Could you please elaborate further on 'make placement decisions'? As previously mentioned, we (including the broader community) really need the user input to determine whether THP allocation is appropriate in a given case.

The glorious future were we make smarter decisions where to actually place THPs even in the "always" mode.

E.g., just because we enable "always" for a process does not mean that we really want a THP everywhere; quite the opposite.

Treat the "always"/"madvise"/"never" as a rough mode, not a future-proof policy that we would want to fine-tune dynamically ... that would be very limiting.

Signature · 3 lines
Cheers,

David / dhildenb
#47 lore raw
Quoted 96 lines · David Hildenbrand
Quoted 80 lines · Yafang Shao
Quoted 79 lines · David Hildenbrand
Quoted 4 lines
Quoted 4 lines

I don't think we want to add such a mechanism (new mode) where the primary configuration mechanism is through bpf.

Quoted 8 lines
Quoted 5 lines

Maybe bpf could be used as an alternative, but we should look into a reasonable alternative first, like the discussed mctrl()/.../ raised in the process_madvise() series.

No "bpf" mode in disguise, please :)

This goal can be readily achieved using a BPF program. In any case, it is a feasible solution.

No BPF-only solution.

Quoted 46 lines
Quoted 46 lines
Quoted 28 lines

We could define
the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never
Quoted 5 lines

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

Quoted 5 lines

This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

I am not sure that is the proper way to handle these scenarios: I never heard that people would be adjusting the system-wide policy dynamically in that way either.

Quoted 7 lines
Quoted 3 lines

Whatever we do, we have to make sure that what we add won't over-complicate things in the future. Having tooling dynamically adjust the THP policy of processes that coarsely sounds ... very wrong long-term.

This is just an example demonstrating how BPF can be used to adjust its flexibility. Notably, all these policies can be implemented without modifying the kernel.

See below on "policy".

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

As Liam pointed out in another thread, naming is challenging here -

"process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be used by multiple processes ...

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I consistently use 'thread' for the latter case.

You can use CLONE_VM without CLONE_THREAD ...

If I understand correctly, this can only occur for shared THP but not anonymous THP. For instance, if either process allocates an anonymous THP, it would trigger the creation of a new MM. Please correct me if I'm mistaken.

What clone(CLONE_VM) will do is essentially create a new process, that shares the MM with the original process. Similar to a thread, just that the new process will show up in /proc/ as ... a new process, not as a thread under /prod/$pid/tasks of the original process.

Quoted 6 lines

Both processes will operate on the shared MM struct as if they were ordinary threads. No Copy-on-Write involved.

One example use case I've been involved in is async teardown in QEMU [1].

[1] https://kvm-forum.qemu.org/2022/ibm_async_destroy.pdf

I understand what you mean, but what I'm really confused about is how this relates to allocating anonymous THP. If either one allocates anon THP, it will definitely create a new MM, right ?

Quoted 20 lines
Quoted 11 lines
Quoted 11 lines

Additionally, this

Quoted 3 lines

can be implemented per-MM without kernel code modifications. With a well-designed API, users can even implement custom THP policies—all without altering kernel code.

You can switch between modes, that' all you can do. I wouldn't really call that "custom policy" as it is extremely limited.

Quoted 8 lines
Quoted 3 lines

And that's exactly my point: it's basic switching between modes ... a reasonable policy in the future will make placement decisions and not just state "always/never/madvise".

Could you please elaborate further on 'make placement decisions'? As previously mentioned, we (including the broader community) really need the user input to determine whether THP allocation is appropriate in a given case.

Quoted 5 lines

The glorious future were we make smarter decisions where to actually place THPs even in the "always" mode.

E.g., just because we enable "always" for a process does not mean that we really want a THP everywhere; quite the opposite.

So 'always' simply means "the system doesn't guarantee THP allocation will succeed" ? If that's the case, we should revisit RFC v1 [0], where we proposed rejecting THP allocations in certain scenarios for specific tasks.

[0] https://lwn.net/Articles/1019290/

Quoted 4 lines

Treat the "always"/"madvise"/"never" as a rough mode, not a future-proof policy that we would want to fine-tune dynamically ... that would be very limiting.

Signature · 2 lines
Regards
Yafang
#48 lore raw
Quoted 100 lines · Yafang Shao
Quoted 99 lines · David Hildenbrand
Quoted 80 lines · Yafang Shao
Quoted 79 lines · David Hildenbrand
Quoted 4 lines
Quoted 4 lines

I don't think we want to add such a mechanism (new mode) where the primary configuration mechanism is through bpf.

Quoted 8 lines
Quoted 5 lines

Maybe bpf could be used as an alternative, but we should look into a reasonable alternative first, like the discussed mctrl()/.../ raised in the process_madvise() series.

No "bpf" mode in disguise, please :)

This goal can be readily achieved using a BPF program. In any case, it is a feasible solution.

No BPF-only solution.

Quoted 46 lines
Quoted 46 lines
Quoted 28 lines

We could define
the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never
Quoted 5 lines

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

Quoted 5 lines

This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

I am not sure that is the proper way to handle these scenarios: I never heard that people would be adjusting the system-wide policy dynamically in that way either.

Quoted 7 lines
Quoted 3 lines

Whatever we do, we have to make sure that what we add won't over-complicate things in the future. Having tooling dynamically adjust the THP policy of processes that coarsely sounds ... very wrong long-term.

This is just an example demonstrating how BPF can be used to adjust its flexibility. Notably, all these policies can be implemented without modifying the kernel.

See below on "policy".

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

As Liam pointed out in another thread, naming is challenging here -

"process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be used by multiple processes ...

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I consistently use 'thread' for the latter case.

You can use CLONE_VM without CLONE_THREAD ...

If I understand correctly, this can only occur for shared THP but not anonymous THP. For instance, if either process allocates an anonymous THP, it would trigger the creation of a new MM. Please correct me if I'm mistaken.

What clone(CLONE_VM) will do is essentially create a new process, that shares the MM with the original process. Similar to a thread, just that the new process will show up in /proc/ as ... a new process, not as a thread under /prod/$pid/tasks of the original process.

Both processes will operate on the shared MM struct as if they were ordinary threads. No Copy-on-Write involved.

Quoted 7 lines
Quoted 3 lines

One example use case I've been involved in is async teardown in QEMU [1].

[1] https://kvm-forum.qemu.org/2022/ibm_async_destroy.pdf

I understand what you mean, but what I'm really confused about is how this relates to allocating anonymous THP. If either one allocates anon THP, it will definitely create a new MM, right ?

No. They work on the same address space - same MM. Either can allocate a new anon THP and the other one would be able to modify it. No fork/CoW.

I only bring it up because it's two "processes" sharing the same MM. And the THP mode in your proposal would actually be per-MM and not per process.

It's confusing ... :)

Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 9 lines

Additionally, this

Quoted 3 lines

can be implemented per-MM without kernel code modifications. With a well-designed API, users can even implement custom THP policies—all without altering kernel code.

Quoted 6 lines

You can switch between modes, that' all you can do. I wouldn't really call that "custom policy" as it is extremely limited.

And that's exactly my point: it's basic switching between modes ... a reasonable policy in the future will make placement decisions and not just state "always/never/madvise".

Could you please elaborate further on 'make placement decisions'? As previously mentioned, we (including the broader community) really need the user input to determine whether THP allocation is appropriate in a given case.

Quoted 8 lines
Quoted 5 lines

The glorious future were we make smarter decisions where to actually place THPs even in the "always" mode.

E.g., just because we enable "always" for a process does not mean that we really want a THP everywhere; quite the opposite.

So 'always' simply means "the system doesn't guarantee THP allocation will succeed" ?

I mean, with THPs, there are no guarantees, ever :(

Quoted 3 lines

If that's the case, we should revisit RFC v1 [0],
where we proposed rejecting THP allocations in certain scenarios for specific tasks.

Hooking into actual page allocation during page faults (e.g., THP size, khugepaged collapse decisions) is IMHO a much better application of ebpf than setting a THP mode per process (or MM ... ) using epbf.

So yes, you could drive the system in "always" mode and decide to not allocate THPs during page faults / khugepaged for specific processes.

IMHO that also does not contradict the VM_HUGEPAGE / VM_NOHUGEPAGE default setting proposal: VM_HUGEPAGE could feed into the epbf program as yet another parameter to make a decision.

Signature · 3 lines
Cheers,

David / dhildenb
#49 lore raw
Quoted 110 lines · David Hildenbrand
Quoted 100 lines · Yafang Shao
Quoted 99 lines · David Hildenbrand
Quoted 80 lines · Yafang Shao
Quoted 79 lines · David Hildenbrand
Quoted 4 lines
Quoted 4 lines

I don't think we want to add such a mechanism (new mode) where the primary configuration mechanism is through bpf.

Quoted 8 lines
Quoted 5 lines

Maybe bpf could be used as an alternative, but we should look into a reasonable alternative first, like the discussed mctrl()/.../ raised in the process_madvise() series.

No "bpf" mode in disguise, please :)

This goal can be readily achieved using a BPF program. In any case, it is a feasible solution.

No BPF-only solution.

Quoted 46 lines
Quoted 46 lines
Quoted 28 lines

We could define
the API as follows:

struct bpf_thp_ops {
/**
* @task_thp_mode: Get the THP mode for a specific task *
* Return:
* - TASK_THP_ALWAYS: "always" mode
* - TASK_THP_MADVISE: "madvise" mode
* - TASK_THP_NEVER: "never" mode
* Future modes can also be added.
*/
int (*task_thp_mode)(struct task_struct *p);
};

For observability, we could add a "THP mode" field to /proc/[pid]/status. For example:

$ grep "THP mode" /proc/123/status
always
$ grep "THP mode" /proc/456/status
madvise
$ grep "THP mode" /proc/789/status
never
Quoted 5 lines

The THP mode for each task would be determined by the attached BPF program based on the task's attributes. We would place the BPF hook in appropriate kernel functions. Note that this setting wouldn't be inherited during fork/exec - the BPF program would make the decision dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

Quoted 5 lines

This approach also enables runtime adjustments to THP modes based on system-wide conditions, such as memory fragmentation or other performance overheads. The BPF program could adapt policies dynamically, optimizing THP behavior in response to changing workloads.

I am not sure that is the proper way to handle these scenarios: I never heard that people would be adjusting the system-wide policy dynamically in that way either.

Quoted 7 lines
Quoted 3 lines

Whatever we do, we have to make sure that what we add won't over-complicate things in the future. Having tooling dynamically adjust the THP policy of processes that coarsely sounds ... very wrong long-term.

This is just an example demonstrating how BPF can be used to adjust its flexibility. Notably, all these policies can be implemented without modifying the kernel.

See below on "policy".

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

As Liam pointed out in another thread, naming is challenging here -

"process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be used by multiple processes ...

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I consistently use 'thread' for the latter case.

You can use CLONE_VM without CLONE_THREAD ...

If I understand correctly, this can only occur for shared THP but not anonymous THP. For instance, if either process allocates an anonymous THP, it would trigger the creation of a new MM. Please correct me if I'm mistaken.

What clone(CLONE_VM) will do is essentially create a new process, that shares the MM with the original process. Similar to a thread, just that the new process will show up in /proc/ as ... a new process, not as a thread under /prod/$pid/tasks of the original process.

Both processes will operate on the shared MM struct as if they were ordinary threads. No Copy-on-Write involved.

Quoted 7 lines
Quoted 3 lines

One example use case I've been involved in is async teardown in QEMU [1].

[1] https://kvm-forum.qemu.org/2022/ibm_async_destroy.pdf

I understand what you mean, but what I'm really confused about is how this relates to allocating anonymous THP. If either one allocates anon THP, it will definitely create a new MM, right ?

Quoted 7 lines

No. They work on the same address space - same MM. Either can allocate a new anon THP and the other one would be able to modify it. No fork/CoW.

I only bring it up because it's two "processes" sharing the same MM. And the THP mode in your proposal would actually be per-MM and not per process.

It's confusing ... :)

Thanks for the explanation.

Quoted 41 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 10 lines

Additionally, this

Quoted 3 lines

can be implemented per-MM without kernel code modifications. With a well-designed API, users can even implement custom THP policies—all without altering kernel code.

Quoted 6 lines

You can switch between modes, that' all you can do. I wouldn't really call that "custom policy" as it is extremely limited.

And that's exactly my point: it's basic switching between modes ... a reasonable policy in the future will make placement decisions and not just state "always/never/madvise".

Could you please elaborate further on 'make placement decisions'? As previously mentioned, we (including the broader community) really need the user input to determine whether THP allocation is appropriate in a given case.

Quoted 8 lines
Quoted 5 lines

The glorious future were we make smarter decisions where to actually place THPs even in the "always" mode.

E.g., just because we enable "always" for a process does not mean that we really want a THP everywhere; quite the opposite.

So 'always' simply means "the system doesn't guarantee THP allocation will succeed" ?

I mean, with THPs, there are no guarantees, ever :(

Quoted 3 lines

If that's the case, we should revisit RFC v1 [0],
where we proposed rejecting THP allocations in certain scenarios for specific tasks.

Hooking into actual page allocation during page faults (e.g., THP size, khugepaged collapse decisions) is IMHO a much better application of ebpf than setting a THP mode per process (or MM ... ) using epbf.

Quoted 6 lines

So yes, you could drive the system in "always" mode and decide to not allocate THPs during page faults / khugepaged for specific processes.

IMHO that also does not contradict the VM_HUGEPAGE / VM_NOHUGEPAGE default setting proposal: VM_HUGEPAGE could feed into the epbf program as yet another parameter to make a decision.

That seems like a viable solution. Thank you for your help.

Signature · 2 lines
Regards
Yafang
#50 lore raw
Quoted 12 lines
Quoted 12 lines

I mean, with THPs, there are no guarantees, ever :(

Quoted 3 lines

If that's the case, we should revisit RFC v1 [0],
where we proposed rejecting THP allocations in certain scenarios for specific tasks.

Hooking into actual page allocation during page faults (e.g., THP size, khugepaged collapse decisions) is IMHO a much better application of ebpf than setting a THP mode per process (or MM ... ) using epbf.

Quoted 8 lines
Quoted 6 lines

So yes, you could drive the system in "always" mode and decide to not allocate THPs during page faults / khugepaged for specific processes.

IMHO that also does not contradict the VM_HUGEPAGE / VM_NOHUGEPAGE default setting proposal: VM_HUGEPAGE could feed into the epbf program as yet another parameter to make a decision.

That seems like a viable solution. Thank you for your help.

Good! And the required ebpf hooks would probably come in handy to write more advanced policies / allocation logic than just "give it THPs" vs. "don't give it THPs".

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#51 lore raw
In reply to Yafang Shao
Quoted 117 lines · Yafang Shao
Quoted 102 lines · Yafang Shao

Background
----------

At my current employer, PDD, we have consistently configured THP to "never" on our production servers due to past incidents caused by its behavior:

- Increased memory consumption
THP significantly raises overall memory usage.

- Latency spikes
Random latency spikes occur due to more frequent memory compaction activity triggered by THP.

These issues have made sysadmins hesitant to switch to "madvise" or "always" modes.

New Motivation
--------------

We have now identified that certain AI workloads achieve substantial performance gains with THP enabled. However, we’ve also verified that some workloads see little to no benefit—or are even negatively impacted—by THP.

In our Kubernetes environment, we deploy mixed workloads on a single server to maximize resource utilization. Our goal is to selectively enable THP for services that benefit from it while keeping it disabled for others. This approach allows us to incrementally enable THP for additional services and assess how to make it more viable in production.

Proposed Solution
-----------------

For this use case, Johannes suggested introducing a dedicated mode [0]. In this new mode, we could implement BPF-based THP adjustment for fine-grained control over tasks or cgroups. If no BPF program is attached, THP remains in "never" mode. This solution elegantly meets our needs while avoiding the complexity of managing BPF alongside other THP modes.

A selftest example demonstrates how to enable THP for the current task while keeping it disabled for others.

Alternative Proposals
---------------------

- Gutierrez’s cgroup-based approach [1]
- Proposed adding a new cgroup file to control THP policy.
- However, as Johannes noted, cgroups are designed for hierarchical resource allocation, not arbitrary policy settings [2].

- Usama’s per-task THP proposal based on prctl() [3]:
- Enabling THP per task via prctl().
- As David pointed out, neither madvise() nor prctl() works in "never" mode [4], making this solution insufficient for our needs.

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

This is currently a PoC implementation. Feedback of any kind is welcome.

RFC v1->v2:
The main changes are as follows,
- Use struct_ops instead of fmod_ret (Alexei)
- Introduce a new THP mode (Johannes)
- Introduce new helpers for BPF hook (Zi)
- Refine the commit log

RFC v1: https://lwn.net/Articles/1019290/

Yafang Shao (5):
mm: thp: Add a new mode "bpf"
mm: thp: Add hook for BPF based THP adjustment
mm: thp: add struct ops for BPF based THP adjustment
bpf: Add get_current_comm to bpf_base_func_proto selftests/bpf: Add selftest for THP adjustment

 include/linux/huge_mm.h                       |  15 +-
 kernel/bpf/cgroup.c                           |   2 -
 kernel/bpf/helpers.c                          |   2 +
 mm/Makefile                                   |   3 +
 mm/bpf_thp.c                                  | 120 ++++++++++++
 mm/huge_memory.c                              |  65 ++++++-
 mm/khugepaged.c                               |   3 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 175 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  39 ++++
 10 files changed, 414 insertions(+), 11 deletions(-)
 create mode 100644 mm/bpf_thp.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
Quoted 3 lines

--
2.43.5

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 10 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;) Both Lorenzo and David propose relying on the madvise mode. However, since madvise is an unprivileged userspace mechanism, any user can freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.
Given these limitations, the community should consider introducing a new "admin" mode for privileged THP policy management.

I agree with the above two points.

Quoted 9 lines

- Can the Kernel Automatically Manage THP Without User Input? In practice, users define their own success metrics—such as latency (RT), queries per second (QPS), or throughput—to evaluate a feature’s usefulness. If a feature fails to improve these metrics, it provides no practical value.
Currently, the kernel lacks visibility into user-defined metrics, making fully automated optimization impossible (at least without user input). More importantly, automatic management offers no benefit if it doesn’t align with user needs.

Yes, kernel is basically guessing what userspace wants with some hints like MADV_HUGEPAGE/MADV_NOHUGEPAGE. But kernel has the global view of memory fragmentation, which userspace cannot get easily. I wonder if it is possible that userspace tuning might benefit one set of applications but hurt others or overall performance. Right now, THP tuning is 0 or 1, either an application wants THPs or not. We might need a way of ranking THP requests from userspace to let kernel prioritize them (I am not sure if we can add another user input parameter, like THP_nice, to get this done, since apparently everyone will set THP_nice to -100 to get themselves at the top of the list).

Quoted 9 lines

Exception: For kernel-enforced changes (e.g., the page-to-folio transition), users must adapt regardless. But THP tuning requires flexibility—forcing automation without measurable gains is counterproductive.
(Please correct me if I’ve overlooked anything.)

Signature · 2 lines
Regards
Yafang

--
Best Regards,
Yan, Zi

#52 lore raw
Quoted 143 lines · Zi Yan
Quoted 117 lines · Yafang Shao
Quoted 102 lines · Yafang Shao

Background
----------

At my current employer, PDD, we have consistently configured THP to "never" on our production servers due to past incidents caused by its behavior:

- Increased memory consumption
THP significantly raises overall memory usage.

- Latency spikes
Random latency spikes occur due to more frequent memory compaction activity triggered by THP.

These issues have made sysadmins hesitant to switch to "madvise" or "always" modes.

New Motivation
--------------

We have now identified that certain AI workloads achieve substantial performance gains with THP enabled. However, we’ve also verified that some workloads see little to no benefit—or are even negatively impacted—by THP.

In our Kubernetes environment, we deploy mixed workloads on a single server to maximize resource utilization. Our goal is to selectively enable THP for services that benefit from it while keeping it disabled for others. This approach allows us to incrementally enable THP for additional services and assess how to make it more viable in production.

Proposed Solution
-----------------

For this use case, Johannes suggested introducing a dedicated mode [0]. In this new mode, we could implement BPF-based THP adjustment for fine-grained control over tasks or cgroups. If no BPF program is attached, THP remains in "never" mode. This solution elegantly meets our needs while avoiding the complexity of managing BPF alongside other THP modes.

A selftest example demonstrates how to enable THP for the current task while keeping it disabled for others.

Alternative Proposals
---------------------

- Gutierrez’s cgroup-based approach [1]
- Proposed adding a new cgroup file to control THP policy.
- However, as Johannes noted, cgroups are designed for hierarchical resource allocation, not arbitrary policy settings [2].

- Usama’s per-task THP proposal based on prctl() [3]:
- Enabling THP per task via prctl().
- As David pointed out, neither madvise() nor prctl() works in "never" mode [4], making this solution insufficient for our needs.

Conclusion
----------

Introducing a new "bpf" mode for BPF-based per-task THP adjustments is the most effective solution for our requirements. This approach represents a small but meaningful step toward making THP truly usable—and manageable—in production environments.

This is currently a PoC implementation. Feedback of any kind is welcome.

RFC v1->v2:
The main changes are as follows,
- Use struct_ops instead of fmod_ret (Alexei)
- Introduce a new THP mode (Johannes)
- Introduce new helpers for BPF hook (Zi)
- Refine the commit log

RFC v1: https://lwn.net/Articles/1019290/

Yafang Shao (5):
mm: thp: Add a new mode "bpf"
mm: thp: Add hook for BPF based THP adjustment
mm: thp: add struct ops for BPF based THP adjustment
bpf: Add get_current_comm to bpf_base_func_proto selftests/bpf: Add selftest for THP adjustment

 include/linux/huge_mm.h                       |  15 +-
 kernel/bpf/cgroup.c                           |   2 -
 kernel/bpf/helpers.c                          |   2 +
 mm/Makefile                                   |   3 +
 mm/bpf_thp.c                                  | 120 ++++++++++++
 mm/huge_memory.c                              |  65 ++++++-
 mm/khugepaged.c                               |   3 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/thp_adjust.c     | 175 ++++++++++++++++++
 .../selftests/bpf/progs/test_thp_adjust.c     |  39 ++++
 10 files changed, 414 insertions(+), 11 deletions(-)
 create mode 100644 mm/bpf_thp.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
Quoted 3 lines

--
2.43.5

Hi all,

Let’s summarize the current state of the discussion and identify how to move forward.

- Global-Only Control is Not Viable
We all seem to agree that a global-only control for THP is unwise. In practice, some workloads benefit from THP while others do not, so a one-size-fits-all approach doesn’t work.

Quoted 10 lines

- Should We Use "Always" or "Madvise"?
I suspect no one would choose 'always' in its current state. ;) Both Lorenzo and David propose relying on the madvise mode. However, since madvise is an unprivileged userspace mechanism, any user can freely adjust their THP policy. This makes fine-grained control impossible without breaking userspace compatibility—an undesirable tradeoff.
Given these limitations, the community should consider introducing a new "admin" mode for privileged THP policy management.

I agree with the above two points.

Quoted 9 lines

- Can the Kernel Automatically Manage THP Without User Input? In practice, users define their own success metrics—such as latency (RT), queries per second (QPS), or throughput—to evaluate a feature’s usefulness. If a feature fails to improve these metrics, it provides no practical value.
Currently, the kernel lacks visibility into user-defined metrics, making fully automated optimization impossible (at least without user input). More importantly, automatic management offers no benefit if it doesn’t align with user needs.

Quoted 3 lines

Yes, kernel is basically guessing what userspace wants with some hints like MADV_HUGEPAGE/MADV_NOHUGEPAGE. But kernel has the global view of memory fragmentation, which userspace cannot get easily.

Correct, memory fragmentation is another critical factor in determining whether to allocate THP.

Quoted 9 lines

I wonder
if it is possible that userspace tuning might benefit one set of applications but hurt others or overall performance. Right now, THP tuning is 0 or 1, either an application wants THPs or not. We might need a way of ranking THP requests from userspace to let kernel prioritize them (I am not sure if we can add another user input parameter, like THP_nice, to get this done, since apparently everyone will set THP_nice to -100 to get themselves at the top of the list).

Interesting idea. Perhaps we could make this configurable only by sysadmins.

Signature · 2 lines
Regards
Yafang