mm, bpf: BPF-MM, BPF-THP
Subsystems (15)
v12 cover mm, bpf: BPF-MM, BPF-THP
History
=======
RFC v1: fmod_ret based BPF-THP hook
https://lore.kernel.org/linux-mm/[email protected]/
RFC v2: struct_ops based BPF-THP hook
https://lore.kernel.org/linux-mm/[email protected]/
RFC v4: Get THP order with interface get_suggested_order()
https://lore.kernel.org/linux-mm/[email protected]/
v4->v9: Simplify the interface to:
unsigned long
bpf_hook_thp_get_orders(struct vm_area_struct *vma, enum tva_type type,
unsigned long orders);
https://lore.kernel.org/linux-mm/[email protected]/v9->RFC v10: Scope BPF-THP to individual processesv10->v11: Remove the RFC tagv11->v12: Fix issues reported by AI
The Design
==========
Scoping BPF-THP to cgroup is rejected
-------------------------------------
As explained by Gutierrez:
1. It breaks the cgroup hierarchy when 2 siblings have different THP policies
2. Cgroup was designed for resource management not for grouping processes and tune those processes
3. We set a precedent for other people adding new flags to cgroup and potentially polluting cgroups. We may end up with cgroups having tens of different flags, making sysadmin's job more complex
The related links are:
https://lore.kernel.org/linux-mm/[email protected]/
https://lore.kernel.org/linux-mm/[email protected]/
So we has to scope it to process.
Scoping BPF-THP to process
--------------------------
To eliminate potential conflicts among competing BPF-THP instances, we enforce that each process is exclusively managed by a single BPF-THP. This approach has received agreement from David. For context, see:
https://lore.kernel.org/linux-mm/[email protected]/
When registering a BPF-THP, we specify the PID of a target task. The BPF-THP is then installed in the task's mm_struct
struct mm_struct {
struct bpf_thp_ops __rcu *thp_thp;
};Inheritance Behavior:
- Existing child processes are unaffected
- Newly forked children inherit the BPF-THP from their parent
- The BPF-THP persists across exec
A new linked list tracks all tasks managed by each BPF-THP instance:
- Newly managed tasks are added to the list
- Exiting tasks are automatically removed from the list
- During BPF-THP unregistration (e.g., when the BPF link is removed), all managed tasks have their bpf_thp pointer set to NULL
- BPF-THP instances can be dynamically updated, with all tracked tasks automatically migrating to the new version.
This design simplifies BPF-THP management in production environments by providing clear lifecycle management and preventing conflicts between multiple BPF-THP instances.
Global Mode
-----------
The per-process BPF-THP mode is unsuitable for managing shared resources such as shmem THP and file-backed THP. This aligns with known cgroup limitations for similar scenarios:
https://lore.kernel.org/linux-mm/[email protected]/
Introduce a global BPF-THP mode to address this gap. When registered:
- All existing per-process instances are disabled
- New per-process registrations are blocked
- Existing per-process instances remain registered (no forced unregistration)
The global mode takes precedence over per-process instances. Updates are
type-isolated: global instances can only be updated by new global instances, and per-process instances by new per-process instances.
BPF CI
------
Several dependency patches are currently in mm-new but haven't been merged into bpf-next. To enable BPF CI testing, I had to make minor changes to patches #1 and #2 and trigger the BPF CI manually. For details, see:
https://github.com/kernel-patches/bpf/pull/10097
An error occurred during the test, but it was unrelated to this series.
Yafang Shao (10):
mm: thp: remove vm_flags parameter from khugepaged_enter_vma()
mm: thp: remove vm_flags parameter from thp_vma_allowable_order()
mm: thp: add support for BPF based THP order selection
mm: thp: decouple THP allocation between swap and page fault paths
mm: thp: enable THP allocation exclusively through khugepaged
mm: bpf-thp: add support for global mode
Documentation: add BPF THP
selftests/bpf: add a simple BPF based THP policy selftests/bpf: add test case to update THP policy selftests/bpf: add test case for BPF-THP inheritance across fork
Documentation/admin-guide/mm/transhuge.rst | 113 +++++ MAINTAINERS | 3 + fs/exec.c | 1 + fs/proc/task_mmu.c | 3 +- include/linux/huge_mm.h | 58 ++- include/linux/khugepaged.h | 10 +- include/linux/mm_types.h | 17 + kernel/fork.c | 1 + mm/Kconfig | 24 + mm/Makefile | 1 + mm/huge_memory.c | 7 +- mm/huge_memory_bpf.c | 423 ++++++++++++++++++ mm/khugepaged.c | 43 +- mm/madvise.c | 7 + mm/memory.c | 22 +- mm/mmap.c | 1 + mm/shmem.c | 2 +- mm/vma.c | 6 +- tools/testing/selftests/bpf/config | 3 + .../selftests/bpf/prog_tests/thp_adjust.c | 357 +++++++++++++++ .../selftests/bpf/progs/test_thp_adjust.c | 53 +++ 21 files changed, 1101 insertions(+), 54 deletions(-) create mode 100644 mm/huge_memory_bpf.c create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
Signature · 1 line
2.47.3
PATCH v12 1/10 mm: thp: remove vm_flags parameter from khugepaged_enter_vma()
The khugepaged_enter_vma() function requires handling in two specific scenarios:
1. New VMA creation
When a new VMA is created (for anon vma, it is deferred to pagefault), if vma->vm_mm is not present in khugepaged_mm_slot, it must be added. In this case, khugepaged_enter_vma() is called after vma->vm_flags have been set, allowing direct use of the VMA's flags.
2. VMA flag modification
When vma->vm_flags are modified (particularly when VM_HUGEPAGE is set), the system must recheck whether to add vma->vm_mm to khugepaged_mm_slot. Currently, khugepaged_enter_vma() is called before the flag update, so the call must be relocated to occur after vma->vm_flags have been set.
In the VMA merging path, khugepaged_enter_vma() is also called. For this case, since VMA merging only occurs when the vm_flags of both VMAs are identical (excluding special flags like VM_SOFTDIRTY), we can safely use target->vm_flags instead. (It is worth noting that khugepaged_enter_vma() can be removed from the VMA merging path because the VMA has already been added in the two aforementioned cases. We will address this cleanup in a separate patch.)
After this change, we can further remove vm_flags parameter from thp_vma_allowable_order(). That will be handled in a followup patch.
--- include/linux/khugepaged.h | 10 ++++++---- mm/huge_memory.c | 2 +- mm/khugepaged.c | 27 ++++++++++++++------------- mm/madvise.c | 7 +++++++ mm/vma.c | 6 +++--- 5 files changed, 31 insertions(+), 21 deletions(-)
include/linux/khugepaged.h +6 -4
@@ -15,8 +15,8 @@ extern void khugepaged_destroy(void);extern int start_stop_khugepaged (void );extern void __khugepaged_enter (struct mm_struct *mm);extern void __khugepaged_exit (struct mm_struct *mm); -extern void khugepaged_enter_vma (struct vm_area_struct *vma, -vm_flags_t vm_flags); +extern void khugepaged_enter_vma (struct vm_area_struct *vma); +extern void khugepaged_enter_mm (struct mm_struct *mm);extern void khugepaged_min_free_kbytes_update (void );extern bool current_is_khugepaged (void ); extern int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
@@ -40,8 +40,10 @@ static inline void khugepaged_fork(struct mm_struct *mm, struct mm_struct *oldmmstatic inline void khugepaged_exit (struct mm_struct *mm) { } -static inline void khugepaged_enter_vma (struct vm_area_struct *vma, -vm_flags_t vm_flags) +static inline void khugepaged_enter_vma (struct vm_area_struct *vma) +{ +} +static inline void khugepaged_enter_mm (struct mm_struct *mm) { } static inline int collapse_pte_mapped_thp(struct mm_struct *mm,
mm/huge_memory.c +1 -1
@@ -1476,7 +1476,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) ret =vmf_anon_prepare (vmf);if (ret)return ret; -khugepaged_enter_vma (vma, vma->vm_flags); +khugepaged_enter_vma (vma); if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm) &&
mm/khugepaged.c +14 -13
@@ -367,12 +367,6 @@ int hugepage_madvise(struct vm_area_struct *vma,#endif *vm_flags &= ~VM_NOHUGEPAGE ; *vm_flags |=VM_HUGEPAGE ; -/* -* If the vma become good for khugepaged to scan, -* register it here without waiting a page fault that -* may not happen any time soon. -*/ -khugepaged_enter_vma (vma, *vm_flags);break ;case MADV_NOHUGEPAGE : *vm_flags &= ~VM_HUGEPAGE ;
@@ -514,14 +508,21 @@ static unsigned long collapse_allowable_orders(struct vm_area_struct *vma,return thp_vma_allowable_orders (vma, vm_flags, tva_flags, orders); } -void khugepaged_enter_vma (struct vm_area_struct *vma, -vm_flags_t vm_flags) +void khugepaged_enter_mm (struct mm_struct *mm) { -if (!mm_flags_test (MMF_VM_HUGEPAGE , vma->vm_mm) && -hugepage_enabled ()) { -if (collapse_allowable_orders (vma, vm_flags, true)) -__khugepaged_enter (vma->vm_mm); - } +if (mm_flags_test (MMF_VM_HUGEPAGE , mm)) +return ; +if (!hugepage_enabled ()) +return ; + +__khugepaged_enter (mm); +} + +void khugepaged_enter_vma (struct vm_area_struct *vma) +{ +if (!collapse_allowable_orders (vma, vma->vm_flags, true)) +return ; +khugepaged_enter_mm (vma->vm_mm); }void __khugepaged_exit (struct mm_struct *mm)
mm/madvise.c +7 -0
@@ -1425,6 +1425,13 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior)VM_WARN_ON_ONCE (madv_behavior->lock_mode !=MADVISE_MMAP_WRITE_LOCK ); error =madvise_update_vma (new_flags, madv_behavior); +/* +* If the vma become good for khugepaged to scan, +* register it here without waiting a page fault that +* may not happen any time soon. +*/ +if (!error && new_flags &VM_HUGEPAGE ) +khugepaged_enter_mm (madv_behavior->vma->vm_mm); out:/* * madvise() returns EAGAIN if kernel resources, such as
mm/vma.c +3 -3
@@ -975,7 +975,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(if (err ||commit_merge (vmg)) goto abort; -khugepaged_enter_vma (vmg->target, vmg->vm_flags); +khugepaged_enter_vma (vmg->target); vmg->state =VMA_MERGE_SUCCESS ;return vmg->target;
@@ -1095,7 +1095,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)* following VMA if we have VMAs on both sides. */ if (vmg->target && !vma_expand (vmg)) { -khugepaged_enter_vma (vmg->target, vmg->vm_flags); +khugepaged_enter_vma (vmg->target); vmg->state =VMA_MERGE_SUCCESS ;return vmg->target; }
@@ -2506,7 +2506,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap)* call covers the non-merge case. */ if (!vma_is_anonymous (vma)) -khugepaged_enter_vma (vma, map->vm_flags); +khugepaged_enter_vma (vma); *vmap = vma;return 0 ;
Signature · 1 line
2.47.3
PATCH v12 2/10 mm: thp: remove vm_flags parameter from thp_vma_allowable_order()
In reply to Yafang ShaoBecause all calls to thp_vma_allowable_order() pass vma->vm_flags as the vma_flags argument, we can remove the parameter and have the function access vma->vm_flags directly.
--- fs/proc/task_mmu.c | 3 +-- include/linux/huge_mm.h | 16 ++++++++-------- mm/huge_memory.c | 4 ++-- mm/khugepaged.c | 18 +++++++++--------- mm/memory.c | 11 +++++------ mm/shmem.c | 2 +- 6 files changed, 26 insertions(+), 28 deletions(-)
fs/proc/task_mmu.c +1 -2
@@ -1369,8 +1369,7 @@ static int show_smap(struct seq_file *m, void *v)__show_smap (m, &mss, false);seq_printf (m,"THPeligible: %8u\n" , - !!thp_vma_allowable_orders (vma, vma->vm_flags,TVA_SMAPS , -THP_ORDERS_ALL )); + !!thp_vma_allowable_orders (vma,TVA_SMAPS ,THP_ORDERS_ALL ));if (arch_pkeys_enabled ())seq_printf (m,"ProtectionKey: %8u\n" ,vma_pkey (vma));
include/linux/huge_mm.h +8 -8
@@ -101,8 +101,8 @@ enum tva_type { TVA_FORCED_COLLAPSE, /* Forced collapse (e.g. MADV_COLLAPSE). */ }; -#define thp_vma_allowable_order (vma, vm_flags, type, order) \ - (!!thp_vma_allowable_orders(vma, vm_flags, type, BIT(order))) +#define thp_vma_allowable_order (vma, type, order) \ + (!!thp_vma_allowable_orders(vma, type, BIT(order)))#define split_folio (f) split_folio_to_list(f, NULL)
@@ -271,14 +271,12 @@ static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma, }unsigned long __thp_vma_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type type,unsigned long orders);/** * thp_vma_allowable_orders - determine hugepage orders that are allowed for vma * @vma: the vm area to check -* @vm_flags: use these vm_flags instead of vma->vm_flags * @type: TVA type * @orders: bitfield of all orders to consider *
@@ -292,10 +290,11 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,*/ static inline unsigned long thp_vma_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type type,unsigned long orders) { +vm_flags_t vm_flags = vma->vm_flags; +/* * Optimization to check if required orders are enabled early. Only * forced collapse ignores sysfs configs.
@@ -314,7 +313,7 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,return 0 ; } -return __thp_vma_allowable_orders (vma, vm_flags, type, orders); +return __thp_vma_allowable_orders (vma, type, orders); }struct thpsize {
@@ -334,8 +333,10 @@ struct thpsize {* through madvise or prctl. */ static inline bool vma_thp_disabled (struct vm_area_struct *vma, -vm_flags_t vm_flags,bool forced_collapse) +bool forced_collapse) { +vm_flags_t vm_flags = vma->vm_flags; +/* Are THPs disabled for this VMA? */ if (vm_flags &VM_NOHUGEPAGE )return true;
@@ -564,7 +565,6 @@ static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma, }static inline unsigned long thp_vma_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type type,unsigned long orders) {
mm/huge_memory.c +2 -2
@@ -98,7 +98,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) }unsigned long __thp_vma_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type type,unsigned long orders) {
@@ -106,6 +105,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,const bool in_pf = type ==TVA_PAGEFAULT ;const bool forced_collapse = type ==TVA_FORCED_COLLAPSE ;unsigned long supported_orders; +vm_flags_t vm_flags = vma->vm_flags;/* Check the intersection of requested and supported orders. */ if (vma_is_anonymous (vma))
@@ -122,7 +122,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,if (!vma->vm_mm)/* vdso */ return 0 ; -if (thp_disabled_by_hw () ||vma_thp_disabled (vma, vm_flags, forced_collapse)) +if (thp_disabled_by_hw () ||vma_thp_disabled (vma, forced_collapse))return 0 ;/* khugepaged doesn't collapse DAX vma, but page fault is fine. */
mm/khugepaged.c +9 -9
@@ -499,13 +499,13 @@ static unsigned int collapse_max_ptes_none(unsigned int order, bool full_scan)/* Check what orders are allowed based on the vma and collapse type */ static unsigned long collapse_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,bool is_khugepaged) +bool is_khugepaged) { -enum tva_type tva_flags = is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ; +enum tva_type tva_type = is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ;unsigned long orders = is_khugepaged &&vma_is_anonymous (vma) ?THP_ORDERS_ALL_ANON :BIT (HPAGE_PMD_ORDER ); -return thp_vma_allowable_orders (vma, vm_flags, tva_flags, orders); +return thp_vma_allowable_orders (vma, tva_type, orders); }void khugepaged_enter_mm (struct mm_struct *mm)
@@ -520,7 +520,7 @@ void khugepaged_enter_mm(struct mm_struct *mm)void khugepaged_enter_vma (struct vm_area_struct *vma) { -if (!collapse_allowable_orders (vma, vma->vm_flags, true)) +if (!collapse_allowable_orders (vma,TVA_KHUGEPAGED ))return ;khugepaged_enter_mm (vma->vm_mm); }
@@ -992,7 +992,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,/* Always check the PMD order to ensure its not shared by another VMA */ if (!thp_vma_suitable_order (vma, address,PMD_ORDER ))return SCAN_ADDRESS_RANGE ; -if (!thp_vma_allowable_orders (vma, vma->vm_flags, type,BIT (order))) +if (!thp_vma_allowable_orders (vma, type,BIT (order)))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then
@@ -1508,7 +1508,7 @@ static int collapse_scan_pmd(struct mm_struct *mm,memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); - enabled_orders =collapse_allowable_orders (vma, vma->vm_flags, cc->is_khugepaged); + enabled_orders =collapse_allowable_orders (vma, cc->is_khugepaged);/* * If PMD is the only enabled order, enforce max_ptes_none, otherwise
@@ -1777,7 +1777,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,* and map it by a PMD, regardless of sysfs THP settings. As such, let's * analogously elide sysfs THP settings here and force collapse. */ -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_FORCED_COLLAPSE ,PMD_ORDER )) +if (!thp_vma_allowable_order (vma,TVA_FORCED_COLLAPSE ,PMD_ORDER ))return SCAN_VMA_CHECK ;/* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */
@@ -2719,7 +2719,7 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, progress++;break ; } -if (!collapse_allowable_orders (vma, vma->vm_flags, true)) { +if (!collapse_allowable_orders (vma, true)) { skip: progress++;continue ;
@@ -3025,7 +3025,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,BUG_ON (vma->vm_start > start);BUG_ON (vma->vm_end < end); -if (!collapse_allowable_orders (vma, vma->vm_flags, false)) +if (!collapse_allowable_orders (vma, false))return -EINVAL ; cc =kmalloc (sizeof (*cc),GFP_KERNEL );
mm/memory.c +5 -6
@@ -4558,7 +4558,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)* Get a list of all the (large) orders below PMD_ORDER that are enabled * and suitable for swapping THP. */ - orders =thp_vma_allowable_orders (vma, vma->vm_flags,TVA_PAGEFAULT , + orders =thp_vma_allowable_orders (vma,TVA_PAGEFAULT ,BIT (PMD_ORDER ) -1 ); orders =thp_vma_suitable_orders (vma, vmf->address, orders); orders = thp_swap_suitable_orders(swp_offset(entry),
@@ -5107,7 +5107,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)* for this vma. Then filter out the orders that can't be allocated over * the faulting address and still be fully contained in the vma. */ - orders =thp_vma_allowable_orders (vma, vma->vm_flags,TVA_PAGEFAULT , + orders =thp_vma_allowable_orders (vma,TVA_PAGEFAULT ,BIT (PMD_ORDER ) -1 ); orders =thp_vma_suitable_orders (vma, vmf->address, orders);
@@ -5379,7 +5379,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *pa* PMD mappings if THPs are disabled. As we already have a THP, * behave as if we are forcing a collapse. */ -if (thp_disabled_by_hw () ||vma_thp_disabled (vma, vma->vm_flags, +if (thp_disabled_by_hw () ||vma_thp_disabled (vma,/* forced_collapse=*/ true))return ret;
@@ -6289,7 +6289,6 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, .gfp_mask = __get_fault_gfp_mask(vma), };struct mm_struct *mm = vma->vm_mm; -vm_flags_t vm_flags = vma->vm_flags;pgd_t *pgd;p4d_t *p4d;vm_fault_t ret;
@@ -6304,7 +6303,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,return VM_FAULT_OOM ; retry_pud:if (pud_none (*vmf.pud) && -thp_vma_allowable_order (vma, vm_flags,TVA_PAGEFAULT ,PUD_ORDER )) { +thp_vma_allowable_order (vma,TVA_PAGEFAULT ,PUD_ORDER )) { ret =create_huge_pud (&vmf);if (!(ret &VM_FAULT_FALLBACK ))return ret;
@@ -6338,7 +6337,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, goto retry_pud;if (pmd_none (*vmf.pmd) && -thp_vma_allowable_order (vma, vm_flags,TVA_PAGEFAULT ,PMD_ORDER )) { +thp_vma_allowable_order (vma,TVA_PAGEFAULT ,PMD_ORDER )) { ret =create_huge_pmd (&vmf);if (!(ret &VM_FAULT_FALLBACK ))return ret;
mm/shmem.c +1 -1
@@ -1809,7 +1809,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,vm_flags_t vm_flags = vma ? vma->vm_flags :0 ;unsigned int global_orders; -if (thp_disabled_by_hw () || (vma &&vma_thp_disabled (vma, vm_flags, shmem_huge_force))) +if (thp_disabled_by_hw () || (vma &&vma_thp_disabled (vma, shmem_huge_force)))return 0 ; global_orders = shmem_huge_global_enabled(inode, index, write_end,
Signature · 1 line
2.47.3
PATCH v12 3/10 mm: thp: add support for BPF based THP order selection
In reply to Yafang ShaoThe Motivation
==============
This patch introduces a new BPF struct_ops called bpf_thp_ops for dynamic THP tuning. It includes a hook bpf_hook_thp_get_order(), allowing BPF programs to influence THP order selection based on factors such as:
- Workload identity
For example, workloads running in specific containers or cgroups.
- Allocation context
Whether the allocation occurs during a page fault, khugepaged, swap or other paths.
- VMA's memory advice settings
MADV_HUGEPAGE or MADV_NOHUGEPAGE
- Memory pressure
PSI system data or associated cgroup PSI metrics
The BPF-THP Interface
=====================
The kernel API of this new BPF hook is as follows,
/**
* thp_get_order: Get the suggested THP order from a BPF program for allocation
* @vma: vm_area_struct associated with the THP allocation
* @type: TVA type for current @vma
* @orders: Bitmask of available THP orders for this allocation
*
* Return: The suggested THP order for allocation from the BPF program.
* Returns a negative value to preserve the original available @orders, * which is useful in specific cases—for example, when only a particular * @type is handled and others are ignored. */ int thp_get_order(struct vm_area_struct *vma, enum tva_type type, unsigned long orders);
This functionality is only active when system-wide THP is configured to madvise or always mode. It remains disabled in never mode. Additionally, if THP is explicitly disabled for a specific task via prctl(), this BPF functionality will also be unavailable for that task.
The Design of Per Process BPF-THP
=================================
As suggested by Alexei, we need to scoping the BPF-THP [0].
Scoping BPF-THP to cgroup is not acceptible
-------------------------------------------
As explained by Gutierrez: [1]
1. It breaks the cgroup hierarchy when 2 siblings have different THP policies
2. Cgroup was designed for resource management not for grouping processes and une those processes
3. We set a precedent for other people adding new flags to cgroup and potentially polluting cgroups. We may end up with cgroups having tens of different flags, making sysadmin's job more complex
Scoping BPF-THP to process
--------------------------
To eliminate potential conflicts among competing BPF-THP instances, we enforce that each process is exclusively managed by a single BPF-THP. This approach has received agreement from David [2].
When registering a BPF-THP, we specify the PID of a target task. The BPF-THP is then installed in the task's mm_struct
struct mm_struct {
struct bpf_thp_ops __rcu *thp_thp;
};Inheritance Behavior:
- Existing child processes are unaffected
- Newly forked children inherit the BPF-THP from their parent
- The BPF-THP persists across execve() calls
A new linked list tracks all tasks managed by each BPF-THP instance:
- Newly managed tasks are added to the list
- Exiting tasks are automatically removed from the list
- During BPF-THP unregistration (e.g., when the BPF link is removed), all managed tasks have their bpf_thp pointer set to NULL
- BPF-THP instances can be dynamically updated, with all tracked tasks automatically migrating to the new version.
This design simplifies BPF-THP management in production environments by providing clear lifecycle management and preventing conflicts between multiple BPF-THP instances.
WARNING
=======
This feature requires CONFIG_BPF_THP (EXPERIMENTAL) to be enabled. Note that this capability is currently unstable and may undergo significant changes—including potential removal—in future kernel versions.
--- MAINTAINERS | 1 + fs/exec.c | 1 + include/linux/huge_mm.h | 39 +++++ include/linux/mm_types.h | 17 +++ kernel/fork.c | 1 + mm/Kconfig | 22 +++ mm/Makefile | 1 + mm/huge_memory_bpf.c | 316 +++++++++++++++++++++++++++++++++++++++ mm/mmap.c | 1 + 9 files changed, 399 insertions(+) create mode 100644 mm/huge_memory_bpf.c
MAINTAINERS +1 -0
@@ -16521,6 +16521,7 @@ F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h F: mm/huge_memory.c +F: mm/huge_memory_bpf.c F: mm/khugepaged.c F: mm/mm_slot.h F: tools/testing/selftests/mm/khugepaged.c
fs/exec.c +1 -0
@@ -890,6 +890,7 @@ static int exec_mmap(struct mm_struct *mm)activate_mm (active_mm, mm);if (IS_ENABLED (CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM ))local_irq_enable (); +bpf_thp_retain_mm (mm, old_mm);lru_gen_add_mm (mm);task_unlock (tsk);lru_gen_use_mm (mm);
include/linux/huge_mm.h +39 -0
@@ -274,6 +274,40 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,enum tva_type type, unsigned long orders); +#ifdef CONFIG_BPF_THP + +unsigned long +bpf_hook_thp_get_orders (struct vm_area_struct *vma,enum tva_type type, +unsigned long orders); +void bpf_thp_exit_mm (struct mm_struct *mm); +void bpf_thp_retain_mm (struct mm_struct *mm,struct mm_struct *old_mm); +void bpf_thp_fork (struct mm_struct *mm,struct mm_struct *old_mm); + +#else + +static inline unsigned long +bpf_hook_thp_get_orders (struct vm_area_struct *vma,enum tva_type type, +unsigned long orders) +{ +return orders; +} + +static inline void bpf_thp_exit_mm (struct mm_struct *mm) +{ +} + +static inline void +bpf_thp_retain_mm (struct mm_struct *mm,struct mm_struct *old_mm) +{ +} + +static inline void +bpf_thp_fork (struct mm_struct *mm,struct mm_struct *old_mm) +{ +} + +#endif +/** * thp_vma_allowable_orders - determine hugepage orders that are allowed for vma * @vma: the vm area to check
@@ -295,6 +329,11 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, {vm_flags_t vm_flags = vma->vm_flags; +/* The BPF-specified order overrides which order is selected. */ + orders &=bpf_hook_thp_get_orders (vma, type, orders); +if (!orders) +return 0 ; +/* * Optimization to check if required orders are enabled early. Only * forced collapse ignores sysfs configs.
include/linux/mm_types.h +17 -0
@@ -983,6 +983,19 @@ struct mm_cid { };#endif +#ifdef CONFIG_BPF_THP +struct bpf_thp_ops ; +#endif + +#ifdef CONFIG_BPF_MM +struct bpf_mm_ops { +#ifdef CONFIG_BPF_THP +struct bpf_thp_ops __rcu *bpf_thp; +struct list_head bpf_thp_list; +#endif +}; +#endif +/* * Opaque type representing current mm_struct flag state. Must be accessed via * mm_flags_xxx() helper functions.
@@ -1280,6 +1293,10 @@ struct mm_struct {#ifdef CONFIG_MM_ID mm_id_t mm_id;#endif /* CONFIG_MM_ID */ + +#ifdef CONFIG_BPF_MM +struct bpf_mm_ops bpf_mm; +#endif } __randomize_layout;/*
kernel/fork.c +1 -0
@@ -1130,6 +1130,7 @@ static inline void __mmput(struct mm_struct *mm)exit_aio (mm);ksm_exit (mm);khugepaged_exit (mm);/* must run before exit_mmap */ +bpf_thp_exit_mm (mm);exit_mmap (mm);mm_put_huge_zero_folio (mm);set_mm_exe_file (mm,NULL );
mm/Kconfig +22 -0
@@ -1457,6 +1457,28 @@ config PT_RECLAIM config FIND_NORMAL_PAGE def_bool n +menuconfig BPF_MM + bool "BPF-based Memory Management (EXPERIMENTAL)" + depends on BPF_SYSCALL + + help + Enable BPF-based Memory Management Policy. This feature is currently + experimental. + + WARNING: This feature is unstable and may change in future kernel + +if BPF_MM +config BPF_THP + bool "BPF-based THP Policy (EXPERIMENTAL)" + depends on TRANSPARENT_HUGEPAGE && BPF_MM + + help + Enable dynamic THP policy adjustment using BPF programs. This feature + is currently experimental. + + WARNING: This feature is unstable and may change in future kernel +endif # BPF_MM + source "mm/damon/Kconfig" endmenu
mm/Makefile +1 -0
@@ -99,6 +99,7 @@ obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_BPF_THP) += huge_memory_bpf.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
mm/huge_memory_bpf.c new file +316 -0
316 lines not on this page · raw message
mm/mmap.c +1 -0
@@ -1841,6 +1841,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)vma_iter_free (&vmi);if (!retval) {mt_set_in_rcu (vmi.mas.tree); +bpf_thp_fork (mm, oldmm);ksm_fork (mm, oldmm);khugepaged_fork (mm, oldmm); }else {
Signature · 1 line
2.47.3
PATCH v12 4/10 mm: thp: decouple THP allocation between swap and page fault paths
In reply to Yafang ShaoThe new BPF capability enables finer-grained THP policy decisions by introducing separate handling for swap faults versus normal page faults.
As highlighted by Barry:
We’ve observed that swapping in large folios can lead to more swap thrashing for some workloads- e.g. kernel build. Consequently, some workloads might prefer swapping in smaller folios than those allocated by alloc_anon_folio().
While prtcl() could potentially be extended to leverage this new policy, doing so would require modifications to the uAPI.
--- include/linux/huge_mm.h | 3 ++- mm/huge_memory.c | 2 +- mm/memory.c | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-)
include/linux/huge_mm.h +2 -1
@@ -96,9 +96,10 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr;enum tva_type {TVA_SMAPS ,/* Exposing "THPeligible:" in smaps. */ -TVA_PAGEFAULT ,/* Serving a page fault. */ +TVA_PAGEFAULT ,/* Serving a non-swap page fault. */ TVA_KHUGEPAGED ,/* Khugepaged collapse. */ TVA_FORCED_COLLAPSE ,/* Forced collapse (e.g. MADV_COLLAPSE). */ +TVA_SWAP_PAGEFAULT ,/* serving a swap page fault. */ };#define thp_vma_allowable_order (vma, type, order) \
mm/huge_memory.c +1 -1
@@ -102,7 +102,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, unsigned long orders) {const bool smaps = type ==TVA_SMAPS ; -const bool in_pf = type ==TVA_PAGEFAULT ; +const bool in_pf = (type ==TVA_PAGEFAULT || type ==TVA_SWAP_PAGEFAULT );const bool forced_collapse = type ==TVA_FORCED_COLLAPSE ;unsigned long supported_orders;vm_flags_t vm_flags = vma->vm_flags;
mm/memory.c +1 -1
@@ -4558,7 +4558,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)* Get a list of all the (large) orders below PMD_ORDER that are enabled * and suitable for swapping THP. */ - orders =thp_vma_allowable_orders (vma,TVA_PAGEFAULT , + orders =thp_vma_allowable_orders (vma,TVA_SWAP_PAGEFAULT ,BIT (PMD_ORDER ) -1 ); orders =thp_vma_suitable_orders (vma, vmf->address, orders); orders = thp_swap_suitable_orders(swp_offset(entry),
Signature · 1 line
2.47.3
Re: [PATCH v12 mm-new 04/10] mm: thp: decouple THP allocation between swap and page fault paths
Quoted 11 lines · Yafang Shao
The new BPF capability enables finer-grained THP policy decisions by introducing separate handling for swap faults versus normal page faults.
As highlighted by Barry:
We’ve observed that swapping in large folios can lead to more swap thrashing for some workloads- e.g. kernel build. Consequently, some workloads might prefer swapping in smaller folios than those allocated by
alloc_anon_folio().
Quoted 7 lines
While
prtcl()could potentially be extended to leverage this new policy, doing so would require modifications to the uAPI.Signed-off-by Yafang Shao <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Acked-by Usama Arif <[email protected]>Cc Barry Song <[email protected]>
Thanks for addressing this.
Thanks
Barry
PATCH v12 5/10 mm: thp: enable THP allocation exclusively through khugepaged
In reply to Yafang Shaokhugepaged_enter_vma() ultimately invokes any attached BPF function with the TVA_KHUGEPAGED flag set when determining whether or not to enable khugepaged THP for a freshly faulted in VMA.
Currently, on fault, we invoke this in do_huge_pmd_anonymous_page(), as invoked by create_huge_pmd() and only when we have already checked to see if an allowable TVA_PAGEFAULT order is specified.
Since we might want to disallow THP on fault-in but allow it via khugepaged, we move things around so we always attempt to enter khugepaged upon fault.
This change is safe because:
- khugepaged operates at the MM level rather than per-VMA. The THP allocation might fail during page faults due to transient conditions (e.g., memory pressure), it is safe to add this MM to khugepaged for subsequent defragmentation.
- If __thp_vma_allowable_orders(TVA_PAGEFAULT) returns 0, then __thp_vma_allowable_orders(TVA_KHUGEPAGED) will also return 0.
While we could also extend prctl() to utilize this new policy, such a change would require a uAPI modification to PR_SET_THP_DISABLE.
--- mm/huge_memory.c | 1 - mm/memory.c | 13 ++++++++----- 2 files changed, 8 insertions(+), 6 deletions(-)
mm/huge_memory.c +0 -1
@@ -1476,7 +1476,6 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) ret =vmf_anon_prepare (vmf);if (ret)return ret; -khugepaged_enter_vma (vma); if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm) &&
mm/memory.c +8 -5
@@ -6336,11 +6336,14 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,if (pud_trans_unstable (vmf.pud)) goto retry_pud; -if (pmd_none (*vmf.pmd) && -thp_vma_allowable_order (vma,TVA_PAGEFAULT ,PMD_ORDER )) { - ret =create_huge_pmd (&vmf); -if (!(ret &VM_FAULT_FALLBACK )) -return ret; +if (pmd_none (*vmf.pmd)) { +if (vma_is_anonymous (vma)) +khugepaged_enter_vma (vma); +if (thp_vma_allowable_order (vma,TVA_PAGEFAULT ,PMD_ORDER )) { + ret =create_huge_pmd (&vmf); +if (!(ret &VM_FAULT_FALLBACK )) +return ret; + } }else { vmf.orig_pmd =pmdp_get_lockless (vmf.pmd);
Signature · 1 line
2.47.3
PATCH v12 6/10 mm: bpf-thp: add support for global mode
In reply to Yafang ShaoThe per-process BPF-THP mode is unsuitable for managing shared resources such as shmem THP and file-backed THP. This aligns with known cgroup limitations for similar scenarios [0].
Introduce a global BPF-THP mode to address this gap. When registered:
- All existing per-process instances are disabled
- New per-process registrations are blocked
- Existing per-process instances remain registered (no forced unregistration)
The global mode takes precedence over per-process instances. Updates are
type-isolated: global instances can only be updated by new global instances, and per-process instances by new per-process instances.
--- mm/huge_memory_bpf.c | 111 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 2 deletions(-)
mm/huge_memory_bpf.c +109 -2
@@ -35,6 +35,30 @@ struct bpf_thp_ops { };static DEFINE_SPINLOCK (thp_ops_lock); +static struct bpf_thp_ops __rcu *bpf_thp_global;/* global mode */ + +static unsigned long +bpf_hook_thp_get_orders_global (struct vm_area_struct *vma, +enum tva_type type, +unsigned long orders) +{ +static struct bpf_thp_ops *bpf_thp; +int bpf_order; + +rcu_read_lock (); + bpf_thp =rcu_dereference (bpf_thp_global); +if (!bpf_thp || !bpf_thp->thp_get_order) + goto out; + + bpf_order = bpf_thp->thp_get_order (vma, type, orders); +if (bpf_order <0 ) + goto out; + orders &=BIT (bpf_order); + +out: +rcu_read_unlock (); +return orders; +} unsigned long bpf_hook_thp_get_orders(struct vm_area_struct *vma, enum tva_type type,
@@ -47,6 +71,10 @@ unsigned long bpf_hook_thp_get_orders(struct vm_area_struct *vma,if (!mm)return orders; +/* Global BPF-THP takes precedence over per-process BPF-THP. */ +if (rcu_access_pointer (bpf_thp_global)) +return bpf_hook_thp_get_orders_global (vma, type, orders); +rcu_read_lock (); bpf_thp =rcu_dereference (mm->bpf_mm.bpf_thp);if (!bpf_thp || !bpf_thp->thp_get_order)
@@ -181,6 +209,23 @@ static int bpf_thp_init_member(const struct btf_type *t,return 0 ; } +static int bpf_thp_reg_gloabl (void *kdata,struct bpf_link *link) +{ +struct bpf_thp_ops *ops = kdata; + +/* Protect the global pointer bpf_thp_global from concurrent writes. */ +spin_lock (&thp_ops_lock); +/* Only one instance is allowed. */ +if (rcu_access_pointer (bpf_thp_global)) { +spin_unlock (&thp_ops_lock); +return -EBUSY ; + } + +rcu_assign_pointer (bpf_thp_global, ops); +spin_unlock (&thp_ops_lock); +return 0 ; +} +static int bpf_thp_reg (void *kdata,struct bpf_link *link) {struct bpf_thp_ops *bpf_thp = kdata;
@@ -191,6 +236,11 @@ static int bpf_thp_reg(void *kdata, struct bpf_link *link)pid_t pid; pid = bpf_thp->pid; + +/* Fallback to global mode if pid is not set. */ +if (!pid) +return bpf_thp_reg_gloabl (kdata, link); + p =find_get_task_by_vpid (pid);if (!p)return -ESRCH ;
@@ -209,8 +259,10 @@ static int bpf_thp_reg(void *kdata, struct bpf_link *link)* might register this task simultaneously. */ spin_lock (&thp_ops_lock); -/* Each process is exclusively managed by a single BPF-THP. */ -if (rcu_access_pointer (mm->bpf_mm.bpf_thp)) { +/* Each process is exclusively managed by a single BPF-THP. +* Global mode disables per-process instances. +*/ +if (rcu_access_pointer (mm->bpf_mm.bpf_thp) ||rcu_access_pointer (bpf_thp_global)) { err = -EBUSY ; goto out; }
@@ -226,12 +278,33 @@ static int bpf_thp_reg(void *kdata, struct bpf_link *link)return err; } +static void bpf_thp_unreg_global (void *kdata,struct bpf_link *link) +{ +struct bpf_thp_ops *bpf_thp; + +spin_lock (&thp_ops_lock); +if (!rcu_access_pointer (bpf_thp_global)) { +spin_unlock (&thp_ops_lock); +return ; + } + + bpf_thp =rcu_replace_pointer (bpf_thp_global,NULL , +lockdep_is_held (&thp_ops_lock)); +WARN_ON_ONCE (!bpf_thp); +spin_unlock (&thp_ops_lock); + +synchronize_rcu (); +} +static void bpf_thp_unreg (void *kdata,struct bpf_link *link) {struct bpf_thp_ops *bpf_thp = kdata;struct bpf_mm_ops *bpf_mm;struct list_head *pos, *n; +if (!bpf_thp->pid) +return bpf_thp_unreg_global (kdata, link); +spin_lock (&thp_ops_lock);list_for_each_safe (pos, n, &bpf_thp->mm_list) { bpf_mm = list_entry(pos, struct bpf_mm_ops, bpf_thp_list);
@@ -244,6 +317,31 @@ static void bpf_thp_unreg(void *kdata, struct bpf_link *link)synchronize_rcu (); } +static int bpf_thp_update_global (void *kdata,void *old_kdata,struct bpf_link *link) +{ +struct bpf_thp_ops *old_bpf_thp = old_kdata; +struct bpf_thp_ops *bpf_thp = kdata; +struct bpf_thp_ops *old_global; + +if (!old_bpf_thp || !bpf_thp) +return -EINVAL ; + +spin_lock (&thp_ops_lock); +/* BPF-THP global instance has already been removed. */ +if (!rcu_access_pointer (bpf_thp_global)) { +spin_unlock (&thp_ops_lock); +return -ENOENT ; + } + + old_global =rcu_replace_pointer (bpf_thp_global, bpf_thp, +lockdep_is_held (&thp_ops_lock)); +WARN_ON_ONCE (!old_global); +spin_unlock (&thp_ops_lock); + +synchronize_rcu (); +return 0 ; +} +static int bpf_thp_update (void *kdata,void *old_kdata,struct bpf_link *link) {struct bpf_thp_ops *old_bpf_thp = old_kdata;
@@ -251,6 +349,15 @@ static int bpf_thp_update(void *kdata, void *old_kdata, struct bpf_link *link)struct bpf_mm_ops *bpf_mm;struct list_head *pos, *n; +/* Updates are confined to instances of the same scope: +* global to global, process-local to process-local. +*/ +if (!!old_bpf_thp->pid != !!bpf_thp->pid) +return -EINVAL ; + +if (!old_bpf_thp->pid) +return bpf_thp_update_global (kdata, old_kdata, link); +INIT_LIST_HEAD (&bpf_thp->mm_list);/* Could be optimized to a per-instance lock if this lock becomes a bottleneck. */
Signature · 1 line
2.47.3
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 13 lines · Yafang Shao
The per-process BPF-THP mode is unsuitable for managing shared resources such as shmem THP and file-backed THP. This aligns with known cgroup limitations for similar scenarios [0].
Introduce a global BPF-THP mode to address this gap. When registered:
- All existing per-process instances are disabled
- New per-process registrations are blocked
- Existing per-process instances remain registered (no forced unregistration)The global mode takes precedence over per-process instances. Updates are
type-isolated: global instances can only be updated by new global instances, and per-process instances by new per-process instances.
...
mm/huge_memory_bpf.c static int bpf_thp_reg(void *kdata, struct bpf_link *link) · line 268
Quoted 10 lines
spin_lock(&thp_ops_lock);mm/huge_memory_bpf.c +4
-2-/* Each process is exclusively managed by a single BPF-THP. */ -if (rcu_access_pointer (mm->bpf_mm.bpf_thp)) { +/* Each process is exclusively managed by a single BPF-THP. +* Global mode disables per-process instances. +*/ +if (rcu_access_pointer (mm->bpf_mm.bpf_thp) ||rcu_access_pointer (bpf_thp_global)) { err = -EBUSY ; goto out; }
You didn't address the issue and instead doubled down on this broken global approach.
This bait-and-switch patchset is frankly disingenuous. 'lets code up some per-mm hack, since people will hate it anyway, and I'm not going to use it either, and add this global mode as a fake "fallback"...'
The way the previous thread evolved and this followup hack I don't see a genuine desire to find a solution. Just relentless push for global mode.
Please carry it in all future patches.
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 37 lines · Alexei Starovoitov
Quoted 13 lines · Yafang Shao
The per-process BPF-THP mode is unsuitable for managing shared resources such as shmem THP and file-backed THP. This aligns with known cgroup limitations for similar scenarios [0].
Introduce a global BPF-THP mode to address this gap. When registered:
- All existing per-process instances are disabled
- New per-process registrations are blocked
- Existing per-process instances remain registered (no forced unregistration)The global mode takes precedence over per-process instances. Updates are
type-isolated: global instances can only be updated by new global instances, and per-process instances by new per-process instances....
mm/huge_memory_bpf.c static int bpf_thp_reg(void *kdata, struct bpf_link *link) · line 268
Quoted 10 lines
spin_lock(&thp_ops_lock);mm/huge_memory_bpf.c +4
-2-/* Each process is exclusively managed by a single BPF-THP. */ -if (rcu_access_pointer (mm->bpf_mm.bpf_thp)) { +/* Each process is exclusively managed by a single BPF-THP. +* Global mode disables per-process instances. +*/ +if (rcu_access_pointer (mm->bpf_mm.bpf_thp) ||rcu_access_pointer (bpf_thp_global)) { err = -EBUSY ; goto out; }You didn't address the issue and instead doubled down on this broken global approach.
This bait-and-switch patchset is frankly disingenuous. 'lets code up some per-mm hack, since people will hate it anyway, and I'm not going to use it either, and add this global mode as a fake "fallback"...'
Quoted 7 lines
The way the previous thread evolved and this followup hack I don't see a genuine desire to find a solution. Just relentless push for global mode.
Nacked-by Alexei Starovoitov <[email protected]>Please carry it in all future patches.
To move forward, I'm happy to set the global mode aside for now and potentially drop it in the next version. I'd really like to hear your perspective on the per-process mode. Does this implementation meet your needs?
--
Regards
Yafang
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 44 lines · Yafang Shao
Quoted 41 lines · Alexei Starovoitov
Quoted 13 lines · Yafang Shao
The per-process BPF-THP mode is unsuitable for managing shared resources such as shmem THP and file-backed THP. This aligns with known cgroup limitations for similar scenarios [0].
Introduce a global BPF-THP mode to address this gap. When registered:
- All existing per-process instances are disabled
- New per-process registrations are blocked
- Existing per-process instances remain registered (no forced unregistration)The global mode takes precedence over per-process instances. Updates are
type-isolated: global instances can only be updated by new global instances, and per-process instances by new per-process instances....
mm/huge_memory_bpf.c static int bpf_thp_reg(void *kdata, struct bpf_link *link) · line 268
Quoted 10 lines
spin_lock(&thp_ops_lock);mm/huge_memory_bpf.c +4
-2-/* Each process is exclusively managed by a single BPF-THP. */ -if (rcu_access_pointer (mm->bpf_mm.bpf_thp)) { +/* Each process is exclusively managed by a single BPF-THP. +* Global mode disables per-process instances. +*/ +if (rcu_access_pointer (mm->bpf_mm.bpf_thp) ||rcu_access_pointer (bpf_thp_global)) { err = -EBUSY ; goto out; }You didn't address the issue and instead doubled down on this broken global approach.
This bait-and-switch patchset is frankly disingenuous. 'lets code up some per-mm hack, since people will hate it anyway, and I'm not going to use it either, and add this global mode as a fake "fallback"...'
The way the previous thread evolved and this followup hack I don't see a genuine desire to find a solution. Just relentless push for global mode.
Quoted 8 lines
Quoted 3 lines
Nacked-by Alexei Starovoitov <[email protected]>Please carry it in all future patches.
To move forward, I'm happy to set the global mode aside for now and potentially drop it in the next version. I'd really like to hear your perspective on the per-process mode. Does this implementation meet your needs?
Attaching st_ops to task_struct or to mm_struct is a can of worms. With cgroup-bpf we went through painful bugs with lifetime of cgroup vs bpf, dying cgroups, wq deadlock, etc. All these problems are behind us. With st_ops in mm_struct it will be more painful. I'd rather not go that route.
And revist cgroup instead, since you were way too quick to accept the pushback because all you wanted is global mode.
The main reason for pushback was:
"
Cgroup was designed for resource management not for grouping processes and tune those processes
"
which was true when cgroup-v2 was designed, but that ship sailed years ago when we introduced cgroup-bpf.
None of the progs are doing resource management and lots of infrastructure, container management, and open source projects use cgroup-bpf as a grouping of processes. bpf progs attached to cgroup/hook tuple only care about processes within that cgroup. No resource management. See __cgroup_bpf_check_dev_permission or __cgroup_bpf_run_filter_sysctl and others.
The path is current->cgroup->bpf_progs and progs do exactly what cgroup wasn't designed to do. They tune a set of processes.
You should do the same.
Also I really don't see a compelling use case for bpf in THP.
Your selftest is beyond primitive:
+int pmd_order;
+
+SEC("struct_ops/thp_get_order")
+int BPF_PROG(thp_not_eligible, struct vm_area_struct *vma, enum tva_type type,
+ unsigned long orders)
+{
+ /* THPeligible in /proc/pid/smaps is 0 */
+ if (type == TVA_SMAPS)
+ return 0;
+ return pmd_order;
+}
hard code this thing. Don't bother with bpf.
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 48 lines · Alexei Starovoitov
Quoted 46 lines · Yafang Shao
Quoted 43 lines · Alexei Starovoitov
Quoted 13 lines · Yafang Shao
The per-process BPF-THP mode is unsuitable for managing shared resources such as shmem THP and file-backed THP. This aligns with known cgroup limitations for similar scenarios [0].
Introduce a global BPF-THP mode to address this gap. When registered:
- All existing per-process instances are disabled
- New per-process registrations are blocked
- Existing per-process instances remain registered (no forced unregistration)The global mode takes precedence over per-process instances. Updates are
type-isolated: global instances can only be updated by new global instances, and per-process instances by new per-process instances....
mm/huge_memory_bpf.c static int bpf_thp_reg(void *kdata, struct bpf_link *link) · line 268
Quoted 10 lines
spin_lock(&thp_ops_lock);mm/huge_memory_bpf.c +4
-2-/* Each process is exclusively managed by a single BPF-THP. */ -if (rcu_access_pointer (mm->bpf_mm.bpf_thp)) { +/* Each process is exclusively managed by a single BPF-THP. +* Global mode disables per-process instances. +*/ +if (rcu_access_pointer (mm->bpf_mm.bpf_thp) ||rcu_access_pointer (bpf_thp_global)) { err = -EBUSY ; goto out; }You didn't address the issue and instead doubled down on this broken global approach.
This bait-and-switch patchset is frankly disingenuous. 'lets code up some per-mm hack, since people will hate it anyway, and I'm not going to use it either, and add this global mode as a fake "fallback"...'
The way the previous thread evolved and this followup hack I don't see a genuine desire to find a solution. Just relentless push for global mode.
Nacked-by Alexei Starovoitov <[email protected]>
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Please carry it in all future patches.
To move forward, I'm happy to set the global mode aside for now and potentially drop it in the next version. I'd really like to hear your perspective on the per-process mode. Does this implementation meet your needs?
Attaching
st_opstotask_structor tomm_structis a can of worms.
The feedback suggests there may not have been an opportunity to review patch #3 in detail yet. I would appreciate it if you could take a look at the specific changes in that patch, as it addresses the core of the implementation.
Quoted 3 lines
With cgroup-bpf we went through painful bugs with lifetime of cgroup vs bpf, dying cgroups, wq deadlock, etc. All these problems are behind us.
The attachment-based design of cgroup-bpf creates significant operational challenges. It lacks visibility, making it difficult to identify which cgroups have active attachments, and requires explicit author knowledge for clean detachment.
Quoted 2 lines
With
st_opsinmm_structit will be more
painful.
To save your time, I've pasted the relevant portion of patch #3 below:
When registering a BPF-THP, we specify the PID of a target task. The BPF-THP is then installed in the task's mm_struct
struct mm_struct {
struct bpf_thp_ops __rcu *thp_thp;
};Inheritance Behavior:
- Existing child processes are unaffected
- Newly forked children inherit the BPF-THP from their parent
- The BPF-THP persists across execve() calls
A new linked list tracks all tasks managed by each BPF-THP instance:
- Newly managed tasks are added to the list
- Exiting tasks are automatically removed from the list
- During BPF-THP unregistration (e.g., when the BPF link is removed), all managed tasks have their bpf_thp pointer set to NULL
- BPF-THP instances can be dynamically updated, with all tracked tasks automatically migrating to the new version.
This design simplifies BPF-THP management in production environments by providing clear lifecycle management and preventing conflicts between multiple BPF-THP instances.
To clarify, this design has no lifecycle issues. It provides clear
traceability: you can always identify who loaded the program and which processes it's attached to. Moreover, removing either the loader or the pinned bpf_link will completely remove the program and all its associated state.
Quoted 1 line
I'd rather not go that route.
I'm glad we can talk about this directly—it saves us both a lot of guesswork.
Quoted 21 lines
And revist cgroup instead, since you were way too quick to accept the pushback because all you wanted is global mode.
The main reason for pushback was:
"
Cgroup was designed for resource management not for grouping processes and tune those processes
"which was true when cgroup-v2 was designed, but that ship sailed years ago when we introduced cgroup-bpf.
None of the progs are doing resource management and lots of infrastructure, container management, and open source projects use cgroup-bpf as a grouping of processes. bpf progs attached to cgroup/hook tuple only care about processes within that cgroup. No resource management. See__cgroup_bpf_check_dev_permissionor__cgroup_bpf_run_filter_sysctland others.
The path iscurrent->cgroup->bpf_progsand progs do exactly what cgroup wasn't designed to do. They tune a set of processes.
Quoted 1 line
You should do the same.
I'm fully supportive of a cgroup-based approach, as it simplifies integration by requiring only a kubelet plugin instead of modifications to containerd.
However, my primary concern is the potential for maintainer pushback, given the historical precedent. For instance, a similar discussion in the NUMA-balancing context saw cgroup maintainers insisting on a process-based method (see link below):
https://lore.kernel.org/lkml/ldyynnd3ngxnu3bie7ezuavewshgfepro5kjids6cuxy4imzy5@nt5id7nj5kt7/
To proactively address this, what alternative plan would you recommend if we encounter such resistance? It's unclear what a viable path forward would be if we are committed to a cgroup-based approach but it is ultimately rejected by the maintainers.
(Adding Michal to CC for visibility)
Quoted 2 lines
Also I really don't see a compelling use case for bpf in THP.
I'd recommend familiarizing yourself with the THP implementation. This would be beneficial for our discussion on the specific changes.
Quoted 12 lines
Your selftest is beyond primitive:
(quoted diff) +11
-0+int pmd_order; + +SEC("struct_ops/thp_get_order") +int BPF_PROG(thp_not_eligible, struct vm_area_struct *vma, enum tva_type type, + unsigned long orders) +{ + /* THPeligible in /proc/pid/smaps is 0 */ + if (type == TVA_SMAPS) + return 0; + return pmd_order; +}
Quoted 1 line
hard code this thing. Don't bother with bpf.
A prior implementation that combined these components existed in an earlier version:
https://lore.kernel.org/linux-mm/[email protected]/
However, based on your previous guidance that fexit and struct_ops should not be mixed, the current approach was adopted.
In summary, I'm happy to proceed with a cgroup-based implementation. I would appreciate your support in addressing any concerns the cgroup maintainers might have.
Signature · 2 lines
Regards Yafang
David Hildenbrand (Red Hat)
maintainerRe: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
In reply to Alexei StarovoitovQuoted 4 lines
Quoted 4 lines
To move forward, I'm happy to set the global mode aside for now and potentially drop it in the next version. I'd really like to hear your perspective on the per-process mode. Does this implementation meet your needs?
I haven't had the capacity to follow the evolution of this patch set unfortunately, just to comment on some points from my perspective.
First, I agree that the global mode is not what we want, not even as a fallback.
Quoted 6 lines
Attaching
st_opstotask_structor tomm_structis a can of worms. With cgroup-bpf we went through painful bugs with lifetime of cgroup vs bpf, dying cgroups, wq deadlock, etc. All these problems are behind us. Withst_opsinmm_structit will be more painful. I'd rather not go that route.
That's valuable information, thanks. I would have hoped that per-MM policies would be easier.
Are there some pointers to explore regarding the "can of worms" you mention when it comes to per-MM policies?
Quoted 4 lines
And revist cgroup instead, since you were way too quick to accept the pushback because all you wanted is global mode.
Quoted 8 lines
The main reason for pushback was:
"
Cgroup was designed for resource management not for grouping processes and tune those processes
"which was true when cgroup-v2 was designed, but that ship sailed years ago when we introduced cgroup-bpf.
Also valuable information.
Personally I don't have a preference regarding per-mm or per-cgroup. Whatever we can get working reliably. Sounds like cgroup-bpf has sorted out most of the mess.
memcg/cgroup maintainers might disagree, but it's probably worth having that discussion once again.
Quoted 9 lines
None of the progs are doing resource management and lots of infrastructure, container management, and open source projects use cgroup-bpf as a grouping of processes. bpf progs attached to cgroup/hook tuple only care about processes within that cgroup. No resource management. See
__cgroup_bpf_check_dev_permissionor__cgroup_bpf_run_filter_sysctland others.
The path iscurrent->cgroup->bpf_progsand progs do exactly what cgroup wasn't designed to do. They tune a set of processes.
Quoted 3 lines
You should do the same.
Also I really don't see a compelling use case for bpf in THP.
There is a lot more potential there to write fine-tuned policies that thack VMA information into account.
The tests likely reflect what Yafang seems to focus on: IIUC primarily enabling+disabling traditional THPs (e.g., 2M) on a per-process basis.
Some of what Yafang might want to achieve could maybe at this point be maybe achieved through the prctl(PR_SET_THP_DISABLE) support, including extensions we recently added [1].
Systemd support still seems to be in the works [2] for some of that.
[1] https://lwn.net/Articles/1032014/
[2] https://github.com/systemd/systemd/pull/39085
Signature · 3 lines
Cheers David
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 13 lines · David Hildenbrand (Red Hat)
Quoted 5 lines
Quoted 5 lines
To move forward, I'm happy to set the global mode aside for now and potentially drop it in the next version. I'd really like to hear your perspective on the per-process mode. Does this implementation meet your needs?
I haven't had the capacity to follow the evolution of this patch set unfortunately, just to comment on some points from my perspective.
First, I agree that the global mode is not what we want, not even as a fallback.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Attaching
st_opstotask_structor tomm_structis a can of worms. With cgroup-bpf we went through painful bugs with lifetime of cgroup vs bpf, dying cgroups, wq deadlock, etc. All these problems are behind us. Withst_opsinmm_structit will be more painful. I'd rather not go that route.That's valuable information, thanks. I would have hoped that per-MM policies would be easier.
The per-MM approach has a performance advantage over per-MEMCG policies. This is because it accesses the policy hook directly via
vma->vm_mm->bpf_mm->policy_hook()
whereas the per-MEMCG method requires a more expensive lookup:
memcg = get_mem_cgroup_from_mm(vma->vm_mm); memcg->bpf_memcg->policy_hook();
This lookup could be a concern in a critical path. However, this performance issue in the per-MEMCG mode can be mitigated. For instance, when a task is added to a new memcg, we can cache the hook pointer:
task->mm->bpf_mm->policy_hook = memcg->bpf_memcg->policy_hook
Ultimately, we might still introduce a mm_struct:bpf_mm field to provide an efficient interface.
Quoted 14 lines
Are there some pointers to explore regarding the "can of worms" you mention when it comes to per-MM policies?
Quoted 10 lines
And revist cgroup instead, since you were way too quick to accept the pushback because all you wanted is global mode.
The main reason for pushback was:
"
Cgroup was designed for resource management not for grouping processes and tune those processes
"
Quoted 7 lines
Quoted 2 lines
which was true when cgroup-v2 was designed, but that ship sailed years ago when we introduced cgroup-bpf.
Also valuable information.
Personally I don't have a preference regarding per-mm or per-cgroup. Whatever we can get working reliably.
I am open to either approach, as long as it's acceptable to the maintainers.
Quoted 2 lines
Sounds like cgroup-bpf has sorted
out most of the mess.
No, the attach-based cgroup-bpf has proven to be ... a "can of worms" in practice ...
(I welcome corrections from the BPF maintainers if my assessment is
inaccurate.)
While the struct-ops-based cgroup-bpf is still under discussion.
Quoted 15 lines
memcg/cgroup maintainers might disagree, but it's probably worth having that discussion once again.
Quoted 11 lines
None of the progs are doing resource management and lots of infrastructure, container management, and open source projects use cgroup-bpf as a grouping of processes. bpf progs attached to cgroup/hook tuple only care about processes within that cgroup. No resource management. See
__cgroup_bpf_check_dev_permissionor__cgroup_bpf_run_filter_sysctland others.
The path iscurrent->cgroup->bpf_progsand progs do exactly what cgroup wasn't designed to do. They tune a set of processes.You should do the same.
Quoted 7 lines
Quoted 1 line
Also I really don't see a compelling use case for bpf in THP.
There is a lot more potential there to write fine-tuned policies that thack VMA information into account.
The tests likely reflect what Yafang seems to focus on: IIUC primarily enabling+disabling traditional THPs (e.g., 2M) on a per-process basis.
Right.
Quoted 10 lines
Some of what Yafang might want to achieve could maybe at this point be maybe achieved through the
prctl(PR_SET_THP_DISABLE)support, including extensions we recently added [1].Systemd support still seems to be in the works [2] for some of that.
[1] https://lwn.net/Articles/1032014/
[2] https://github.com/systemd/systemd/pull/39085
Thank you for sharing this.
However, BPF-THP is already deployed across our server fleet and both our users and my boss are satisfied with it. As such, we are not considering a switch. The current solution also offers us a valuable opportunity to experiment with additional policies in production.
In summary, I am fine with either the per-MM or per-MEMCG method. Furthermore, I don't believe this is an either-or decision; both can be implemented to work together.
--
Regards
Yafang
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
TL;DR - NAK this series as-is.
Quoted 5 lines · Yafang Shao
Thank you for sharing this.
However, BPF-THP is already deployed across our server fleet and both our users and my boss are satisfied with it. As such, we are not considering a switch. The current solution also offers us a valuable opportunity to experiment with additional policies in production.
Sorry Yafang, this isn't how upstream works.
I've not been paying attention to this series as I have been waiting for you and Alexei to reach some kind of resolution before diving back in.
But your response here is _very_ concerning to me.
Of course you're welcome to deploy unmerged arbitrary patches to your kernel (as long as you abide by the GPL naturally).
But we've made it _very_ clear that this is an - experimental - feature, that might go away at any time, while we iterate and determine how useful it might be to users in general.
Now it seems that exactly the thing I feared has already happened - people ignoring the fact we are hiding this behind an, in effect, CONFIG_EXPERIMENTAL_PLEASE_DO_NOT_RELY_ON_THIS flag.
This means that I am no longer confident this approach is going to work, which inclines me to reject this proposal outright.
The bar is now a lot higher in my view, and now we're going to need extensive and overwhelming evidence that whatever BPF hook we provide is both future proof as to how we intend THP to develop and of use to more than one user.
Again as David mentioned, you seem to be able to achieve what you want to achieve via the extensions we added to PR_SET_THP_DISABLE.
That then reduces the number of users of this feature to 0 and again inclines me to reject this approach entirely.
So for now it's a NAK.
Quoted 4 lines
In summary, I am fine with either the per-MM or per-MEMCG method. Furthermore, I don't believe this is an either-or decision; both can be implemented to work together.
No, it is - the global approach is broken and we won't be having that.
Quoted 5 lines
--
Regards
Yafang
Thanks, Lorenzo
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 20 lines · Lorenzo Stoakes
TL;DR - NAK this series as-is.
Quoted 5 lines · Yafang Shao
Thank you for sharing this.
However, BPF-THP is already deployed across our server fleet and both our users and my boss are satisfied with it. As such, we are not considering a switch. The current solution also offers us a valuable opportunity to experiment with additional policies in production.Sorry Yafang, this isn't how upstream works.
I've not been paying attention to this series as I have been waiting for you and Alexei to reach some kind of resolution before diving back in.
But your response here is _very_ concerning to me.
Of course you're welcome to deploy unmerged arbitrary patches to your kernel (as long as you abide by the GPL naturally).
Quoted 7 lines
But we've made it _very_ clear that this is an - experimental - feature, that might go away at any time, while we iterate and determine how useful it might be to users in general.
Now it seems that exactly the thing I feared has already happened - people ignoring the fact we are hiding this behind an, in effect,
CONFIG_EXPERIMENTAL_PLEASE_DO_NOT_RELY_ON_THISflag.
Thank you for your concern. We have a dedicated kernel team that maintains our runtime. Our standard practice for new kernel features is to first validate them in our production environment. This ensures that any feature we propose to upstream has been proven in a real-world, large-scale use case.
Quoted 11 lines
This means that I am no longer confident this approach is going to work, which inclines me to reject this proposal outright.
The bar is now a lot higher in my view, and now we're going to need extensive and overwhelming evidence that whatever BPF hook we provide is both future proof as to how we intend THP to develop and of use to more than one user.
Again as David mentioned, you seem to be able to achieve what you want to achieve via the extensions we added to
PR_SET_THP_DISABLE.
We see no compelling reason to switch to PR_SET_THP_DISABLE. BPF-THP has proven to be perfectly stable across our production fleet, and we have the full capability to maintain it.
Quoted 3 lines
That then reduces the number of users of this feature to 0 and again inclines me to reject this approach entirely.
I understand your concern. Our intention is simply to contribute a feature that we have found valuable in production, in the hope that it may benefit others as well. We of course respect the upstream process and are fully prepared for the possibility that it may not be accepted.
Quoted 9 lines
So for now it's a NAK.
Quoted 4 lines
In summary, I am fine with either the per-MM or per-MEMCG method. Furthermore, I don't believe this is an either-or decision; both can be implemented to work together.
No, it is - the global approach is broken and we won't be having that.
Let me rephrase for clarity: I see the per-MM and per-MEMCG approaches as compatible. They can be implemented together, potentially as a hybrid approach.
Signature · 2 lines
Regards Yafang
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 30 lines · Yafang Shao
Quoted 20 lines · Lorenzo Stoakes
TL;DR - NAK this series as-is.
Quoted 5 lines · Yafang Shao
Thank you for sharing this.
However, BPF-THP is already deployed across our server fleet and both our users and my boss are satisfied with it. As such, we are not considering a switch. The current solution also offers us a valuable opportunity to experiment with additional policies in production.Sorry Yafang, this isn't how upstream works.
I've not been paying attention to this series as I have been waiting for you and Alexei to reach some kind of resolution before diving back in.
But your response here is _very_ concerning to me.
Of course you're welcome to deploy unmerged arbitrary patches to your kernel (as long as you abide by the GPL naturally).
Quoted 7 lines
But we've made it _very_ clear that this is an - experimental - feature, that might go away at any time, while we iterate and determine how useful it might be to users in general.
Now it seems that exactly the thing I feared has already happened - people ignoring the fact we are hiding this behind an, in effect,
CONFIG_EXPERIMENTAL_PLEASE_DO_NOT_RELY_ON_THISflag.
Quoted 5 lines
Thank you for your concern. We have a dedicated kernel team that maintains our runtime. Our standard practice for new kernel features is to first validate them in our production environment. This ensures that any feature we propose to upstream has been proven in a real-world, large-scale use case.
This strictly contradicts the intent of the config flag. I seem to recall asking to put 'experimental' in the flag name also to avoid people assuming this is permanent or at least permanently implemented as-is. But this iteration of the series doesn't...
I no longer believe this flag achieves the stated goal, which is to give us latitude to make changes in the future based on internal changes to THP (which so sorely needs them).
I fear we will end up with users depending on it should we ship any form of BPF hook that we aren't 100% certain is 'future proof', so it raises the bar for this work very substantially.
So I am really of a mind that we shouldn't be taking any such series at this point in time.
Quoted 10 lines
Quoted 10 lines
This means that I am no longer confident this approach is going to work, which inclines me to reject this proposal outright.
The bar is now a lot higher in my view, and now we're going to need extensive and overwhelming evidence that whatever BPF hook we provide is both future proof as to how we intend THP to develop and of use to more than one user.
Quoted 6 lines
Quoted 2 lines
Again as David mentioned, you seem to be able to achieve what you want to achieve via the extensions we added to
PR_SET_THP_DISABLE.We see no compelling reason to switch to
PR_SET_THP_DISABLE. BPF-THP has proven to be perfectly stable across our production fleet, and we have the full capability to maintain it.
Again, this is entirely your prerogative, but it doesn't imply that other users will need this feature themselves.
Quoted 10 lines
Quoted 4 lines
That then reduces the number of users of this feature to 0 and again inclines me to reject this approach entirely.
I understand your concern. Our intention is simply to contribute a feature that we have found valuable in production, in the hope that it may benefit others as well. We of course respect the upstream process and are fully prepared for the possibility that it may not be accepted.
Right.
Quoted 9 lines
Quoted 9 lines
So for now it's a NAK.
Quoted 4 lines
In summary, I am fine with either the per-MM or per-MEMCG method. Furthermore, I don't believe this is an either-or decision; both can be implemented to work together.
Quoted 5 lines
Quoted 1 line
No, it is - the global approach is broken and we won't be having that.
Let me rephrase for clarity: I see the per-MM and per-MEMCG approaches as compatible. They can be implemented together, potentially as a hybrid approach.
OK sorry I think I misread this/misinterpreted you here - the objection was to the global approach.
Yes sure perhaps we could.
I mean we end up back in the silly 'THPs are not a resource' argument the cgroup people put forward when it comes to memcg + THP (I don't agree...). But let's not open that can of worms again :)
Quoted 5 lines
--
Regards
Yafang
Sorry to push back so harshly on this, but I do it out of concern for our future ability to tame THP into something more sensible than the - frankly
- mess we have now.
I feel like we must defend against painting ourselves into any kind of corner worse than we already have :)
Thanks, Lorenzo
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 38 lines · Lorenzo Stoakes
Quoted 30 lines · Yafang Shao
Quoted 20 lines · Lorenzo Stoakes
TL;DR - NAK this series as-is.
Quoted 5 lines · Yafang Shao
Thank you for sharing this.
However, BPF-THP is already deployed across our server fleet and both our users and my boss are satisfied with it. As such, we are not considering a switch. The current solution also offers us a valuable opportunity to experiment with additional policies in production.Sorry Yafang, this isn't how upstream works.
I've not been paying attention to this series as I have been waiting for you and Alexei to reach some kind of resolution before diving back in.
But your response here is _very_ concerning to me.
Of course you're welcome to deploy unmerged arbitrary patches to your kernel (as long as you abide by the GPL naturally).
Quoted 7 lines
But we've made it _very_ clear that this is an - experimental - feature, that might go away at any time, while we iterate and determine how useful it might be to users in general.
Now it seems that exactly the thing I feared has already happened - people ignoring the fact we are hiding this behind an, in effect,
CONFIG_EXPERIMENTAL_PLEASE_DO_NOT_RELY_ON_THISflag.Quoted 5 lines
Thank you for your concern. We have a dedicated kernel team that maintains our runtime. Our standard practice for new kernel features is to first validate them in our production environment. This ensures that any feature we propose to upstream has been proven in a real-world, large-scale use case.
Quoted 4 lines
This strictly contradicts the intent of the config flag. I seem to recall asking to put 'experimental' in the flag name also to avoid people assuming this is permanent or at least permanently implemented as-is. But this iteration of the series doesn't...
Ah, I understand your point now.
The CONFIG_EXPERIMENTAL_PLEASE_DO_NOT_RELY_ON_THIS flag was changed in v9:
https://lore.kernel.org/linux-mm/[email protected]/
The change was suggested by Randy and Usama:
https://lwn.net/ml/all/[email protected]/
At that time, you were on holiday, so you may have missed this update.
Quoted 11 lines
I no longer believe this flag achieves the stated goal, which is to give us latitude to make changes in the future based on internal changes to THP (which so sorely needs them).
I fear we will end up with users depending on it should we ship any form of BPF hook that we aren't 100% certain is 'future proof', so it raises the bar for this work very substantially.
So I am really of a mind that we shouldn't be taking any such series at this point in time.
understood.
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
This means that I am no longer confident this approach is going to work, which inclines me to reject this proposal outright.
The bar is now a lot higher in my view, and now we're going to need extensive and overwhelming evidence that whatever BPF hook we provide is both future proof as to how we intend THP to develop and of use to more than one user.
Again as David mentioned, you seem to be able to achieve what you want to achieve via the extensions we added to
PR_SET_THP_DISABLE.
Quoted 6 lines
Quoted 3 lines
We see no compelling reason to switch to
PR_SET_THP_DISABLE. BPF-THP has proven to be perfectly stable across our production fleet, and we have the full capability to maintain it.Again, this is entirely your prerogative, but it doesn't imply that other users will need this feature themselves.
Right, we’re not trying to force anyone else to use it. We’re simply sharing our use case with upstream. It’s up to the maintainers to decide whether to accept it.
Quoted 44 lines
Quoted 11 lines
Quoted 5 lines
That then reduces the number of users of this feature to 0 and again inclines me to reject this approach entirely.
I understand your concern. Our intention is simply to contribute a feature that we have found valuable in production, in the hope that it may benefit others as well. We of course respect the upstream process and are fully prepared for the possibility that it may not be accepted.
Right.
Quoted 9 lines
Quoted 9 lines
So for now it's a NAK.
Quoted 4 lines
In summary, I am fine with either the per-MM or per-MEMCG method. Furthermore, I don't believe this is an either-or decision; both can be implemented to work together.
Quoted 5 lines
Quoted 1 line
No, it is - the global approach is broken and we won't be having that.
Let me rephrase for clarity: I see the per-MM and per-MEMCG approaches as compatible. They can be implemented together, potentially as a hybrid approach.
OK sorry I think I misread this/misinterpreted you here - the objection was to the global approach.
Yes sure perhaps we could.
I mean we end up back in the silly 'THPs are not a resource' argument the cgroup people put forward when it comes to memcg + THP (I don't agree...). But let's not open that can of worms again :)
Quoted 5 lines
--
Regards
Yafang
Quoted 6 lines
Sorry to push back so harshly on this, but I do it out of concern for our future ability to tame THP into something more sensible than the - frankly
- mess we have now.I feel like we must defend against painting ourselves into any kind of corner worse than we already have :)
Understood.
--
Regards
Yafang
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 10 lines · Yafang Shao
The
CONFIG_EXPERIMENTAL_PLEASE_DO_NOT_RELY_ON_THISflag was changed in v9:https://lore.kernel.org/linux-mm/[email protected]/
The change was suggested by Randy and Usama:
https://lwn.net/ml/all/[email protected]/
At that time, you were on holiday, so you may have missed this update.
It's moot because this series isn't upstreamable, but... :)
To risk sounding grumpy, in future do please make sure to check about changes that contradict things maintainers _explicitly_ ask you to do.
You can always off-list mail if people take time to come back to review.
Thanks, Lorenzo
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 13 lines · Lorenzo Stoakes
Quoted 10 lines · Yafang Shao
The
CONFIG_EXPERIMENTAL_PLEASE_DO_NOT_RELY_ON_THISflag was changed in v9:https://lore.kernel.org/linux-mm/[email protected]/
The change was suggested by Randy and Usama:
https://lwn.net/ml/all/[email protected]/
At that time, you were on holiday, so you may have missed this update.
Quoted 6 lines
It's moot because this series isn't upstreamable, but... :)
To risk sounding grumpy, in future do please make sure to check about changes that contradict things maintainers _explicitly_ ask you to do.
You can always off-list mail if people take time to come back to review.
Thanks for your suggestion :-)
Signature · 2 lines
Regards Yafang
David Hildenbrand (Red Hat)
maintainerRe: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
In reply to Yafang ShaoQuoted 2 lines · Yafang Shao
On Thu, Nov 27, 2025 at 7:48 PM David Hildenbrand (Red Hat) <[email protected]> wrote:
Lorenzo commented on the upstream topic, let me mostly comment on the other parts:
Quoted 19 lines
Quoted 8 lines
Quoted 5 lines
Attaching
st_opstotask_structor tomm_structis a can of worms. With cgroup-bpf we went through painful bugs with lifetime of cgroup vs bpf, dying cgroups, wq deadlock, etc. All these problems are behind us. Withst_opsinmm_structit will be more painful. I'd rather not go that route.That's valuable information, thanks. I would have hoped that per-MM policies would be easier.
The per-MM approach has a performance advantage over per-MEMCG policies. This is because it accesses the policy hook directly via
vma->vm_mm->bpf_mm->policy_hook()whereas the per-MEMCG method requires a more expensive lookup:
memcg = get_mem_cgroup_from_mm(vma->vm_mm); memcg->bpf_memcg->policy_hook();Quoted 1 line
This lookup could be a concern in a critical path. However, this
Quoted 8 lines
performance issue in the per-MEMCG mode can be mitigated. For instance, when a task is added to a new memcg, we can cache the hook pointer:
task->mm->bpf_mm->policy_hook = memcg->bpf_memcg->policy_hookUltimately, we might still introduce a mm_struct:bpf_mm field to provide an efficient interface.
Right, caching is what I would have proposed. I would expect some headakes with lifetime, but probably nothing unsolvable.
Quoted 7 lines
Quoted 2 lines
Sounds like cgroup-bpf has sorted
out most of the mess.No, the attach-based cgroup-bpf has proven to be ... a "can of worms" in practice ...
(I welcome corrections from the BPF maintainers if my assessment is
inaccurate.)
I don't know what's right or wrong here, as Alexei said the "mm_struct" based one would be a can of worms and that the the cgroup-based one apparently solved these issues ("All these problems are behind us."), that's why I asked for some clarifications. :)
[...]
Quoted 8 lines
Quoted 8 lines
Some of what Yafang might want to achieve could maybe at this point be maybe achieved through the
prctl(PR_SET_THP_DISABLE)support, including extensions we recently added [1].Systemd support still seems to be in the works [2] for some of that.
Quoted 8 lines
Quoted 2 lines
[1] https://lwn.net/Articles/1032014/
[2] https://github.com/systemd/systemd/pull/39085Thank you for sharing this.
However, BPF-THP is already deployed across our server fleet and both our users and my boss are satisfied with it. As such, we are not considering a switch. The current solution also offers us a valuable opportunity to experiment with additional policies in production.
Just to emphasize: we usually don't add two mechanisms to achieve the very same end goal. There really must be something delivering more value for us to accept something more complex. Focusing on solving a solved problem is not good.
If some company went with a downstream-only approach they might be stuck having to maintain that forever.
That's why other companies prefer upstream-first :)
Having that said, the original reason why I agreed that having bpf for THP can be valuable is that I see a lot more value for rapid prototyping and policies once you can actually control on a per-VMA basis (using vma size, flags, anon-vma names etc) where specific folio orders could be valuable, and where not. But also, possibly where we would want to waste memory and where not.
As we are speaking I have a customer running into issues [1] with virtio-balloon discarding pages in a VM and khugepaged undoing part of that work in the hypervisor. The workaround of telling khugepaged to not waste memory in all of the system really feels suboptimal when we know that it's only the VM memory of such VMs (with balloon deflation enabled) where we would not want to waste memory but still use THPs.
[1] https://issues.redhat.com/browse/RHEL-121177
Signature · 3 lines
Cheers David
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 70 lines · David Hildenbrand (Red Hat)
Quoted 2 lines · Yafang Shao
On Thu, Nov 27, 2025 at 7:48 PM David Hildenbrand (Red Hat) <[email protected]> wrote:
Lorenzo commented on the upstream topic, let me mostly comment on the other parts:
Quoted 19 lines
Quoted 8 lines
Quoted 5 lines
Attaching
st_opstotask_structor tomm_structis a can of worms. With cgroup-bpf we went through painful bugs with lifetime of cgroup vs bpf, dying cgroups, wq deadlock, etc. All these problems are behind us. Withst_opsinmm_structit will be more painful. I'd rather not go that route.That's valuable information, thanks. I would have hoped that per-MM policies would be easier.
The per-MM approach has a performance advantage over per-MEMCG policies. This is because it accesses the policy hook directly via
vma->vm_mm->bpf_mm->policy_hook()whereas the per-MEMCG method requires a more expensive lookup:
memcg = get_mem_cgroup_from_mm(vma->vm_mm); memcg->bpf_memcg->policy_hook();Quoted 1 line
This lookup could be a concern in a critical path. However, this
Quoted 8 lines
performance issue in the per-MEMCG mode can be mitigated. For instance, when a task is added to a new memcg, we can cache the hook pointer:
task->mm->bpf_mm->policy_hook = memcg->bpf_memcg->policy_hookUltimately, we might still introduce a mm_struct:bpf_mm field to provide an efficient interface.
Right, caching is what I would have proposed. I would expect some headakes with lifetime, but probably nothing unsolvable.
Quoted 7 lines
Quoted 2 lines
Sounds like cgroup-bpf has sorted
out most of the mess.No, the attach-based cgroup-bpf has proven to be ... a "can of worms" in practice ...
(I welcome corrections from the BPF maintainers if my assessment is
inaccurate.)I don't know what's right or wrong here, as Alexei said the "
mm_struct" based one would be a can of worms and that the the cgroup-based one apparently solved these issues ("All these problems are behind us."), that's why I asked for some clarifications. :)[...]
Quoted 8 lines
Quoted 8 lines
Some of what Yafang might want to achieve could maybe at this point be maybe achieved through the
prctl(PR_SET_THP_DISABLE)support, including extensions we recently added [1].Systemd support still seems to be in the works [2] for some of that.
Quoted 8 lines
Quoted 2 lines
[1] https://lwn.net/Articles/1032014/
[2] https://github.com/systemd/systemd/pull/39085Thank you for sharing this.
However, BPF-THP is already deployed across our server fleet and both our users and my boss are satisfied with it. As such, we are not considering a switch. The current solution also offers us a valuable opportunity to experiment with additional policies in production.
Quoted 4 lines
Just to emphasize: we usually don't add two mechanisms to achieve the very same end goal. There really must be something delivering more value for us to accept something more complex. Focusing on solving a solved problem is not good.
Yes.
Quoted 5 lines
If some company went with a downstream-only approach they might be stuck having to maintain that forever.
That's why other companies prefer upstream-first :)
I think trying to do downstream-only is going to cause very big headaches if we choose to substantially alter THP in future (and of course - we do intend to).
Quoted 8 lines
Having that said, the original reason why I agreed that having bpf for THP can be valuable is that I see a lot more value for rapid prototyping and policies once you can actually control on a per-VMA basis (using vma size, flags, anon-vma names etc) where specific folio orders could be valuable, and where not. But also, possibly where we would want to waste memory and where not.
The same for me.
But given the actual author of the feature has already treated this as a permanent and unchanging feature, I absolutely do not have confidence that we can do this.
The situation I feared us running in to is that we'd release this even with CONFIG_EXPERIMENTAL_DO_NOT_RELY etc. (note the flag is somehow now CONFIG_BPF_THP which... isn't what I wanted) and people would STILL rely on it, then when we try to change it loudly complain and make it difficult to remove.
I am now convinced that this is just going to happen no matter what we do.
So the 'rapid prototyping' approach is just not workable, at all in my view.
Quoted 9 lines
As we are speaking I have a customer running into issues [1] with virtio-balloon discarding pages in a VM and khugepaged undoing part of that work in the hypervisor. The workaround of telling khugepaged to not waste memory in all of the system really feels suboptimal when we know that it's only the VM memory of such VMs (with balloon deflation enabled) where we would not want to waste memory but still use THPs.
Right, and that's very sad that we now lose the ability to do so, but rapid prototyping isn't feasible - I think we're seeing that.
That doesn't mean we can't have BPF for THP. It just means we have to set the bar CONSIDERABLY higher - whatever interface we provide _has_ to be future-proofed to any future changes we make to THP in terms of making things more 'automatic' - and has to provide sufficient power to be useful.
I wonder how easy it will be to figure out such an interface without accidentally causing ourselves issues down the line.
THP is a special case like that - right now we have very broken interfaces (as evidenced by users requesting things like the prctl extensions) - and we want to be able to fix those in the future.
Of course we have to maintain uAPI compatibility, but even the discussion around mTHP khugepaged and 'eagerness' points to a desire to change how existing interfaces work - imagine if we had some BPF hook that then ended up needing to introspect current max_pte_none for instance.
So perhaps the answer is that a BPF interface should come later when we have a better idea of the future of THP?
The whole cgroup vs mm thing again raises old issues about isolation - the cgroup people reject the idea of THP being a resource that can be managed by cgroups - so by even allowing a per-memcg thing we're opening that can of worms.
Anyway overall this series as-is is not really upstreamable I don't think.
Maybe we can figure out a read-only introspection hook that makes the least assumptions that can be provided at low-risk that'd help with issues such as the one you mention at least in respect of informing what's going on?
That could form the basis of future work towards a hook that actually changes things?
There's no need to rush.
Quoted 5 lines
--
CheersDavid
Thanks, Lorenzo
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
In reply to David Hildenbrand (Red Hat)Quoted 76 lines · David Hildenbrand (Red Hat)
Quoted 2 lines · Yafang Shao
On Thu, Nov 27, 2025 at 7:48 PM David Hildenbrand (Red Hat) <[email protected]> wrote:
Lorenzo commented on the upstream topic, let me mostly comment on the other parts:
Quoted 19 lines
Quoted 8 lines
Quoted 5 lines
Attaching
st_opstotask_structor tomm_structis a can of worms. With cgroup-bpf we went through painful bugs with lifetime of cgroup vs bpf, dying cgroups, wq deadlock, etc. All these problems are behind us. Withst_opsinmm_structit will be more painful. I'd rather not go that route.That's valuable information, thanks. I would have hoped that per-MM policies would be easier.
The per-MM approach has a performance advantage over per-MEMCG policies. This is because it accesses the policy hook directly via
vma->vm_mm->bpf_mm->policy_hook()whereas the per-MEMCG method requires a more expensive lookup:
memcg = get_mem_cgroup_from_mm(vma->vm_mm); memcg->bpf_memcg->policy_hook();Quoted 1 line
This lookup could be a concern in a critical path. However, this
Quoted 8 lines
performance issue in the per-MEMCG mode can be mitigated. For instance, when a task is added to a new memcg, we can cache the hook pointer:
task->mm->bpf_mm->policy_hook = memcg->bpf_memcg->policy_hookUltimately, we might still introduce a mm_struct:bpf_mm field to provide an efficient interface.
Right, caching is what I would have proposed. I would expect some headakes with lifetime, but probably nothing unsolvable.
Quoted 7 lines
Quoted 2 lines
Sounds like cgroup-bpf has sorted
out most of the mess.No, the attach-based cgroup-bpf has proven to be ... a "can of worms" in practice ...
(I welcome corrections from the BPF maintainers if my assessment is
inaccurate.)I don't know what's right or wrong here, as Alexei said the "
mm_struct" based one would be a can of worms and that the the cgroup-based one apparently solved these issues ("All these problems are behind us."), that's why I asked for some clarifications. :)[...]
Quoted 8 lines
Quoted 8 lines
Some of what Yafang might want to achieve could maybe at this point be maybe achieved through the
prctl(PR_SET_THP_DISABLE)support, including extensions we recently added [1].Systemd support still seems to be in the works [2] for some of that.
Quoted 8 lines
Quoted 2 lines
[1] https://lwn.net/Articles/1032014/
[2] https://github.com/systemd/systemd/pull/39085Thank you for sharing this.
However, BPF-THP is already deployed across our server fleet and both our users and my boss are satisfied with it. As such, we are not considering a switch. The current solution also offers us a valuable opportunity to experiment with additional policies in production.Just to emphasize: we usually don't add two mechanisms to achieve the very same end goal. There really must be something delivering more value for us to accept something more complex. Focusing on solving a solved problem is not good.
Quoted 4 lines
If some company went with a downstream-only approach they might be stuck having to maintain that forever.
That's why other companies prefer upstream-first :)
The upstream kernel process is often too slow for our users' needs and frequently results in the rejection of our submissions. Therefore, we maintain a set of local features that, despite being rejected upstream, are critical for delivering user benefits.
Quoted 7 lines
Having that said, the original reason why I agreed that having bpf for THP can be valuable is that I see a lot more value for rapid prototyping and policies once you can actually control on a per-VMA basis (using vma size, flags, anon-vma names etc) where specific folio orders could be valuable, and where not.
agreed.
Quoted 2 lines
But also, possibly where we would want to waste
memory and where not.
This is a challenge we have also encountered since enabling THP for production services. We are continuing to develop our BPF-THP system to make it more automated.
Quoted 9 lines
As we are speaking I have a customer running into issues [1] with virtio-balloon discarding pages in a VM and khugepaged undoing part of that work in the hypervisor. The workaround of telling khugepaged to not waste memory in all of the system really feels suboptimal when we know that it's only the VM memory of such VMs (with balloon deflation enabled) where we would not want to waste memory but still use THPs.
This is an excellent analysis—thank you for sharing it.
I don't have a better solution than your current approach of setting max_ptes_none to 0. However, I believe this situation serves as a compelling example for why we should implement a per-process control for /sys/kernel/mm/transparent_hugepage/ parameters, such as khugepaged/max_ptes_none. This direction also aligns perfectly with our roadmap for evolving the BPF-THP system on our production servers.
Signature · 2 lines
Regards Yafang
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
In reply to Alexei StarovoitovQuoted 36 lines · Alexei Starovoitov
Quoted 18 lines · Yafang Shao
The per-process BPF-THP mode is unsuitable for managing shared resources
such as shmem THP and file-backed THP. This aligns with known cgroup
limitations for similar scenarios [0].Introduce a global BPF-THP mode to address this gap. When registered:
- All existing per-process instances are disabled
- New per-process registrations are blocked
- Existing per-process instances remain registered (no forced unregistration)The global mode takes precedence over per-process instances. Updates are
type-isolated: global instances can only be updated by new global instances, and per-process instances by new per-process instances....
Quoted 4 lines
spin_lock(&thp_ops_lock);
- /* Each process is exclusively managed by a single BPF-THP. */
- if (rcu_access_pointer(mm->bpf_mm.bpf_thp)) {Quoted 8 lines
+ /* Each process is exclusively managed by a single BPF-THP. + * Global mode disables per-process instances.
+ */
+ if (rcu_access_pointer(mm->bpf_mm.bpf_thp)||rcu_access_pointer(bpf_thp_global)) {
err =-EBUSY;
goto out;
}
Quoted 7 lines
You didn't address the issue and instead doubled down on this broken global approach.
This bait-and-switch patchset is frankly disingenuous. 'lets code up some per-mm hack, since people will hate it anyway, and I'm not going to use it either, and add this global mode as a fake "fallback"...'
Should things be the other way around, where
per-process BPF THP policy overrides global
policy?
I can definitely see a use for global policy,
but also a reason to override it for some
programs or containers.
Signature · 1 line
All Rights Reversed.
Re: [PATCH v12 mm-new 06/10] mm: bpf-thp: add support for global mode
Quoted 41 lines · Rik van Riel
Quoted 39 lines · Alexei Starovoitov
Quoted 18 lines · Yafang Shao
The per-process BPF-THP mode is unsuitable for managing shared resources
such as shmem THP and file-backed THP. This aligns with known cgroup
limitations for similar scenarios [0].Introduce a global BPF-THP mode to address this gap. When registered:
- All existing per-process instances are disabled
- New per-process registrations are blocked
- Existing per-process instances remain registered (no forced unregistration)The global mode takes precedence over per-process instances. Updates are
type-isolated: global instances can only be updated by new global instances, and per-process instances by new per-process instances....
Quoted 8 lines
spin_lock(&thp_ops_lock);- /* Each process is exclusively managed by a single BPF-THP.
*/mm/huge_memory_bpf.c +4
-1-if (rcu_access_pointer (mm->bpf_mm.bpf_thp)) { +/* Each process is exclusively managed by a single BPF-THP. +* Global mode disables per-process instances. +*/ + if (rcu_access_pointer(mm->bpf_mm.bpf_thp) ||Quoted 4 lines
rcu_access_pointer(bpf_thp_global)) { err = -EBUSY; goto out; }You didn't address the issue and instead doubled down on this broken global approach.
Quoted 8 lines
Quoted 4 lines
This bait-and-switch patchset is frankly disingenuous. 'lets code up some per-mm hack, since people will hate it anyway, and I'm not going to use it either, and add this global mode as a fake "fallback"...'
Should things be the other way around, where
per-process BPF THP policy overrides global
policy?
Makes sense
Quoted 4 lines
I can definitely see a use for global policy,
but also a reason to override it for some
programs or containers.
We have deployed BPF-THP across nearly all of our fleets for over six months and have enabled THP for dozens of our services.
Based on our practical experience, the global mode has proven highly useful as it establishes a default policy for all services. When a specific THP policy is required for a particular service, we implement it using dedicated BPF maps—such as thp-always, thp-madvise, thp-never, or other custom policy maps.
That said, I also find value in combining a default global policy with the ability to override it for certain processes or containers.
The global mode and the per-process/cgroup mode are not mutually exclusive; they can coexist.
Through our use of BPF-THP, we have found that the most reliable approach is to allocate all THP at the initial stage. If a service dynamically allocates a large number of THP during runtime, it can easily trigger compaction stalls—even on the latest upstream kernel. Therefore, monitoring compaction stalls and memory pressure is essential to determine when a service should stop allocating additional THP.
--
Regards
Yafang
PATCH v12 7/10 Documentation: add BPF THP
In reply to Yafang ShaoAdd the documentation.
--- Documentation/admin-guide/mm/transhuge.rst | 113 +++++++++++++++++++++ mm/Kconfig | 2 + 2 files changed, 115 insertions(+)
Documentation/admin-guide/mm/transhuge.rst +113 -0
@@ -776,3 +776,116 @@ support enabled just fine as always. No difference can be noted in hugetlbfs other than there will be less overall fragmentation. All usual features belonging to hugetlbfs are preserved and unaffected. libhugetlbfs will also work fine as usual. + +BPF THP +======= + +:Author: Yafang Shao <[email protected]> +:Date: October 2025 + +Overview +-------- + +When the system is configured with "always" or "madvise" THP mode, a BPF program +can be used to adjust THP allocation policies dynamically. This enables +fine-grained control over THP decisions based on various factors including +workload identity, allocation context, and system memory pressure. + +Program Interface +----------------- + +This feature implements a struct_ops BPF program with the following interface:: + + struct bpf_thp_ops { + pid_t pid; + thp_order_fn_t *thp_get_order; + }; + +Callback Functions +------------------ + +thp_get_order() +~~~~~~~~~~~~~~~ + +.. code-block:: c + + int thp_get_order(struct vm_area_struct *vma, + enum tva_type type, + unsigned long orders); + +Parameters +^^^^^^^^^^ + +``vma`` + ``vm_area_struct`` associated with the THP allocation. + +``type`` + TVA type for the current ``vma``. + +``orders`` + Bitmask of available THP orders for this allocation. + +Return value +^^^^^^^^^^^^ + +- The suggested THP order for allocation from the BPF program +- Must be a valid, available order from the provided ``orders`` bitmask + +Operation Modes +--------------- + +Per Process Mode +~~~~~~~~~~~~~~~~ + +When registering a BPF-THP with a specific PID, the program is installed in the +target task's ``mm_struct``:: + + struct mm_struct { + struct bpf_thp_ops __rcu *bpf_thp; + }; + +Inheritance Behavior +^^^^^^^^^^^^^^^^^^^^ + +- Existing child processes are unaffected +- Newly forked children inherit the BPF-THP from their parent +- The BPF-THP persists across execve() calls + +Management Rules +^^^^^^^^^^^^^^^^ + +- When a BPF-THP instance is unregistered, all managed tasks' ``bpf_thp`` + pointers are reset to ``NULL`` +- When a BPF-THP instance is updated, all managed tasks' ``bpf_thp`` pointers + are automatically updated to the new version +- Each process can be managed by only one BPF-THP instance at a time + +Global Mode +~~~~~~~~~~~ + +If no PID is specified during registration, the BPF-THP operates in global mode. +In this mode, all tasks in the system are managed by the global instance. + +Global Mode Precedence +^^^^^^^^^^^^^^^^^^^^^^ + +- The global instance takes precedence over all per-process instances +- All existing per-process instances are disabled when a global instance is + registered +- New per-process registrations are blocked while a global instance is active +- Existing per-process instances remain registered (no forced unregistration) + +Instance Management +^^^^^^^^^^^^^^^^^^^ + +- Updates are type-isolated: global instances can only be updated by new global + instances, and per-process instances by new per-process instances +- Only one global BPF-THP can be registered at a time +- Global instances can be updated dynamically without requiring task restarts + +Implementation Notes +-------------------- + +- This is currently an experimental feature +- ``CONFIG_BPF_THP`` must be enabled to use this functionality +- The feature depends on proper THP configuration ("always" or "madvise" mode)
mm/Kconfig +2 -0
@@ -1476,6 +1476,8 @@ config BPF_THP Enable dynamic THP policy adjustment using BPF programs. This feature is currently experimental. + See Documentation/admin-guide/mm/transhuge.rst for more information. + WARNING: This feature is unstable and may change in future kernel endif # BPF_MM
Signature · 1 line
2.47.3
PATCH v12 8/10 selftests/bpf: add a simple BPF based THP policy
In reply to Yafang ShaoThis test case implements a basic THP policy that sets THPeligible to 0 for a specific task. I selected THPeligible for verification because its straightforward nature makes it ideal for validating the BPF THP policy functionality.
Below configs must be enabled for this test:
CONFIG_BPF_MM=y
CONFIG_BPF_THP=y
CONFIG_TRANSPARENT_HUGEPAGE=y
--- MAINTAINERS | 2 + tools/testing/selftests/bpf/config | 3 + .../selftests/bpf/prog_tests/thp_adjust.c | 245 ++++++++++++++++++ .../selftests/bpf/progs/test_thp_adjust.c | 24 ++ 4 files changed, 274 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
MAINTAINERS +2 -0
@@ -16524,6 +16524,8 @@ F: mm/huge_memory.c F: mm/huge_memory_bpf.c F: mm/khugepaged.c F: mm/mm_slot.h +F: tools/testing/selftests/bpf/prog_tests/thp_adjust.c +F: tools/testing/selftests/bpf/progs/test_thp_adjust* F: tools/testing/selftests/mm/khugepaged.c F: tools/testing/selftests/mm/split_huge_page_test.c F: tools/testing/selftests/mm/transhuge-stress.c
tools/testing/selftests/bpf/config +3 -0
@@ -7,8 +7,10 @@ CONFIG_BPF_JIT=y CONFIG_BPF_KPROBE_OVERRIDE=y CONFIG_BPF_LIRC_MODE2=y CONFIG_BPF_LSM=y +CONFIG_BPF_MM=y CONFIG_BPF_STREAM_PARSER=y CONFIG_BPF_SYSCALL=y +CONFIG_BPF_THP=y # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set CONFIG_CGROUP_BPF=y CONFIG_CRYPTO_HMAC=y
@@ -115,6 +117,7 @@ CONFIG_SECURITY=y CONFIG_SECURITYFS=y CONFIG_SYN_COOKIES=y CONFIG_TEST_BPF=m +CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_UDMABUF=y CONFIG_USERFAULTFD=y CONFIG_VSOCKETS=y
tools/testing/selftests/bpf/prog_tests/thp_adjust.c new file +245 -0
245 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_thp_adjust.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +char _license[] SEC("license") = "GPL"; + +int pmd_order; + +SEC("struct_ops/thp_get_order") +int BPF_PROG (thp_not_eligible ,struct vm_area_struct *vma,enum tva_type type, +unsigned long orders) +{ +/* THPeligible in /proc/pid/smaps is 0 */ +if (type ==TVA_SMAPS ) +return 0 ; +return pmd_order; +} + +SEC(".struct_ops.link") +struct bpf_thp_ops thp_eligible_ops = { + .thp_get_order = (void *)thp_not_eligible, +};
Signature · 1 line
2.47.3
PATCH v12 9/10 selftests/bpf: add test case to update THP policy
In reply to Yafang ShaoThis test case exercises the BPF THP update mechanism by modifying an existing policy. The behavior confirms that:
- EBUSY error occurs when attempting to install a BPF program on a process that already has an active BPF program
- Updates to currently running programs are successfully processed
- Local prog can't be updated by a global prog
- Global prog can't be updated by a local prog
- Global prog can be attached even if there's a local prog
- Local prog can't be attached if there's a global prog
--- .../selftests/bpf/prog_tests/thp_adjust.c | 79 +++++++++++++++++++ .../selftests/bpf/progs/test_thp_adjust.c | 29 +++++++ 2 files changed, 108 insertions(+)
tools/testing/selftests/bpf/prog_tests/thp_adjust.c +79 -0
@@ -194,6 +194,79 @@ static void subtest_thp_eligible(void)bpf_link__destroy (ops_link); } +static void subtest_thp_policy_update (void ) +{ +struct bpf_link *old_link, *new_link; +int elighble, err, pid; +char *ptr; + + pid =getpid (); + ptr =thp_alloc (); + + old_link =bpf_map__attach_struct_ops (skel->maps.thp_eligible_ops); +if (!ASSERT_OK_PTR (old_link,"attach_old_link" )) + goto free; + + elighble =get_thp_eligible (pid, (unsigned long )ptr); +ASSERT_EQ (elighble,0 ,"THPeligible" ); + +/* Attach multi BPF-THP to a single process is rejected. */ + new_link =bpf_map__attach_struct_ops (skel->maps.thp_eligible_ops2); +if (!ASSERT_NULL (new_link,"attach_new_link" )) + goto destory_old; +ASSERT_EQ (errno,EBUSY ,"attach_new_link" ); + + elighble =get_thp_eligible (pid, (unsigned long )ptr); +ASSERT_EQ (elighble,0 ,"THPeligible" ); + + err =bpf_link__update_map (old_link, skel->maps.thp_eligible_ops2); +ASSERT_EQ (err,0 ,"update_old_link" ); + + elighble =get_thp_eligible (pid, (unsigned long )ptr); +ASSERT_EQ (elighble,1 ,"THPeligible" ); + +/* Per process prog can't be update by a global prog */ + err =bpf_link__update_map (old_link, skel->maps.swap_ops); +ASSERT_EQ (err, -EINVAL ,"update_old_link" ); + +destory_old: +bpf_link__destroy (old_link); +free: +thp_free (ptr); +} + +static void subtest_thp_global_policy (void ) +{ +struct bpf_link *local_link, *global_link; +int err; + + local_link =bpf_map__attach_struct_ops (skel->maps.thp_eligible_ops); +if (!ASSERT_OK_PTR (local_link,"attach_local_link" )) +return ; + +/* global prog can be attached even if there is a local prog */ + global_link =bpf_map__attach_struct_ops (skel->maps.swap_ops); +if (!ASSERT_OK_PTR (global_link,"attach_global_link" )) { +bpf_link__destroy (local_link); +return ; + } + +bpf_link__destroy (local_link); + +/* local prog can't be attaached if there is a global prog */ + local_link =bpf_map__attach_struct_ops (skel->maps.thp_eligible_ops); +if (!ASSERT_NULL (local_link,"attach_new_link" )) + goto destory_global; +ASSERT_EQ (errno,EBUSY ,"attach_new_link" ); + +/* global prog can't be updated by a local prog */ + err =bpf_link__update_map (global_link, skel->maps.thp_eligible_ops); +ASSERT_EQ (err, -EINVAL ,"update_old_link" ); + +destory_global: +bpf_link__destroy (global_link); +} +static int thp_adjust_setup (void ) {int err =-1 , pmd_order;
@@ -214,6 +287,8 @@ static int thp_adjust_setup(void) skel->bss->pmd_order = pmd_order; skel->struct_ops.thp_eligible_ops->pid =getpid (); + skel->struct_ops.thp_eligible_ops2->pid =getpid (); +/* swap_ops is a global prog since its pid is not set. */ err =test_thp_adjust__load (skel);if (!ASSERT_OK (err,"load" ))
@@ -240,6 +315,10 @@ void test_thp_adjust(void)if (test__start_subtest ("thp_eligible" ))subtest_thp_eligible (); +if (test__start_subtest ("policy_update" )) +subtest_thp_policy_update (); +if (test__start_subtest ("global_policy" )) +subtest_thp_global_policy ();thp_adjust_destroy (); }
tools/testing/selftests/bpf/progs/test_thp_adjust.c +29 -0
@@ -22,3 +22,32 @@ SEC(".struct_ops.link")struct bpf_thp_ops thp_eligible_ops = { .thp_get_order = (void *)thp_not_eligible, }; + +SEC("struct_ops/thp_get_order") +int BPF_PROG (thp_eligible ,struct vm_area_struct *vma,enum tva_type type, +unsigned long orders) +{ +/* THPeligible in /proc/pid/smaps is 1 */ +if (type ==TVA_SMAPS ) +return pmd_order; +return pmd_order; +} + +SEC(".struct_ops.link") +struct bpf_thp_ops thp_eligible_ops2 = { + .thp_get_order = (void *)thp_eligible, +}; + +SEC("struct_ops/thp_get_order") +int BPF_PROG (alloc_not_in_swap ,struct vm_area_struct *vma,enum tva_type type, +unsigned long orders) +{ +if (type ==TVA_SWAP_PAGEFAULT ) +return 0 ; +return -1 ; +} + +SEC(".struct_ops.link") +struct bpf_thp_ops swap_ops = { + .thp_get_order = (void *)alloc_not_in_swap, +};
Signature · 1 line
2.47.3
PATCH v12 10/10 selftests/bpf: add test case for BPF-THP inheritance across fork
In reply to Yafang ShaoVerify that child processes correctly inherit BPF-THP policy from their parent during fork() operations.
--- .../selftests/bpf/prog_tests/thp_adjust.c | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+)
tools/testing/selftests/bpf/prog_tests/thp_adjust.c +33 -0
@@ -267,6 +267,37 @@ static void subtest_thp_global_policy(void)bpf_link__destroy (global_link); } +static void subtest_thp_fork (void ) +{ +int elighble, child, pid, status; +struct bpf_link *ops_link; +char *ptr; + + ops_link =bpf_map__attach_struct_ops (skel->maps.thp_eligible_ops); +if (!ASSERT_OK_PTR (ops_link,"attach struct_ops" )) +return ; + + child =fork (); +if (!ASSERT_GE (child,0 ,"fork" )) + goto destroy; + +if (child ==0 ) { + ptr =thp_alloc (); + elighble =get_thp_eligible (getpid (), (unsigned long )ptr); +ASSERT_EQ (elighble,0 ,"THPeligible" ); +thp_free (ptr); + +exit (EXIT_SUCCESS ); + } + + pid =waitpid (child, &status,0 ); +ASSERT_EQ (pid, child,"waitpid" ); + +destroy: +bpf_link__destroy (ops_link); + +} +static int thp_adjust_setup (void ) {int err =-1 , pmd_order;
@@ -319,6 +350,8 @@ void test_thp_adjust(void)subtest_thp_policy_update ();if (test__start_subtest ("global_policy" ))subtest_thp_global_policy (); +if (test__start_subtest ("thp_fork" )) +subtest_thp_fork ();thp_adjust_destroy (); }
Signature · 1 line
2.47.3