mm, bpf: BPF-MM, BPF-THP
Subsystems (16)
v10 cover mm, bpf: BPF-MM, BPF-THP
History
=======
RFC v1: fmod_ret based BPF-THP hook
https://lore.kernel.org/linux-mm/[email protected]/
RFC v2: struct_ops based BPF-THP hook
https://lore.kernel.org/linux-mm/[email protected]/
RFC v4: Get THP order with interface get_suggested_order()
https://lore.kernel.org/linux-mm/[email protected]/
v4->v9: Simplify the interface to:
unsigned long
bpf_hook_thp_get_orders(struct vm_area_struct *vma, enum tva_type type,
unsigned long orders);
https://lore.kernel.org/linux-mm/[email protected]/v9->RFC v10: Scope BPF-THP to individual processes
The Design
==========
Scoping BPF-THP to cgroup is rejected
-------------------------------------
As explained by Gutierrez:
1. It breaks the cgroup hierarchy when 2 siblings have different THP policies
2. Cgroup was designed for resource management not for grouping processes and tune those processes
3. We set a precedent for other people adding new flags to cgroup and potentially polluting cgroups. We may end up with cgroups having tens of different flags, making sysadmin's job more complex
The related links are:
https://lore.kernel.org/linux-mm/[email protected]/
https://lore.kernel.org/linux-mm/[email protected]/
So we has to scope it to process.
Scoping BPF-THP to process
--------------------------
To eliminate potential conflicts among competing BPF-THP instances, we enforce that each process is exclusively managed by a single BPF-THP. This approach has received agreement from David. For context, see:
https://lore.kernel.org/linux-mm/[email protected]/
When registering a BPF-THP, we specify the PID of a target task. The BPF-THP is then installed in the task's mm_struct
struct mm_struct {
struct bpf_thp_ops __rcu *thp_thp;
};Inheritance Behavior:
- Existing child processes are unaffected
- Newly forked children inherit the BPF-THP from their parent
- The BPF-THP persists across execve() calls
A new linked list tracks all tasks managed by each BPF-THP instance:
- Newly managed tasks are added to the list
- Exiting tasks are automatically removed from the list
- During BPF-THP unregistration (e.g., when the BPF link is removed), all managed tasks have their bpf_thp pointer set to NULL
- BPF-THP instances can be dynamically updated, with all tracked tasks automatically migrating to the new version.
This design simplifies BPF-THP management in production environments by providing clear lifecycle management and preventing conflicts between multiple BPF-THP instances.
Any feedback is welcomed.
Future Work
===========
Introduce a global fallback mechanism to address shared resource management limitations in process and cgroup-based methods:
https://lore.kernel.org/linux-mm/[email protected]/
Yafang Shao (9):
mm: thp: remove vm_flags parameter from khugepaged_enter_vma()
mm: thp: remove vm_flags parameter from thp_vma_allowable_order()
mm: thp: add support for BPF based THP order selection
mm: thp: decouple THP allocation between swap and page fault paths
mm: thp: enable THP allocation exclusively through khugepaged
bpf: mark mm->owner as __safe_rcu_or_null
bpf: mark vma->vm_mm as __safe_trusted_or_null selftests/bpf: add a simple BPF based THP policy
Documentation: add BPF-based THP policy management
Documentation/admin-guide/mm/transhuge.rst | 39 +++ MAINTAINERS | 3 + fs/exec.c | 1 + fs/proc/task_mmu.c | 3 +- include/linux/huge_mm.h | 59 +++- include/linux/khugepaged.h | 10 +- include/linux/mm_types.h | 18 ++ kernel/bpf/verifier.c | 8 + kernel/fork.c | 1 + mm/Kconfig | 22 ++ mm/Makefile | 1 + mm/huge_memory.c | 7 +- mm/huge_memory_bpf.c | 306 ++++++++++++++++++ mm/khugepaged.c | 35 +- mm/madvise.c | 7 + mm/memory.c | 22 +- mm/mmap.c | 1 + mm/shmem.c | 2 +- mm/vma.c | 6 +- tools/testing/selftests/bpf/config | 3 + .../selftests/bpf/prog_tests/thp_adjust.c | 245 ++++++++++++++ tools/testing/selftests/bpf/progs/lsm.c | 8 +- .../selftests/bpf/progs/test_thp_adjust.c | 23 ++ 23 files changed, 777 insertions(+), 53 deletions(-) create mode 100644 mm/huge_memory_bpf.c create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
Signature · 1 line
2.47.3
RFC PATCH v10 1/9 mm: thp: remove vm_flags parameter from khugepaged_enter_vma()
The khugepaged_enter_vma() function requires handling in two specific scenarios:
1. New VMA creation
When a new VMA is created (for anon vma, it is deferred to pagefault), if vma->vm_mm is not present in khugepaged_mm_slot, it must be added. In this case, khugepaged_enter_vma() is called after vma->vm_flags have been set, allowing direct use of the VMA's flags.
2. VMA flag modification
When vma->vm_flags are modified (particularly when VM_HUGEPAGE is set), the system must recheck whether to add vma->vm_mm to khugepaged_mm_slot. Currently, khugepaged_enter_vma() is called before the flag update, so the call must be relocated to occur after vma->vm_flags have been set.
In the VMA merging path, khugepaged_enter_vma() is also called. For this case, since VMA merging only occurs when the vm_flags of both VMAs are identical (excluding special flags like VM_SOFTDIRTY), we can safely use target->vm_flags instead. (It is worth noting that khugepaged_enter_vma() can be removed from the VMA merging path because the VMA has already been added in the two aforementioned cases. We will address this cleanup in a separate patch.)
After this change, we can further remove vm_flags parameter from thp_vma_allowable_order(). That will be handled in a followup patch.
--- include/linux/khugepaged.h | 10 ++++++---- mm/huge_memory.c | 2 +- mm/khugepaged.c | 27 ++++++++++++++------------- mm/madvise.c | 7 +++++++ mm/vma.c | 6 +++--- 5 files changed, 31 insertions(+), 21 deletions(-)
include/linux/khugepaged.h +6 -4
@@ -13,8 +13,8 @@ extern void khugepaged_destroy(void);extern int start_stop_khugepaged (void );extern void __khugepaged_enter (struct mm_struct *mm);extern void __khugepaged_exit (struct mm_struct *mm); -extern void khugepaged_enter_vma (struct vm_area_struct *vma, -vm_flags_t vm_flags); +extern void khugepaged_enter_vma (struct vm_area_struct *vma); +extern void khugepaged_enter_mm (struct mm_struct *mm);extern void khugepaged_min_free_kbytes_update (void );extern bool current_is_khugepaged (void ); extern int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
@@ -38,8 +38,10 @@ static inline void khugepaged_fork(struct mm_struct *mm, struct mm_struct *oldmmstatic inline void khugepaged_exit (struct mm_struct *mm) { } -static inline void khugepaged_enter_vma (struct vm_area_struct *vma, -vm_flags_t vm_flags) +static inline void khugepaged_enter_vma (struct vm_area_struct *vma) +{ +} +static inline void khugepaged_enter_mm (struct mm_struct *mm) { } static inline int collapse_pte_mapped_thp(struct mm_struct *mm,
mm/huge_memory.c +1 -1
@@ -1346,7 +1346,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) ret =vmf_anon_prepare (vmf);if (ret)return ret; -khugepaged_enter_vma (vma, vma->vm_flags); +khugepaged_enter_vma (vma); if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm) &&
mm/khugepaged.c +14 -13
@@ -353,12 +353,6 @@ int hugepage_madvise(struct vm_area_struct *vma,#endif *vm_flags &= ~VM_NOHUGEPAGE ; *vm_flags |=VM_HUGEPAGE ; -/* -* If the vma become good for khugepaged to scan, -* register it here without waiting a page fault that -* may not happen any time soon. -*/ -khugepaged_enter_vma (vma, *vm_flags);break ;case MADV_NOHUGEPAGE : *vm_flags &= ~VM_HUGEPAGE ;
@@ -460,14 +454,21 @@ void __khugepaged_enter(struct mm_struct *mm)wake_up_interruptible (&khugepaged_wait); } -void khugepaged_enter_vma (struct vm_area_struct *vma, -vm_flags_t vm_flags) +void khugepaged_enter_mm (struct mm_struct *mm) { -if (!mm_flags_test (MMF_VM_HUGEPAGE , vma->vm_mm) && -hugepage_pmd_enabled ()) { -if (thp_vma_allowable_order (vma, vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) -__khugepaged_enter (vma->vm_mm); - } +if (mm_flags_test (MMF_VM_HUGEPAGE , mm)) +return ; +if (!hugepage_pmd_enabled ()) +return ; + +__khugepaged_enter (mm); +} + +void khugepaged_enter_vma (struct vm_area_struct *vma) +{ +if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) +return ; +khugepaged_enter_mm (vma->vm_mm); }void __khugepaged_exit (struct mm_struct *mm)
mm/madvise.c +7 -0
@@ -1425,6 +1425,13 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior)VM_WARN_ON_ONCE (madv_behavior->lock_mode !=MADVISE_MMAP_WRITE_LOCK ); error =madvise_update_vma (new_flags, madv_behavior); +/* +* If the vma become good for khugepaged to scan, +* register it here without waiting a page fault that +* may not happen any time soon. +*/ +if (!error && new_flags &VM_HUGEPAGE ) +khugepaged_enter_mm (vma->vm_mm); out:/* * madvise() returns EAGAIN if kernel resources, such as
mm/vma.c +3 -3
@@ -973,7 +973,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(if (err ||commit_merge (vmg)) goto abort; -khugepaged_enter_vma (vmg->target, vmg->vm_flags); +khugepaged_enter_vma (vmg->target); vmg->state =VMA_MERGE_SUCCESS ;return vmg->target;
@@ -1093,7 +1093,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)* following VMA if we have VMAs on both sides. */ if (vmg->target && !vma_expand (vmg)) { -khugepaged_enter_vma (vmg->target, vmg->vm_flags); +khugepaged_enter_vma (vmg->target); vmg->state =VMA_MERGE_SUCCESS ;return vmg->target; }
@@ -2520,7 +2520,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap)* call covers the non-merge case. */ if (!vma_is_anonymous (vma)) -khugepaged_enter_vma (vma, map->vm_flags); +khugepaged_enter_vma (vma); *vmap = vma;return 0 ;
Signature · 1 line
2.47.3
RFC PATCH v10 2/9 mm: thp: remove vm_flags parameter from thp_vma_allowable_order()
In reply to Yafang ShaoBecause all calls to thp_vma_allowable_order() pass vma->vm_flags as the vma_flags argument, we can remove the parameter and have the function access vma->vm_flags directly.
--- fs/proc/task_mmu.c | 3 +-- include/linux/huge_mm.h | 16 ++++++++-------- mm/huge_memory.c | 4 ++-- mm/khugepaged.c | 10 +++++----- mm/memory.c | 11 +++++------ mm/shmem.c | 2 +- 6 files changed, 22 insertions(+), 24 deletions(-)
fs/proc/task_mmu.c +1 -2
@@ -1369,8 +1369,7 @@ static int show_smap(struct seq_file *m, void *v)__show_smap (m, &mss, false);seq_printf (m,"THPeligible: %8u\n" , - !!thp_vma_allowable_orders (vma, vma->vm_flags,TVA_SMAPS , -THP_ORDERS_ALL )); + !!thp_vma_allowable_orders (vma,TVA_SMAPS ,THP_ORDERS_ALL ));if (arch_pkeys_enabled ())seq_printf (m,"ProtectionKey: %8u\n" ,vma_pkey (vma));
include/linux/huge_mm.h +8 -8
@@ -101,8 +101,8 @@ enum tva_type { TVA_FORCED_COLLAPSE, /* Forced collapse (e.g. MADV_COLLAPSE). */ }; -#define thp_vma_allowable_order (vma, vm_flags, type, order) \ - (!!thp_vma_allowable_orders(vma, vm_flags, type, BIT(order))) +#define thp_vma_allowable_order (vma, type, order) \ + (!!thp_vma_allowable_orders(vma, type, BIT(order)))#define split_folio (f) split_folio_to_list(f, NULL)
@@ -266,14 +266,12 @@ static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma, }unsigned long __thp_vma_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type type,unsigned long orders);/** * thp_vma_allowable_orders - determine hugepage orders that are allowed for vma * @vma: the vm area to check -* @vm_flags: use these vm_flags instead of vma->vm_flags * @type: TVA type * @orders: bitfield of all orders to consider *
@@ -287,10 +285,11 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,*/ static inline unsigned long thp_vma_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type type,unsigned long orders) { +vm_flags_t vm_flags = vma->vm_flags; +/* * Optimization to check if required orders are enabled early. Only * forced collapse ignores sysfs configs.
@@ -309,7 +308,7 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,return 0 ; } -return __thp_vma_allowable_orders (vma, vm_flags, type, orders); +return __thp_vma_allowable_orders (vma, type, orders); }struct thpsize {
@@ -329,8 +328,10 @@ struct thpsize {* through madvise or prctl. */ static inline bool vma_thp_disabled (struct vm_area_struct *vma, -vm_flags_t vm_flags,bool forced_collapse) +bool forced_collapse) { +vm_flags_t vm_flags = vma->vm_flags; +/* Are THPs disabled for this VMA? */ if (vm_flags &VM_NOHUGEPAGE )return true;
@@ -560,7 +561,6 @@ static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma, }static inline unsigned long thp_vma_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type type,unsigned long orders) {
mm/huge_memory.c +2 -2
@@ -98,7 +98,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) }unsigned long __thp_vma_allowable_orders (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type type,unsigned long orders) {
@@ -106,6 +105,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,const bool in_pf = type ==TVA_PAGEFAULT ;const bool forced_collapse = type ==TVA_FORCED_COLLAPSE ;unsigned long supported_orders; +vm_flags_t vm_flags = vma->vm_flags;/* Check the intersection of requested and supported orders. */ if (vma_is_anonymous (vma))
@@ -122,7 +122,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,if (!vma->vm_mm)/* vdso */ return 0 ; -if (thp_disabled_by_hw () ||vma_thp_disabled (vma, vm_flags, forced_collapse)) +if (thp_disabled_by_hw () ||vma_thp_disabled (vma, forced_collapse))return 0 ;/* khugepaged doesn't collapse DAX vma, but page fault is fine. */
mm/khugepaged.c +5 -5
@@ -466,7 +466,7 @@ void khugepaged_enter_mm(struct mm_struct *mm)void khugepaged_enter_vma (struct vm_area_struct *vma) { -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) +if (!thp_vma_allowable_order (vma,TVA_KHUGEPAGED ,PMD_ORDER ))return ;khugepaged_enter_mm (vma->vm_mm); }
@@ -917,7 +917,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,if (!thp_vma_suitable_order (vma, address,PMD_ORDER ))return SCAN_ADDRESS_RANGE ; -if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_order (vma, type,PMD_ORDER ))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then
@@ -1531,7 +1531,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,* and map it by a PMD, regardless of sysfs THP settings. As such, let's * analogously elide sysfs THP settings here and force collapse. */ -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_FORCED_COLLAPSE ,PMD_ORDER )) +if (!thp_vma_allowable_order (vma,TVA_FORCED_COLLAPSE ,PMD_ORDER ))return SCAN_VMA_CHECK ;/* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */
@@ -2426,7 +2426,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, progress++;break ; } -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) { +if (!thp_vma_allowable_order (vma,TVA_KHUGEPAGED ,PMD_ORDER )) { skip: progress++;continue ;
@@ -2757,7 +2757,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,BUG_ON (vma->vm_start > start);BUG_ON (vma->vm_end < end); -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_FORCED_COLLAPSE ,PMD_ORDER )) +if (!thp_vma_allowable_order (vma,TVA_FORCED_COLLAPSE ,PMD_ORDER ))return -EINVAL ; cc =kmalloc (sizeof (*cc),GFP_KERNEL );
mm/memory.c +5 -6
@@ -4558,7 +4558,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)* Get a list of all the (large) orders below PMD_ORDER that are enabled * and suitable for swapping THP. */ - orders =thp_vma_allowable_orders (vma, vma->vm_flags,TVA_PAGEFAULT , + orders =thp_vma_allowable_orders (vma,TVA_PAGEFAULT ,BIT (PMD_ORDER ) -1 ); orders =thp_vma_suitable_orders (vma, vmf->address, orders); orders = thp_swap_suitable_orders(swp_offset(entry),
@@ -5107,7 +5107,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)* for this vma. Then filter out the orders that can't be allocated over * the faulting address and still be fully contained in the vma. */ - orders =thp_vma_allowable_orders (vma, vma->vm_flags,TVA_PAGEFAULT , + orders =thp_vma_allowable_orders (vma,TVA_PAGEFAULT ,BIT (PMD_ORDER ) -1 ); orders =thp_vma_suitable_orders (vma, vmf->address, orders);
@@ -5379,7 +5379,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *pa* PMD mappings if THPs are disabled. As we already have a THP, * behave as if we are forcing a collapse. */ -if (thp_disabled_by_hw () ||vma_thp_disabled (vma, vma->vm_flags, +if (thp_disabled_by_hw () ||vma_thp_disabled (vma,/* forced_collapse=*/ true))return ret;
@@ -6280,7 +6280,6 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, .gfp_mask = __get_fault_gfp_mask(vma), };struct mm_struct *mm = vma->vm_mm; -vm_flags_t vm_flags = vma->vm_flags;pgd_t *pgd;p4d_t *p4d;vm_fault_t ret;
@@ -6295,7 +6294,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,return VM_FAULT_OOM ; retry_pud:if (pud_none (*vmf.pud) && -thp_vma_allowable_order (vma, vm_flags,TVA_PAGEFAULT ,PUD_ORDER )) { +thp_vma_allowable_order (vma,TVA_PAGEFAULT ,PUD_ORDER )) { ret =create_huge_pud (&vmf);if (!(ret &VM_FAULT_FALLBACK ))return ret;
@@ -6329,7 +6328,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, goto retry_pud;if (pmd_none (*vmf.pmd) && -thp_vma_allowable_order (vma, vm_flags,TVA_PAGEFAULT ,PMD_ORDER )) { +thp_vma_allowable_order (vma,TVA_PAGEFAULT ,PMD_ORDER )) { ret =create_huge_pmd (&vmf);if (!(ret &VM_FAULT_FALLBACK ))return ret;
mm/shmem.c +1 -1
@@ -1780,7 +1780,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,vm_flags_t vm_flags = vma ? vma->vm_flags :0 ;unsigned int global_orders; -if (thp_disabled_by_hw () || (vma &&vma_thp_disabled (vma, vm_flags, shmem_huge_force))) +if (thp_disabled_by_hw () || (vma &&vma_thp_disabled (vma, shmem_huge_force)))return 0 ; global_orders = shmem_huge_global_enabled(inode, index, write_end,
Signature · 1 line
2.47.3
RFC PATCH v10 3/9 mm: thp: add support for BPF based THP order selection
In reply to Yafang ShaoThis patch introduces a new BPF struct_ops called bpf_thp_ops for dynamic THP tuning. It includes a hook bpf_hook_thp_get_order(), allowing BPF programs to influence THP order selection based on factors such as:
- Workload identity
For example, workloads running in specific containers or cgroups.
- Allocation context
Whether the allocation occurs during a page fault, khugepaged, swap or other paths.
- VMA's memory advice settings
MADV_HUGEPAGE or MADV_NOHUGEPAGE
- Memory pressure
PSI system data or associated cgroup PSI metrics
The kernel API of this new BPF hook is as follows,
/**
* thp_order_fn_t: Get the suggested THP order from a BPF program for allocation
* @vma: vm_area_struct associated with the THP allocation
* @type: TVA type for current @vma
* @orders: Bitmask of available THP orders for this allocation
*
* Return: The suggested THP order for allocation from the BPF program. Must be
* a valid, available order.
*/
typedef int thp_order_fn_t(struct vm_area_struct *vma,
enum tva_type type, unsigned long orders);
Only a single BPF program can be attached at any given time, though it can be dynamically updated to adjust the policy. The implementation supports anonymous THP, shmem THP, and mTHP, with future extensions planned for file-backed THP.
This functionality is only active when system-wide THP is configured to madvise or always mode. It remains disabled in never mode. Additionally, if THP is explicitly disabled for a specific task via prctl(), this BPF functionality will also be unavailable for that task.
This BPF hook enables the implementation of flexible THP allocation policies at the system, per-cgroup, or per-task level.
This feature requires CONFIG_BPF_THP (EXPERIMENTAL) to be enabled. Note that this capability is currently unstable and may undergo significant changes—including potential removal—in future kernel versions.
--- MAINTAINERS | 1 + fs/exec.c | 1 + include/linux/huge_mm.h | 40 +++++ include/linux/mm_types.h | 18 +++ kernel/fork.c | 1 + mm/Kconfig | 22 +++ mm/Makefile | 1 + mm/huge_memory_bpf.c | 306 +++++++++++++++++++++++++++++++++++++++ mm/mmap.c | 1 + 9 files changed, 391 insertions(+) create mode 100644 mm/huge_memory_bpf.c
MAINTAINERS +1 -0
@@ -16257,6 +16257,7 @@ F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h F: mm/huge_memory.c +F: mm/huge_memory_bpf.c F: mm/khugepaged.c F: mm/mm_slot.h F: tools/testing/selftests/mm/khugepaged.c
fs/exec.c +1 -0
@@ -890,6 +890,7 @@ static int exec_mmap(struct mm_struct *mm)activate_mm (active_mm, mm);if (IS_ENABLED (CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM ))local_irq_enable (); +bpf_thp_retain_mm (mm, old_mm);lru_gen_add_mm (mm);task_unlock (tsk);lru_gen_use_mm (mm);
include/linux/huge_mm.h +40 -0
@@ -269,6 +269,41 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,enum tva_type type, unsigned long orders); +#ifdef CONFIG_BPF_THP + +unsigned long +bpf_hook_thp_get_orders (struct vm_area_struct *vma,enum tva_type type, +unsigned long orders); + +void bpf_thp_exit_mm (struct mm_struct *mm); +void bpf_thp_retain_mm (struct mm_struct *mm,struct mm_struct *old_mm); +void bpf_thp_fork (struct mm_struct *mm,struct mm_struct *old_mm); + +#else + +static inline unsigned long +bpf_hook_thp_get_orders (struct vm_area_struct *vma,enum tva_type type, +unsigned long orders) +{ +return orders; +} + +static inline void bpf_thp_ops_exit (struct mm_struct *mm) +{ +} + +static inline void +bpf_thp_retain_mm (struct mm_struct *mm,struct mm_struct *old_mm) +{ +} + +static inline void +bpf_thp_fork (struct mm_struct *mm,struct mm_struct *old_mm) +{ +} + +#endif +/** * thp_vma_allowable_orders - determine hugepage orders that are allowed for vma * @vma: the vm area to check
@@ -290,6 +325,11 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, {vm_flags_t vm_flags = vma->vm_flags; +/* The BPF-specified order overrides which order is selected. */ + orders &=bpf_hook_thp_get_orders (vma, type, orders); +if (!orders) +return 0 ; +/* * Optimization to check if required orders are enabled early. Only * forced collapse ignores sysfs configs.
include/linux/mm_types.h +18 -0
@@ -33,6 +33,7 @@struct address_space ;struct futex_private_hash ;struct mem_cgroup ; +struct bpf_mm_ops ; typedef struct { unsigned long f;
@@ -976,6 +977,19 @@ struct mm_cid { };#endif +#ifdef CONFIG_BPF_THP +struct bpf_thp_ops ; +#endif + +#ifdef CONFIG_BPF_MM +struct bpf_mm_ops { +#ifdef CONFIG_BPF_THP +struct bpf_thp_ops __rcu *bpf_thp; +struct list_head bpf_thp_list; +#endif +}; +#endif +/* * Opaque type representing current mm_struct flag state. Must be accessed via * mm_flags_xxx() helper functions.
@@ -1268,6 +1282,10 @@ struct mm_struct {#ifdef CONFIG_MM_ID mm_id_t mm_id;#endif /* CONFIG_MM_ID */ + +#ifdef CONFIG_BPF_MM +struct bpf_mm_ops bpf_mm; +#endif } __randomize_layout;/*
kernel/fork.c +1 -0
@@ -1130,6 +1130,7 @@ static inline void __mmput(struct mm_struct *mm)exit_aio (mm);ksm_exit (mm);khugepaged_exit (mm);/* must run before exit_mmap */ +bpf_thp_exit_mm (mm);exit_mmap (mm);mm_put_huge_zero_folio (mm);set_mm_exe_file (mm,NULL );
mm/Kconfig +22 -0
@@ -1371,6 +1371,28 @@ config PT_RECLAIM config FIND_NORMAL_PAGE def_bool n +menuconfig BPF_MM + bool "BPF-based Memory Management (EXPERIMENTAL)" + depends on BPF_SYSCALL + + help + Enable BPF-based Memory Management Policy. This feature is currently + experimental. + + WARNING: This feature is unstable and may change in future kernel + +if BPF_MM +config BPF_THP + bool "BPF-based THP Policy (EXPERIMENTAL)" + depends on TRANSPARENT_HUGEPAGE && BPF_MM + + help + Enable dynamic THP policy adjustment using BPF programs. This feature + is currently experimental. + + WARNING: This feature is unstable and may change in future kernel +endif # BPF_MM + source "mm/damon/Kconfig" endmenu
mm/Makefile +1 -0
@@ -99,6 +99,7 @@ obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_BPF_THP) += huge_memory_bpf.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
mm/huge_memory_bpf.c new file +306 -0
306 lines not on this page · raw message
mm/mmap.c +1 -0
@@ -1844,6 +1844,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)vma_iter_free (&vmi);if (!retval) {mt_set_in_rcu (vmi.mas.tree); +bpf_thp_fork (mm, oldmm);ksm_fork (mm, oldmm);khugepaged_fork (mm, oldmm); }else {
Signature · 1 line
2.47.3
kernel test robot
build botRe: [RFC PATCH v10 mm-new 3/9] mm: thp: add support for BPF based THP order selection
kernel test robot noticed the following build errors:
Hi Yafang,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Yafang-Shao/mm-thp-remove-vm_flags-parameter-from-khugepaged_enter_vma/20251015-222537 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20251015141716.887-4-laoar.shao%40gmail.com patch subject: [RFC PATCH v10 mm-new 3/9] mm: thp: add support for BPF based THP order selection config: x86_64-buildonly-randconfig-003-20251016 (https://download.01.org/0day-ci/archive/20251016/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251016/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
mm/mmap.c: In function 'dup_mmap':
>> mm/mmap.c:1844:17: error: implicit declaration of function 'bpf_thp_fork' [-Wimplicit-function-declaration]
1844 | bpf_thp_fork(mm, oldmm);
| ^~~~~~~~~~~~
--
kernel/fork.c: In function '__mmput':
>> kernel/fork.c:1133:9: error: implicit declaration of function 'bpf_thp_exit_mm' [-Wimplicit-function-declaration]
1133 | bpf_thp_exit_mm(mm);
| ^~~~~~~~~~~~~~~
--
fs/exec.c: In function 'exec_mmap':
>> fs/exec.c:893:9: error: implicit declaration of function 'bpf_thp_retain_mm' [-Wimplicit-function-declaration]
893 | bpf_thp_retain_mm(mm, old_mm);
| ^~~~~~~~~~~~~~~~~vim +/bpf_thp_fork +1844 mm/mmap.c
1715
1716 __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
1717 {
1718 struct vm_area_struct *mpnt, *tmp;
1719 int retval;
1720 unsigned long charge = 0;
1721 LIST_HEAD(uf);
1722 VMA_ITERATOR(vmi, mm, 0);
1723
1724 if (mmap_write_lock_killable(oldmm))
1725 return -EINTR;
1726 flush_cache_dup_mm(oldmm);
1727 uprobe_dup_mmap(oldmm, mm);
1728 /*
1729 * Not linked in yet - no deadlock potential:
1730 */
1731 mmap_write_lock_nested(mm, SINGLE_DEPTH_NESTING);
1732
1733 /* No ordering required: file already has been exposed. */
1734 dup_mm_exe_file(mm, oldmm);
1735
1736 mm->total_vm = oldmm->total_vm;
1737 mm->data_vm = oldmm->data_vm;
1738 mm->exec_vm = oldmm->exec_vm;
1739 mm->stack_vm = oldmm->stack_vm;
1740
1741 /* Use __mt_dup() to efficiently build an identical maple tree. */
1742 retval = __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_KERNEL);
1743 if (unlikely(retval))
1744 goto out;
1745
1746 mt_clear_in_rcu(vmi.mas.tree);
1747 for_each_vma(vmi, mpnt) {
1748 struct file *file;
1749
1750 vma_start_write(mpnt);
1751 if (mpnt->vm_flags & VM_DONTCOPY) {
1752 retval = vma_iter_clear_gfp(&vmi, mpnt->vm_start,
1753 mpnt->vm_end, GFP_KERNEL);
1754 if (retval)
1755 goto loop_out;
1756
1757 vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt));
1758 continue;
1759 }
1760 charge = 0;
1761 /*
1762 * Don't duplicate many vmas if we've been oom-killed (for
1763 * example)
1764 */
1765 if (fatal_signal_pending(current)) {
1766 retval = -EINTR;
1767 goto loop_out;
1768 }
1769 if (mpnt->vm_flags & VM_ACCOUNT) {
1770 unsigned long len = vma_pages(mpnt);
1771
1772 if (security_vm_enough_memory_mm(oldmm, len)) /* sic */
1773 goto fail_nomem;
1774 charge = len;
1775 }
1776
1777 tmp = vm_area_dup(mpnt);
1778 if (!tmp)
1779 goto fail_nomem;
1780 retval = vma_dup_policy(mpnt, tmp);
1781 if (retval)
1782 goto fail_nomem_policy;
1783 tmp->vm_mm = mm;
1784 retval = dup_userfaultfd(tmp, &uf);
1785 if (retval)
1786 goto fail_nomem_anon_vma_fork;
1787 if (tmp->vm_flags & VM_WIPEONFORK) {
1788 /*
1789 * VM_WIPEONFORK gets a clean slate in the child.
1790 * Don't prepare anon_vma until fault since we don't
1791 * copy page for current vma.
1792 */
1793 tmp->anon_vma = NULL;
1794 } else if (anon_vma_fork(tmp, mpnt))
1795 goto fail_nomem_anon_vma_fork;
1796 vm_flags_clear(tmp, VM_LOCKED_MASK);
1797 /*
1798 * Copy/update hugetlb private vma information.
1799 */
1800 if (is_vm_hugetlb_page(tmp))
1801 hugetlb_dup_vma_private(tmp);
1802
1803 /*
1804 * Link the vma into the MT. After using __mt_dup(), memory
1805 * allocation is not necessary here, so it cannot fail.
1806 */
1807 vma_iter_bulk_store(&vmi, tmp);
1808
1809 mm->map_count++;
1810
1811 if (tmp->vm_ops && tmp->vm_ops->open)
1812 tmp->vm_ops->open(tmp);
1813
1814 file = tmp->vm_file;
1815 if (file) {
1816 struct address_space *mapping = file->f_mapping;
1817
1818 get_file(file);
1819 i_mmap_lock_write(mapping);
1820 if (vma_is_shared_maywrite(tmp))
1821 mapping_allow_writable(mapping);
1822 flush_dcache_mmap_lock(mapping);
1823 /* insert tmp into the share list, just after mpnt */
1824 vma_interval_tree_insert_after(tmp, mpnt,
1825 &mapping->i_mmap);
1826 flush_dcache_mmap_unlock(mapping);
1827 i_mmap_unlock_write(mapping);
1828 }
1829
1830 if (!(tmp->vm_flags & VM_WIPEONFORK))
1831 retval = copy_page_range(tmp, mpnt);
1832
1833 if (retval) {
1834 mpnt = vma_next(&vmi);
1835 goto loop_out;
1836 }
1837 }
1838 /* a new mm has just been created */
1839 retval = arch_dup_mmap(oldmm, mm);
1840 loop_out:
1841 vma_iter_free(&vmi);
1842 if (!retval) {
1843 mt_set_in_rcu(vmi.mas.tree);
> 1844 bpf_thp_fork(mm, oldmm);
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH v10 mm-new 3/9] mm: thp: add support for BPF based THP order selection
In reply to Yafang Shaokernel test robot noticed the following build errors:
Hi Yafang,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Yafang-Shao/mm-thp-remove-vm_flags-parameter-from-khugepaged_enter_vma/20251015-222537 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20251015141716.887-4-laoar.shao%40gmail.com patch subject: [RFC PATCH v10 mm-new 3/9] mm: thp: add support for BPF based THP order selection config: x86_64-buildonly-randconfig-001-20251016 (https://download.01.org/0day-ci/archive/20251016/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251016/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> kernel/fork.c:1133:2: error: call to undeclared function 'bpf_thp_exit_mm'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
1133 | bpf_thp_exit_mm(mm);
| ^ kernel/fork.c:1133:2: note: did you mean 'bpf_thp_retain_mm'? include/linux/huge_mm.h:296:1: note: 'bpf_thp_retain_mm' declared here
296 | bpf_thp_retain_mm(struct mm_struct *mm, struct mm_struct *old_mm)
| ^
1 error generated.
vim +/bpf_thp_exit_mm +1133 kernel/fork.c
1124
1125 static inline void __mmput(struct mm_struct *mm)
1126 {
1127 VM_BUG_ON(atomic_read(&mm->mm_users));
1128
1129 uprobe_clear_state(mm);
1130 exit_aio(mm);
1131 ksm_exit(mm);
1132 khugepaged_exit(mm); /* must run before exit_mmap */
> 1133 bpf_thp_exit_mm(mm);
1134 exit_mmap(mm);
1135 mm_put_huge_zero_folio(mm);
1136 set_mm_exe_file(mm, NULL);
1137 if (!list_empty(&mm->mmlist)) {
1138 spin_lock(&mmlist_lock);
1139 list_del(&mm->mmlist);
1140 spin_unlock(&mmlist_lock);
1141 }
1142 if (mm->binfmt)
1143 module_put(mm->binfmt->module);
1144 lru_gen_del_mm(mm);
1145 futex_hash_free(mm);
1146 mmdrop(mm);
1147 }
1148
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC PATCH v10 4/9 mm: thp: decouple THP allocation between swap and page fault paths
In reply to Yafang ShaoThe new BPF capability enables finer-grained THP policy decisions by introducing separate handling for swap faults versus normal page faults.
As highlighted by Barry:
We’ve observed that swapping in large folios can lead to more swap thrashing for some workloads- e.g. kernel build. Consequently, some workloads might prefer swapping in smaller folios than those allocated by alloc_anon_folio().
While prtcl() could potentially be extended to leverage this new policy, doing so would require modifications to the uAPI.
--- include/linux/huge_mm.h | 3 ++- mm/huge_memory.c | 2 +- mm/memory.c | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-)
include/linux/huge_mm.h +2 -1
@@ -96,9 +96,10 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr;enum tva_type {TVA_SMAPS ,/* Exposing "THPeligible:" in smaps. */ -TVA_PAGEFAULT ,/* Serving a page fault. */ +TVA_PAGEFAULT ,/* Serving a non-swap page fault. */ TVA_KHUGEPAGED ,/* Khugepaged collapse. */ TVA_FORCED_COLLAPSE ,/* Forced collapse (e.g. MADV_COLLAPSE). */ +TVA_SWAP_PAGEFAULT ,/* serving a swap page fault. */ };#define thp_vma_allowable_order (vma, type, order) \
mm/huge_memory.c +1 -1
@@ -102,7 +102,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, unsigned long orders) {const bool smaps = type ==TVA_SMAPS ; -const bool in_pf = type ==TVA_PAGEFAULT ; +const bool in_pf = (type ==TVA_PAGEFAULT || type ==TVA_SWAP_PAGEFAULT );const bool forced_collapse = type ==TVA_FORCED_COLLAPSE ;unsigned long supported_orders;vm_flags_t vm_flags = vma->vm_flags;
mm/memory.c +1 -1
@@ -4558,7 +4558,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)* Get a list of all the (large) orders below PMD_ORDER that are enabled * and suitable for swapping THP. */ - orders =thp_vma_allowable_orders (vma,TVA_PAGEFAULT , + orders =thp_vma_allowable_orders (vma,TVA_SWAP_PAGEFAULT ,BIT (PMD_ORDER ) -1 ); orders =thp_vma_suitable_orders (vma, vmf->address, orders); orders = thp_swap_suitable_orders(swp_offset(entry),
Signature · 1 line
2.47.3
RFC PATCH v10 5/9 mm: thp: enable THP allocation exclusively through khugepaged
In reply to Yafang Shaokhugepaged_enter_vma() ultimately invokes any attached BPF function with the TVA_KHUGEPAGED flag set when determining whether or not to enable khugepaged THP for a freshly faulted in VMA.
Currently, on fault, we invoke this in do_huge_pmd_anonymous_page(), as invoked by create_huge_pmd() and only when we have already checked to see if an allowable TVA_PAGEFAULT order is specified.
Since we might want to disallow THP on fault-in but allow it via khugepaged, we move things around so we always attempt to enter khugepaged upon fault.
This change is safe because:
- khugepaged operates at the MM level rather than per-VMA. The THP allocation might fail during page faults due to transient conditions (e.g., memory pressure), it is safe to add this MM to khugepaged for subsequent defragmentation.
- If __thp_vma_allowable_orders(TVA_PAGEFAULT) returns 0, then __thp_vma_allowable_orders(TVA_KHUGEPAGED) will also return 0.
While we could also extend prctl() to utilize this new policy, such a change would require a uAPI modification to PR_SET_THP_DISABLE.
--- mm/huge_memory.c | 1 - mm/memory.c | 13 ++++++++----- 2 files changed, 8 insertions(+), 6 deletions(-)
mm/huge_memory.c +0 -1
@@ -1346,7 +1346,6 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) ret =vmf_anon_prepare (vmf);if (ret)return ret; -khugepaged_enter_vma (vma); if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm) &&
mm/memory.c +8 -5
@@ -6327,11 +6327,14 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,if (pud_trans_unstable (vmf.pud)) goto retry_pud; -if (pmd_none (*vmf.pmd) && -thp_vma_allowable_order (vma,TVA_PAGEFAULT ,PMD_ORDER )) { - ret =create_huge_pmd (&vmf); -if (!(ret &VM_FAULT_FALLBACK )) -return ret; +if (pmd_none (*vmf.pmd)) { +if (vma_is_anonymous (vma)) +khugepaged_enter_vma (vma); +if (thp_vma_allowable_order (vma,TVA_PAGEFAULT ,PMD_ORDER )) { + ret =create_huge_pmd (&vmf); +if (!(ret &VM_FAULT_FALLBACK )) +return ret; + } }else { vmf.orig_pmd =pmdp_get_lockless (vmf.pmd);
Signature · 1 line
2.47.3
RFC PATCH v10 6/9 bpf: mark mm->owner as __safe_rcu_or_null
In reply to Yafang ShaoWhen CONFIG_MEMCG is enabled, we can access mm->owner under RCU. The owner can be NULL. With this change, BPF helpers can safely access mm->owner to retrieve the associated task from the mm. We can then make policy decision based on the task attribute.
The typical use case is as follows,
bpf_rcu_read_lock(); // rcu lock must be held for rcu trusted field @owner = @mm->owner; // mm_struct::owner is rcu trusted or null if (!@owner)
goto out;
/* Do something based on the task attribute */
out:
bpf_rcu_read_unlock();
--- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/bpf/verifier.c +3 -0
@@ -7123,6 +7123,9 @@ BTF_TYPE_SAFE_RCU(struct cgroup_subsys_state) {/* RCU trusted: these fields are trusted in RCU CS and can be NULL */ BTF_TYPE_SAFE_RCU_OR_NULL(struct mm_struct) {struct file __rcu *exe_file; +#ifdef CONFIG_MEMCG +struct task_struct __rcu *owner; +#endif };/* skb->sk, req->sk are not RCU protected, but we mark them as such
Signature · 1 line
2.47.3
Re: [RFC PATCH v10 mm-new 6/9] bpf: mark mm->owner as __safe_rcu_or_null
Quoted 18 lines · Yafang Shao
When
CONFIG_MEMCGis enabled, we can accessmm->ownerunder RCU. The owner can be NULL. With this change, BPF helpers can safely accessmm->ownerto retrieve the associated task from the mm. We can then make policy decision based on the task attribute.The typical use case is as follows,
bpf_rcu_read_lock(); // rcu lock must be held for rcu trusted field @owner = @mm->owner; //mm_struct::owneris rcu trusted or null if (!@owner)goto out;/* Do something based on the task attribute */
out:
bpf_rcu_read_unlock();
Quoted 7 lines
Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Yafang Shao <[email protected]>Acked-by Lorenzo Stoakes <[email protected]>--- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+)
I thought you were going to send this and next patches outside of your thp patch set to land them sooner, as they don't have dependency on the rest of the patches and are useful on their own?
Quoted 17 lines
kernel/bpf/verifier.c +3
-1@@ -7123,6 +7123,9 @@ BTF_TYPE_SAFE_RCU(struct cgroup_subsys_state) {/* RCU trusted: these fields are trusted in RCU CS and can be NULL */ BTF_TYPE_SAFE_RCU_OR_NULL(struct mm_struct) {struct file __rcu *exe_file; +#ifdef CONFIG_MEMCG +struct task_struct __rcu *owner; +#endif };/* skb->sk, req->sk are not RCU protected, but we mark them as such -- 2.47.3
Re: [RFC PATCH v10 mm-new 6/9] bpf: mark mm->owner as __safe_rcu_or_null
Quoted 28 lines · Andrii Nakryiko
Quoted 18 lines · Yafang Shao
When
CONFIG_MEMCGis enabled, we can accessmm->ownerunder RCU. The owner can be NULL. With this change, BPF helpers can safely accessmm->ownerto retrieve the associated task from the mm. We can then make policy decision based on the task attribute.The typical use case is as follows,
bpf_rcu_read_lock(); // rcu lock must be held for rcu trusted field @owner = @mm->owner; //mm_struct::owneris rcu trusted or null if (!@owner)goto out;/* Do something based on the task attribute */
out:
bpf_rcu_read_unlock();Quoted 7 lines
Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Yafang Shao <[email protected]>Acked-by Lorenzo Stoakes <[email protected]>--- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+)
Quoted 3 lines
I thought you were going to send this and next patches outside of your thp patch set to land them sooner, as they don't have dependency on the rest of the patches and are useful on their own?
Thanks for your reminder.
They have been sent separately:
https://lore.kernel.org/bpf/[email protected]/
Signature · 2 lines
Regards Yafang
Re: [RFC PATCH v10 mm-new 6/9] bpf: mark mm->owner as __safe_rcu_or_null
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Andrii Nakryiko
Quoted 25 lines · Yafang Shao
When
CONFIG_MEMCGis enabled, we can accessmm->ownerunder RCU. The owner can be NULL. With this change, BPF helpers can safely accessmm->ownerto retrieve the associated task from the mm. We can then make policy decision based on the task attribute.The typical use case is as follows,
bpf_rcu_read_lock(); // rcu lock must be held for rcu trusted field @owner = @mm->owner; //mm_struct::owneris rcu trusted or null if (!@owner)goto out;/* Do something based on the task attribute */
out:
bpf_rcu_read_unlock();Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Yafang Shao <[email protected]>Acked-by Lorenzo Stoakes <[email protected]>--- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+)
Quoted 8 lines
Quoted 3 lines
I thought you were going to send this and next patches outside of your thp patch set to land them sooner, as they don't have dependency on the rest of the patches and are useful on their own?
Thanks for your reminder.
They have been sent separately:
Could we respin this as a v2 without them then please so we can sensibly keep the two separate?
Thanks!
Quoted 4 lines
--
Regards
Yafang
I do intend to have a look through the various conversations on this, just catching up after 2 weeks vacation :) in the kernel this is an eternity, even during the merge window it seems :P
Cheers, Lorenzo
Re: [RFC PATCH v10 mm-new 6/9] bpf: mark mm->owner as __safe_rcu_or_null
Quoted 36 lines · Lorenzo Stoakes
Quoted 34 lines · Yafang Shao
Quoted 31 lines · Andrii Nakryiko
Quoted 18 lines · Yafang Shao
When
CONFIG_MEMCGis enabled, we can accessmm->ownerunder RCU. The owner can be NULL. With this change, BPF helpers can safely accessmm->ownerto retrieve the associated task from the mm. We can then make policy decision based on the task attribute.The typical use case is as follows,
bpf_rcu_read_lock(); // rcu lock must be held for rcu trusted field @owner = @mm->owner; //mm_struct::owneris rcu trusted or null if (!@owner)goto out;/* Do something based on the task attribute */
out:
bpf_rcu_read_unlock();Quoted 7 lines
Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Yafang Shao <[email protected]>Acked-by Lorenzo Stoakes <[email protected]>--- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+)I thought you were going to send this and next patches outside of your thp patch set to land them sooner, as they don't have dependency on the rest of the patches and are useful on their own?
Quoted 7 lines
Quoted 4 lines
Thanks for your reminder.
They have been sent separately:Could we respin this as a v2 without them then please so we can sensibly keep the two separate?
Sure, I will send a v2.
Quoted 11 lines
Thanks!
Quoted 4 lines
--
Regards
YafangI do intend to have a look through the various conversations on this, just catching up after 2 weeks vacation :) in the kernel this is an eternity, even during the merge window it seems :P
Huh, we've been refactoring a bit too fast and furious since your last review ;-)
Signature · 2 lines
Regards Yafang
RFC PATCH v10 7/9 bpf: mark vma->vm_mm as __safe_trusted_or_null
In reply to Yafang ShaoThe vma->vm_mm might be NULL and it can be accessed outside of RCU. Thus, we can mark it as trusted_or_null. With this change, BPF helpers can safely access vma->vm_mm to retrieve the associated mm_struct from the VMA. Then we can make policy decision from the VMA.
The "trusted" annotation enables direct access to vma->vm_mm within kfuncs marked with KF_TRUSTED_ARGS or KF_RCU, such as bpf_task_get_cgroup1() and bpf_task_under_cgroup(). Conversely, "null" enforcement requires all callsites using vma->vm_mm to perform NULL checks.
The lsm selftest must be modified because it directly accesses vma->vm_mm without a NULL pointer check; otherwise it will break due to this change.
For the VMA based THP policy, the use case is as follows,
@mm = @vma->vm_mm; // vm_area_struct::vm_mm is trusted or null if (!@mm)
return;
bpf_rcu_read_lock(); // rcu lock must be held to dereference the owner
@owner = @mm->owner; // mm_struct::owner is rcu trusted or null
if (!@owner)
goto out; @cgroup1 = bpf_task_get_cgroup1(@owner, MEMCG_HIERARCHY_ID);
/* make the decision based on the @cgroup1 attribute */
bpf_cgroup_release(@cgroup1); // release the associated cgroup
out:
bpf_rcu_read_unlock();
PSI memory information can be obtained from the associated cgroup to inform policy decisions. Since upstream PSI support is currently limited to cgroup v2, the following example demonstrates cgroup v2 implementation:
@owner = @mm->owner;
if (@owner) {
// @ancestor_cgid is user-configured
@ancestor = bpf_cgroup_from_id(@ancestor_cgid);
if (bpf_task_under_cgroup(@owner, @ancestor)) {
@psi_group = @ancestor->psi; /* Extract PSI metrics from @psi_group and
* implement policy logic based on the values
*/
}}
--- kernel/bpf/verifier.c | 5 +++++ tools/testing/selftests/bpf/progs/lsm.c | 8 +++++--- 2 files changed, 10 insertions(+), 3 deletions(-)
kernel/bpf/verifier.c +5 -0
@@ -7165,6 +7165,10 @@ BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct socket) {struct sock *sk; }; +BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct vm_area_struct) { +struct mm_struct *vm_mm; +}; +static bool type_is_rcu (struct bpf_verifier_env *env,struct bpf_reg_state *reg,const char *field_name,u32 btf_id)
@@ -7206,6 +7210,7 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env, {BTF_TYPE_EMIT (BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct socket ));BTF_TYPE_EMIT (BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct dentry )); +BTF_TYPE_EMIT (BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct vm_area_struct ));return btf_nested_type_is_trusted (&env->log, reg, field_name, btf_id,"__safe_trusted_or_null" );
tools/testing/selftests/bpf/progs/lsm.c +5 -3
@@ -89,14 +89,16 @@ SEC("lsm/file_mprotect")int BPF_PROG (test_int_hook ,struct vm_area_struct *vma,unsigned long reqprot,unsigned long prot,int ret) { -if (ret !=0 ) +struct mm_struct *mm = vma->vm_mm; + +if (ret !=0 || !mm)return ret;__s32 pid =bpf_get_current_pid_tgid () >>32 ;int is_stack =0 ; - is_stack = (vma->vm_start <= vma->vm_mm->start_stack && - vma->vm_end >= vma->vm_mm->start_stack); + is_stack = (vma->vm_start <= mm->start_stack && + vma->vm_end >= mm->start_stack);if (is_stack && monitored_pid == pid) { mprotect_count++;
Signature · 1 line
2.47.3
RFC PATCH v10 8/9 selftests/bpf: add a simple BPF based THP policy
In reply to Yafang ShaoThis test case implements a basic THP policy that sets THPeligible to 1 for a specific task and to 0 for all others. I selected THPeligible for verification because its straightforward nature makes it ideal for validating the BPF THP policy functionality.
Below configs must be enabled for this test:
CONFIG_BPF_THP=y
CONFIG_MEMCG=y
CONFIG_TRANSPARENT_HUGEPAGE=y
--- MAINTAINERS | 2 + tools/testing/selftests/bpf/config | 3 + .../selftests/bpf/prog_tests/thp_adjust.c | 245 ++++++++++++++++++ .../selftests/bpf/progs/test_thp_adjust.c | 23 ++ 4 files changed, 273 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c
MAINTAINERS +2 -0
@@ -16260,6 +16260,8 @@ F: mm/huge_memory.c F: mm/huge_memory_bpf.c F: mm/khugepaged.c F: mm/mm_slot.h +F: tools/testing/selftests/bpf/prog_tests/thp_adjust.c +F: tools/testing/selftests/bpf/progs/test_thp_adjust* F: tools/testing/selftests/mm/khugepaged.c F: tools/testing/selftests/mm/split_huge_page_test.c F: tools/testing/selftests/mm/transhuge-stress.c
tools/testing/selftests/bpf/config +3 -0
@@ -9,6 +9,7 @@ CONFIG_BPF_LIRC_MODE2=y CONFIG_BPF_LSM=y CONFIG_BPF_STREAM_PARSER=y CONFIG_BPF_SYSCALL=y +CONFIG_BPF_THP=y # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set CONFIG_CGROUP_BPF=y CONFIG_CRYPTO_HMAC=y
@@ -51,6 +52,7 @@ CONFIG_IPV6_TUNNEL=y CONFIG_KEYS=y CONFIG_LIRC=y CONFIG_LWTUNNEL=y +CONFIG_MEMCG=y CONFIG_MODULE_SIG=y CONFIG_MODULE_SRCVERSION_ALL=y CONFIG_MODULE_UNLOAD=y
@@ -114,6 +116,7 @@ CONFIG_SECURITY=y CONFIG_SECURITYFS=y CONFIG_SYN_COOKIES=y CONFIG_TEST_BPF=m +CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_UDMABUF=y CONFIG_USERFAULTFD=y CONFIG_VSOCKETS=y
tools/testing/selftests/bpf/prog_tests/thp_adjust.c new file +245 -0
245 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_thp_adjust.c new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +char _license[] SEC("license") = "GPL"; + +int pmd_order; + +SEC("struct_ops/thp_get_order") +int BPF_PROG (thp_eligible ,struct vm_area_struct *vma,enum tva_type type, +unsigned long orders) +{ +if (type !=TVA_SMAPS ) +return 0 ; +return pmd_order; +} + +SEC(".struct_ops.link") +struct bpf_thp_ops thp_eligible_ops = { + .thp_get_order = (void *)thp_eligible, +};
Signature · 1 line
2.47.3
RFC PATCH v10 9/9 Documentation: add BPF-based THP policy management
In reply to Yafang ShaoAdd the documentation.
--- Documentation/admin-guide/mm/transhuge.rst | 39 ++++++++++++++++++++++ 1 file changed, 39 insertions(+)
Documentation/admin-guide/mm/transhuge.rst +39 -0
@@ -738,3 +738,42 @@ support enabled just fine as always. No difference can be noted in hugetlbfs other than there will be less overall fragmentation. All usual features belonging to hugetlbfs are preserved and unaffected. libhugetlbfs will also work fine as usual. + +BPF THP +======= + +Overview +-------- + +When the system is configured with "always" or "madvise" THP mode, a BPF program +can be used to adjust THP allocation policies dynamically. This enables +fine-grained control over THP decisions based on various factors including +workload identity, allocation context, and system memory pressure. + +Program Interface +----------------- + +This feature implements a struct_ops BPF program with the following interface:: + + int thp_get_order(struct vm_area_struct *vma, + enum tva_type type, + unsigned long orders); + +Parameters:: + + @vma: vm_area_struct associated with the THP allocation + @type: TVA type for current @vma + @orders: Bitmask of available THP orders for this allocation + +Return value:: + + The suggested THP order for allocation from the BPF program. Must be + a valid, available order. + +Implementation Notes +-------------------- + +This is currently an experimental feature. CONFIG_BPF_THP (EXPERIMENTAL) must be +enabled to use it. Only one BPF program can be attached at a time, but the +program can be updated dynamically to adjust policies without requiring affected +tasks to be restarted.
Signature · 1 line
2.47.3