Skip to contentSkip to conversation

mm/page_owner: add bpf_iter and kfuncs

PATCH In review Zhen Ni 5 participants 23 messages Oct 21 – Oct 25

Subsystems (6)
lore
Share & tools
lore

Zhen Ni

author
#0 lore raw

v3 cover mm/page_owner: add bpf_iter and kfuncs

Hi all,

Introduction
============

page_owner records who allocated every page and is a core memory-debug facility, but it currently reaches userspace as the debugfs flat dump:
slicing allocations by pid, comm, NUMA node, cgroup or stack, etc., means streaming and parsing the entire file in userspace, which is expensive on large hosts and hard to query live.

Exposing the same walk to eBPF lets a program filter and aggregate in the kernel and surface only the records it cares about. This series adds that access as a seq_file bpf_iter target and an open-coded iterator.

Implementation
==============

The page walk is factored into page_owner_next_eligible(), shared by the read path and the iterators so eligibility stays identical. Two kfuncs expose information that a BPF program cannot decode from the snapshot alone: per-page memcg info including the full cgroup path, and the stack trace behind a stack_depot handle. Each record is snapshotted under the page_ext RCU lock, so consumers read a consistent copy.

Testing
=======

The selftest exercises the seq_file and open-coded iterators, and validates the stack_snprint and get_memcg_info kfuncs.

Changes in v3
=============
- Add a mem-hotplug read lock across the page scan.
- Drop the live struct page from the scan cursor and the bpf_iter context.
- Check the page_owner_inited static key in the iterator.
- Without scan->page, start at min_low_pfn - 1 and let next() pre-increment.
- Gate the kfuncs with KF_PERFMON.
- Enable BPF_ITER_RESCHED for the bpf_iter target.
- Check ENODATA/EOPNOTSUPP from <errno.h>.

Changes in v2
=============
- Route the zero-handle case through ext_put_continue.
- Drop the scan->pfn++.
- Fill page/po/pfn only when v is non-NULL.
- Implement page_owner_seq_stop().
- bpf_page_owner_stack_snprint() now takes the scan cursor.
- Buffer args renamed to buf__uninit/buf__sz.
- Removes the v1 get_nid() kfunc. The node is now stored in scan->nid.
- bpf_page_owner_get_memcg_info() now resolves the page with pfn_to_online_page() for an offline or removed pfn.
- Buffer/info args renamed to info__uninit/buf__uninit/buf__sz.
- Drop the get_nid kfunc coverage.
- Enable CONFIG_PAGE_OWNER in the bpf selftests config.

v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/

Zhen Ni (7):
mm/page_owner: extract page_owner_next_eligible() from read loop mm/page_owner: protect page access with mem-hotplug read lock mm/page_owner: add bpf_iter target "page_owner" mm/page_owner: add open-coded page_owner iterator mm/page_owner: record the NUMA node in the scan cursor mm/page_owner: add bpf_page_owner_get_memcg_info() kfunc selftests/bpf: add tests for the page_owner bpf_iter target

 kernel/bpf/helpers.c                          |  10 +
 mm/Makefile                                   |   3 +
 mm/page_owner.c                               | 252 ++++++++++++------
 mm/page_owner.h                               |  60 +++++
 mm/page_owner_iter.c                          | 232 ++++++++++++++++
 tools/testing/selftests/bpf/config            |   2 +
 .../bpf/prog_tests/page_owner_iter.c          | 193 ++++++++++++++
 .../selftests/bpf/progs/bpf_iter_page_owner.c | 157 +++++++++++
 8 files changed, 826 insertions(+), 83 deletions(-)
 create mode 100644 mm/page_owner.h
 create mode 100644 mm/page_owner_iter.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/page_owner_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_page_owner.c
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v3 1/7 mm/page_owner: extract page_owner_next_eligible() from read loop

Move the page scanning logic out of read_page_owner() into a dedicated helper, in preparation for using it in the page_owner bpf_iter target.

The helper advances a struct page_owner_scan cursor to the next page eligible for page_owner output and returns true on a hit. On a hit it takes a snapshot of the page_owner record inside the page_ext RCU window.

No change to the output is intended. handle is read from the snapshot, taken before the non-zero check, so the validated and reported handle are the same value.

---
Changes in v3:
- Drop the page field in struct page_owner_scan.

Changes in v2:
- Route the zero-handle case through ext_put_continue.

v1: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/

---
 mm/page_owner.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
mm/page_owner.c +64 -37
index fbbda7ba914b..92a92cdf74ec 100644
@@ -36,6 +36,15 @@ struct page_owner {
 	pid_t free_tgid;
 };
 
+/*
+ * Cursor and per-page result of page_owner_next_eligible().
+ */
+struct page_owner_scan {
+	/* resume point */
+	unsigned long pfn;
+	struct page_owner po_snap;
+};
+
 struct stack {
 	struct stack_record *stack_record;
 	struct stack *next;
@@ -725,37 +734,24 @@ void __dump_page_owner(const struct page *page)
 	page_ext_put(page_ext);
 }
 
-static ssize_t
-read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
+/*
+ * Advance @scan to the next page eligible for page_owner output.
+ * On a hit, fill scan->pfn/po_snap and return true; the caller
+ * must advance scan->pfn past the hit before calling again to resume
+ * the scan.
+ */
+static bool page_owner_next_eligible(struct page_owner_scan *scan)
 {
-	unsigned long pfn;
-	struct page *page;
-	struct page_ext *page_ext;
-	struct page_owner *page_owner;
-	depot_stack_handle_t handle;
-	struct page_owner_filter_state *state = file->private_data;
+	unsigned long pfn = scan->pfn;
 
-	if (!static_branch_unlikely(&page_owner_inited))
-		return -EINVAL;
-
-	page = NULL;
-	if (*ppos == 0)
-		pfn = min_low_pfn;
-	else
-		pfn = *ppos;
 	/* Find a valid PFN or the start of a MAX_ORDER_NR_PAGES area */
 	while (!pfn_valid(pfn) && (pfn & (MAX_ORDER_NR_PAGES - 1)) != 0)
 		pfn++;
 
-	/* Find an allocated page */
 	for (; pfn < max_pfn; pfn++) {
-		/*
-		 * This temporary page_owner is required so
-		 * that we can avoid the context switches while holding
-		 * the rcu lock and copying the page owner information to
-		 * user through copy_to_user() or GFP_KERNEL allocations.
-		 */
-		struct page_owner page_owner_tmp;
+		struct page_owner *page_owner;
+		struct page_ext *page_ext;
+		struct page *page;
 
 		/*
 		 * If the new page is in a new MAX_ORDER_NR_PAGES area,
@@ -798,13 +794,45 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 			goto ext_put_continue;
 
 		/*
-		 * Access to page_ext->handle isn't synchronous so we should
-		 * be careful to access it.
+		 * Take a snapshot of the page_owner record inside the RCU
+		 * window, then validate it, so that print_page_owner()
+		 * reads the checked handle from the same snapshot.
 		 */
-		handle = READ_ONCE(page_owner->handle);
-		if (!handle)
+		scan->pfn = pfn;
+		scan->po_snap = *page_owner;
+
+		if (!scan->po_snap.handle)
 			goto ext_put_continue;
 
+		page_ext_put(page_ext);
+		return true;
+
+ext_put_continue:
+		page_ext_put(page_ext);
+		cond_resched();
+	}
+
+	return false;
+}
+
+static ssize_t
+read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
+{
+	struct page_owner_scan scan;
+	struct page_owner_filter_state *state = file->private_data;
+
+	if (!static_branch_unlikely(&page_owner_inited))
+		return -EINVAL;
+
+	if (*ppos == 0)
+		scan.pfn = min_low_pfn;
+	else
+		scan.pfn = *ppos;
+
+	/* Find an allocated page */
+	while (page_owner_next_eligible(&scan)) {
+		struct page *page = pfn_to_page(scan.pfn);
+
 		if (state->nid_filter_enabled) {
 			int nid;
 			memdesc_flags_t page_flags = READ_ONCE(page->flags);
@@ -814,21 +842,20 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 			 * VM_BUG_ON when accessing poisoned pages.
 			 */
 			if (page_flags.f == PAGE_POISON_PATTERN)
-				goto ext_put_continue;
+				goto skip_continue;
 			nid = memdesc_nid(&page_flags);
 			if (!node_isset(nid, state->nid_filter))
-				goto ext_put_continue;
+				goto skip_continue;
 		}
 
 		/* Record the next PFN to read in the file offset */
-		*ppos = pfn + 1;
+		*ppos = scan.pfn + 1;
 
-		page_owner_tmp = *page_owner;
-		page_ext_put(page_ext);
-		return print_page_owner(buf, count, pfn, page,
-				&page_owner_tmp, handle, state);
-ext_put_continue:
-		page_ext_put(page_ext);
+		return print_page_owner(buf, count, scan.pfn, page,
+				&scan.po_snap, scan.po_snap.handle, state);
+
+skip_continue:
+		scan.pfn++;
 		cond_resched();
 	}
 
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v3 1/7] mm/page_owner: extract page_owner_next_eligible() from read loop

TL;DR: The nid filter's page->flags read now happens outside the page_ext RCU window, so it may fault on an unmapped vmemmap during memory hot-remove; later patches fix it, so it is only a bisect…
Quoted 1 line

mm/page_owner: extract page_owner_next_eligible() from read loop

TL;DR: The nid filter's page->flags read now happens outside the page_ext RCU window, so it may fault on an unmapped vmemmap during memory hot-remove; later patches fix it, so it is only a bisect window.

Quoted 4 lines
mm/page_owner.c +0 -0
index fbbda7ba914b..92a92cdf74ec 100644

[ ... ]

Quoted 29 lines
mm/page_owner.c +18 -5
@@ -798,13 +794,45 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 			goto ext_put_continue;
 
-		/*
-		 * Access to page_ext->handle isn't synchronous so we should
-		 * be careful to access it.
+		/*
+		 * Take a snapshot of the page_owner record inside the RCU
+		 * window, then validate it, so that print_page_owner()
+		 * reads the checked handle from the same snapshot.
 		 */
-		handle = READ_ONCE(page_owner->handle);
-		if (!handle)
+		scan->pfn = pfn;
+		scan->po_snap = *page_owner;
+
+		if (!scan->po_snap.handle)
 			goto ext_put_continue;
 
+		page_ext_put(page_ext);
+		return true;
+
+ext_put_continue:
+		page_ext_put(page_ext);
+		cond_resched();
+	}
+
+	return false;
+}

[ ... ]

mm/page_owner.c read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos) · line 838

Quoted 7 lines
mm/page_owner.c +4 -0
+	/* Find an allocated page */
+	while (page_owner_next_eligible(&scan)) {
+		struct page *page = pfn_to_page(scan.pfn);
+
 		if (state->nid_filter_enabled) {
 			int nid;
 			memdesc_flags_t page_flags = READ_ONCE(page->flags);

Can this READ_ONCE(page->flags) fault on an unmapped vmemmap?

In the parent commit the nid check ran between page_ext_get() and page_ext_put(), so it was inside rcu_read_lock():

    page_ext = page_ext_get(page);   /* rcu_read_lock() */
    ...
    if (state->nid_filter_enabled) {
        memdesc_flags_t page_flags = READ_ONCE(page->flags);
        ...
            goto ext_put_continue;
    }
    ...
    ext_put_continue:
        page_ext_put(page_ext);      /* rcu_read_unlock() */

Now page_owner_next_eligible() calls page_ext_put() before it returns true, and read_page_owner() reads page->flags with no RCU section, no get_online_mems(), and no page reference held.

The RCU section mattered for memory hot-remove:

offline_pages()
memory_notify(MEM_OFFLINE)
page_ext_callback()
offline_page_ext()
synchronize_rcu()

try_remove_memory() requires the blocks to be offline before arch_remove_memory() -> ... -> section_deactivate() -> depopulate_section_memmap() unmaps the vmemmap. While a page_ext_get() reader was active, that section's memmap could not be torn down.

After this patch, an offline plus remove can complete between page_ext_put() and the READ_ONCE(page->flags), because synchronize_rcu() no longer waits for this reader. This needs CONFIG_PAGE_OWNER with page_owner=on, CONFIG_MEMORY_HOTREMOVE, a nid filter set through page_owner_write(), and a concurrent hot-remove of movable memory.

The commit message says "No change to the output is intended", but it does not mention that the nid filter's struct page access moves out of the page_ext RCU window.

For context, the parent already read struct page without a pin in skip_buddy_pages() (PageBuddy() before page_ext_get()) and in print_page_owner() (page->flags, pageblock flags and page->memcg_data after page_ext_put()), so this widens an existing race rather than adding a new class of problem. It does turn one protected access into an unprotected one, though.

This is fixed later in the series, so it is only a bisect window issue:

- 915f86c7862f ("mm/page_owner: protect page access with mem-hotplug read lock") re-resolves the page in read_page_owner() with pfn_to_online_page() under get_online_mems() before the nid check.

- 2cf991a93cbd ("mm/page_owner: record the NUMA node in the scan cursor") moves the nid computation back into page_owner_next_eligible(), "to avoid dereferencing a possibly-unmapped page".

Could the nid be read in page_owner_next_eligible() before page_ext_put(), for example by storing it in the cursor? Alternatively, could the mem-hotplug protection patch be reordered ahead of this one?

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39593164040

author
#3 lore raw

PATCH v3 2/7 mm/page_owner: protect page access with mem-hotplug read lock

In reply to Zhen Ni

The page_owner scan walks PFNs locklessly and dereferences the resolved struct page in the scan itself and in print_page_owner(). With CONFIG_MEMORY_HOTREMOVE, a concurrent remove_memory() can tear down the section's vmemmap, so a dereference after a stale validity check may fault.

Take a mem-hotplug read lock across each page's check-then-deref. In the scan, gate every PFN with pfn_to_online_page() under the reader and drop it before cond_resched(), so hotplug can still interleave between pages; in print_page_owner(), re-resolve the page under a reader covering just the print of one record.

The reader is cheap: get_online_mems()/put_online_mems() are just per-CPU counter increments/decrements when no hotplug is active, so they contend only with an actual add/remove. Taking the lock per-page bounds the writer's wait to a single page's check-and-deref, keeping hot-remove stalls brief while all accesses stay protected.

This makes the traversal crash-free. A recycled-but-present page can still yield stale data -- the lockless scan is best-effort and unchanged by this patch.

---
Changes in v3:
- New patch. Add the mem-hotplug protect.

---
 mm/page_owner.c | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
mm/page_owner.c +30 -5
index 92a92cdf74ec..8e72a85c663c 100644
@@ -762,13 +762,22 @@ static bool page_owner_next_eligible(struct page_owner_scan *scan)
 			continue;
 		}
 
-		page = pfn_to_page(pfn);
+		/*
+		 * Hold a mem-hotplug reader across the page derefs so the
+		 * vmemmap isn't torn down mid-scan; released before
+		 * cond_resched().
+		 */
+		get_online_mems();
+		page = pfn_to_online_page(pfn);
+		if (!page)
+			goto put_online_continue;
+
 		if (skip_buddy_pages(&pfn, page))
-			continue;
+			goto put_online_continue;
 
 		page_ext = page_ext_get(page);
 		if (unlikely(!page_ext))
-			continue;
+			goto put_online_continue;
 
 		/*
 		 * Some pages could be missed by concurrent allocation or free,
@@ -805,10 +814,13 @@ static bool page_owner_next_eligible(struct page_owner_scan *scan)
 			goto ext_put_continue;
 
 		page_ext_put(page_ext);
+		put_online_mems();
 		return true;
 
 ext_put_continue:
 		page_ext_put(page_ext);
+put_online_continue:
+		put_online_mems();
 		cond_resched();
 	}
 
@@ -831,7 +843,17 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 
 	/* Find an allocated page */
 	while (page_owner_next_eligible(&scan)) {
-		struct page *page = pfn_to_page(scan.pfn);
+		struct page *page;
+		ssize_t ret;
+
+		/*
+		 * The scan is lockless. Re-resolve the page under a mem-hotplug
+		 * read-side lock so that we never access a hot-removed section.
+		 */
+		get_online_mems();
+		page = pfn_to_online_page(scan.pfn);
+		if (!page)
+			goto skip_continue;
 
 		if (state->nid_filter_enabled) {
 			int nid;
@@ -851,10 +873,13 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 		/* Record the next PFN to read in the file offset */
 		*ppos = scan.pfn + 1;
 
-		return print_page_owner(buf, count, scan.pfn, page,
+		ret = print_page_owner(buf, count, scan.pfn, page,
 				&scan.po_snap, scan.po_snap.handle, state);
+		put_online_mems();
+		return ret;
 
 skip_continue:
+		put_online_mems();
 		scan.pfn++;
 		cond_resched();
 	}
Signature · 1 line
2.20.1
#4 lore raw

Re: [PATCH v3 2/7] mm/page_owner: protect page access with mem-hotplug read lock

TL;DR: read_page_owner() holds the mem-hotplug read lock across copy_to_user(); a page fault there may deadlock against mmap_lock (lock-order inversion with memremap_pages callers) and userspace can…
Quoted 1 line

mm/page_owner: protect page access with mem-hotplug read lock

TL;DR: read_page_owner() holds the mem-hotplug read lock across copy_to_user(); a page fault there may deadlock against mmap_lock (lock-order inversion with memremap_pages callers) and userspace can stall hot-remove indefinitely.

Quoted 3 lines
mm/page_owner.c +0 -0

[ ... ]

Quoted 16 lines
mm/page_owner.c +11 -1
@@ -831,7 +843,17 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 
 	/* Find an allocated page */
 	while (page_owner_next_eligible(&scan)) {
-		struct page *page = pfn_to_page(scan.pfn);
+		struct page *page;
+		ssize_t ret;
+
+		/*
+		 * The scan is lockless. Re-resolve the page under a mem-hotplug
+		 * read-side lock so that we never access a hot-removed section.
+		 */
+		get_online_mems();
+		page = pfn_to_online_page(scan.pfn);
+		if (!page)
+			goto skip_continue;

[ ... ]

mm/page_owner.c read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos) · line 879

Quoted 9 lines
mm/page_owner.c +3 -1
@@ -851,10 +873,13 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 		/* Record the next PFN to read in the file offset */
 		*ppos = scan.pfn + 1;
 
-		return print_page_owner(buf, count, scan.pfn, page,
+		ret = print_page_owner(buf, count, scan.pfn, page,
 				&scan.po_snap, scan.po_snap.handle, state);
+		put_online_mems();
+		return ret;
                ^^^^

Can this deadlock against mmap_lock? read_page_owner() now holds the mem_hotplug_lock reader across the whole print_page_owner() call, and that includes the copy_to_user() of the formatted record:

print_page_owner()

    ...
    if (copy_to_user(buf, kbuf, ret))
            ret = -EFAULT;

Once kbuf is formatted, nothing in that copy touches struct page, so the reader isn't needed there. A user page fault on buf taken inside this region takes current->mm->mmap_lock for read (on x86 a fault from kernel-mode uaccess always does, see the 'goto lock_mmap' path in arch/x86/mm/fault.c for !FAULT_FLAG_USER), and __might_fault() records that dependency for lockdep. That creates the lock order:

mem_hotplug_lock(R) -> mmap_lock(R)

The opposite order already exists in the tree. Code that hotplugs device memory while holding mmap_read_lock goes through memremap_pages() -> pagemap_range() -> mem_hotplug_begin(), which does cpus_read_lock() and then percpu_down_write(&mem_hotplug_lock):

nouveau_svmm_bind()
mmap_read_lock(mm)
nouveau_dmem_migrate_vma()
nouveau_dmem_migrate_chunk()
nouveau_dmem_migrate_copy_one()
nouveau_dmem_page_alloc_locked()
nouveau_dmem_chunk_alloc()
memremap_pages()

lib/test_hmm.c:dmirror_migrate_to_device()
mmap_read_lock(mm)
dmirror_migrate_alloc_and_copy()
dmirror_devmem_alloc_page()
dmirror_allocate_chunk()
memremap_pages()

With three threads sharing one mm, could this sequence occur?

T1 read(page_owner):
get_online_mems(), then copy_to_user() faults on buf
T2 SVM bind ioctl:
holds mmap_read_lock(mm), blocks in percpu_down_write(&mem_hotplug_lock) waiting for T1
T3 mmap()/munmap():
queues on mmap_write_lock(mm) behind T2
T1 fault:
mmap_read_lock(mm) blocks behind T3

None of these waits has a timeout. While T2 waits it also holds cpus_read_lock(), so CPU hotplug writers and every later get_online_mems() caller stall too.

Even without that cycle, the reader can now be held for as long as userspace wants. A buf backed by userfaultfd or by a slow or remote file mapping could stall memory hot-remove for an unbounded time. Does that match the commit message claim:

"Taking the lock per-page bounds the writer's wait to a single page's check-and-deref, keeping hot-remove stalls brief while all accesses stay protected."

For comparison, ptdump_walk_pgd() already takes get_online_mems() followed by mmap_write_lock(current->mm). This new instance sits on a much more commonly used debug path, and here the lock isn't needed around copy_to_user() at all.

Would it work to format the record under the reader (or snapshot the page-derived fields: pageblock migratetype, page->flags, memcg info), call put_online_mems(), and only then call copy_to_user()? The kmalloc() in print_page_owner() could also move outside the locked region.

The last commit in the series (5b123f2cd336 "selftests/bpf: add tests for the page_owner bpf_iter target") still calls print_page_owner() under the reader, so the series does not address this later.

mm/page_owner.c read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos) · line 885

Quoted 6 lines
mm/page_owner.c +1 -0
 skip_continue:
+		put_online_mems();
 		scan.pfn++;
 		cond_resched();
 	}

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39593164040

author
#5 lore raw

PATCH v3 3/7 mm/page_owner: add bpf_iter target "page_owner"

In reply to Zhen Ni

Register a bpf_iter target "page_owner" that walks the pages eligible for page_owner output, with the same eligibility rules as the /sys/kernel/debug/page_owner read path.

For each eligible page the attached BPF program receives the pfn and the page_owner record, and can freely filter and format output.

The po_snap record in the scan cursor is taken inside the page_ext RCU window and handed to the BPF program as a copy, so no RCU window is needed at consumption time. Torn reads remain possible if the record is concurrently updated while it is copied, but this is the same best-effort behavior as the read path has always had.

The target is registered from pageowner_init() when both CONFIG_PAGE_OWNER and CONFIG_BPF_SYSCALL are enabled; a registration failure only disables the bpf_iter target.

Userspace tools such as bpftrace can attach to this target like any other bpf_iter. For example, to count pages owned by init (pid 1):

    bpftrace -e 'iter:page_owner / ctx->po->pid == 1 /
                 { @ = sum(1 << ctx->po->order); }'

---
Changes in v3:
- Drop the page pointer from the BPF context.
- Enable BPF_ITER_RESCHED.
- Fix the bpftrace example in the commit message.

Changes in v2:
- Drop the scan->pfn++.
- Fill page/po/pfn only when v is non-NULL.
- Implement page_owner_seq_stop().

v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/

---
 mm/Makefile          |   3 ++
 mm/page_owner.c      |  32 +++---------
 mm/page_owner.h      |  47 +++++++++++++++++
 mm/page_owner_iter.c | 118 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 175 insertions(+), 25 deletions(-)
 create mode 100644 mm/page_owner.h
 create mode 100644 mm/page_owner_iter.c
mm/Makefile +3 -0
index e7245cb88c66..d7307b5a964c 100644
@@ -115,6 +115,9 @@ obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o
 obj-$(CONFIG_DEBUG_RODATA_TEST) += rodata_test.o
 obj-$(CONFIG_DEBUG_VM_PGTABLE) += debug_vm_pgtable.o
 obj-$(CONFIG_PAGE_OWNER) += page_owner.o
+ifdef CONFIG_BPF_SYSCALL
+obj-$(CONFIG_PAGE_OWNER) += page_owner_iter.o
+endif
 obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o
 obj-$(CONFIG_ZSMALLOC)	+= zsmalloc.o
 obj-$(CONFIG_GENERIC_EARLY_IOREMAP) += early_ioremap.o
mm/page_owner.c +7 -25
index 8e72a85c663c..ece3b69a2579 100644
@@ -14,6 +14,7 @@
 #include <linux/sched/clock.h>
 
 #include "page_alloc.h"
+#include "page_owner.h"
 
 /*
  * TODO: teach PAGE_OWNER_STACK_DEPTH (__dump_page_owner and save_stack)
@@ -21,30 +22,6 @@
  */
 #define PAGE_OWNER_STACK_DEPTH (16)
 
-struct page_owner {
-	unsigned short order;
-	short last_migrate_reason;
-	gfp_t gfp_mask;
-	depot_stack_handle_t handle;
-	depot_stack_handle_t free_handle;
-	u64 ts_nsec;
-	u64 free_ts_nsec;
-	char comm[TASK_COMM_LEN];
-	pid_t pid;
-	pid_t tgid;
-	pid_t free_pid;
-	pid_t free_tgid;
-};
-
-/*
- * Cursor and per-page result of page_owner_next_eligible().
- */
-struct page_owner_scan {
-	/* resume point */
-	unsigned long pfn;
-	struct page_owner po_snap;
-};
-
 struct stack {
 	struct stack_record *stack_record;
 	struct stack *next;
@@ -740,7 +717,7 @@ void __dump_page_owner(const struct page *page)
  * must advance scan->pfn past the hit before calling again to resume
  * the scan.
  */
-static bool page_owner_next_eligible(struct page_owner_scan *scan)
+bool page_owner_next_eligible(struct page_owner_scan *scan)
 {
 	unsigned long pfn = scan->pfn;
 
@@ -1221,6 +1198,11 @@ static int __init pageowner_init(void)
 			    &stack_fops);
 	debugfs_create_file("count_threshold", 0600, dir, NULL,
 			    &threshold_fops);
+
+#ifdef CONFIG_BPF_SYSCALL
+	if (page_owner_iter_register())
+		pr_warn("page_owner: failed to register bpf_iter target\n");
+#endif
 	return 0;
 }
 late_initcall(pageowner_init)
mm/page_owner.h new file +47 -0

47 lines not on this page · raw message

mm/page_owner_iter.c new file +118 -0

118 lines not on this page · raw message

Signature · 1 line
2.20.1
#6 lore raw

Re: [PATCH v3 3/7] mm/page_owner: add bpf_iter target "page_owner"

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] mm/page_owner: rescan and post-stop show() in page_owner_seq_start() at EOF --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] mm/page_owner: rescan and post-stop show() in page_owner_seq_start() at EOF --

commit a6ba125169596a18417440d179f83a946227810e
Author: Zhen Ni <[email protected]>

mm/page_owner: add bpf_iter target "page_owner"

This commit adds a "page_owner" bpf_iter target that iterates over pages eligible for page_owner output using page_owner_next_eligible() and passes the PFN and a snapshot of struct page_owner to the attached BPF program.

Quoted 6 lines
mm/page_owner_iter.c new file +0 -0
new file mode 100644
index 0000000000000..529b2c4a589a9
@@ -0,0 +1,118 @@

[ ... ]

Quoted 13 lines
mm/page_owner_iter.c new file +13 -0
+static void *page_owner_seq_start(struct seq_file *seq, loff_t *posp)
+{
+	struct page_owner_scan *scan = seq->private;
+
+	if (*posp == 0)
+		scan->pfn = min_low_pfn;
+
+	if (!page_owner_next_eligible(scan))
+		return NULL;
+	return &scan->po_snap;
+}
+
+static void *page_owner_seq_next(struct seq_file *seq, void *v, loff_t *posp)

mm/page_owner_iter.c line 41

Quoted 10 lines
mm/page_owner_iter.c new file +10 -0
+{
+	struct page_owner_scan *scan = seq->private;
+
+	++*posp;
+	scan->pfn++;
+	if (!page_owner_next_eligible(scan))
+		return NULL;
+
+	return &scan->po_snap;
+}

[Severity: Medium]
Can this rescan PFNs and invoke page_owner_seq_show() again after the final page_owner_seq_stop() callback has already run?

In mm/page_owner.c:page_owner_next_eligible(), scan->pfn is copied into a local variable and only written back inside the loop before checking scan->po_snap.handle:

mm/page_owner.c:page_owner_next_eligible() {
	unsigned long pfn = scan->pfn;
	...
	for (; pfn < max_pfn; pfn++) {
		...
		scan->pfn = pfn;
		scan->po_snap = *page_owner;

		if (!scan->po_snap.handle)
			goto ext_put_continue;
		...
		return true;
		...
	}

	return false;
}

When page_owner_seq_next() advances scan->pfn past the last eligible page and calls page_owner_next_eligible(scan), the loop reaches max_pfn and returns false without updating scan->pfn to max_pfn. As a result, scan->pfn is left at the PFN right after the last hit (or at an intermediate PFN where po_snap.handle was zero).

In kernel/bpf/bpf_iter.c:bpf_seq_read(), when page_owner_seq_next() returns NULL, bpf_seq_read() calls page_owner_seq_stop(seq, NULL) (running the BPF program with ctx.po == NULL), marks iter_priv->done_stop = true, and flushes any buffered output in seq->buf to userspace with a positive return value.

When userspace issues the next read() call to observe EOF, bpf_seq_read() calls page_owner_seq_start(), which calls page_owner_next_eligible(scan) and rescans from the stale scan->pfn up to max_pfn. If any page in that range was allocated between the two read() calls, page_owner_next_eligible(scan) returns true and page_owner_seq_show() invokes the BPF program again with ctx.po != NULL after the terminal ctx.po == NULL stop callback has already run.

Similarly, if the initial scan finds no eligible pages (*posp remains 0) and the BPF program writes output during page_owner_seq_stop(seq, NULL), the follow-up read() enters page_owner_seq_start() with *posp == 0, resets scan->pfn = min_low_pfn, and rescans the entire PFN range.

Should page_owner_next_eligible() update scan->pfn = pfn when the loop terminates at max_pfn (and only store scan->pfn = pfn on a hit after checking scan->po_snap.handle), and should page_owner_seq_start() also ensure *posp is advanced so a follow-up read() after an empty scan does not reset scan->pfn to min_low_pfn?

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author
#7 lore raw

PATCH v3 4/7 mm/page_owner: add open-coded page_owner iterator

In reply to Zhen Ni

Add an open-coded bpf_iter over the pages eligible for page_owner output, reusing page_owner_next_eligible().

A sleepable BPF program drives it with bpf_iter_page_owner_new()/next()/destroy(); each next() returns the page_owner_scan. po_snap is copied inside the page_ext RCU window, so the program needs no RCU section when it reads it.

Also add bpf_page_owner_stack_snprint(), which renders the stack trace of the current page into a caller-provided buffer.

---
Changes in v3:
- Gate the kfuncs with KF_PERFMON.
- Check the page_owner_inited static key in the iterator.
- Without scan->page, start at min_low_pfn - 1 and let next() pre-increment.
- bpf_page_owner_stack_snprint() rejects a NULL or zero-size buffer.

Changes in v2:
- bpf_page_owner_stack_snprint() now takes the scan cursor.
- Buffer args renamed to buf__uninit/buf__sz.

v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/

---
 kernel/bpf/helpers.c |  9 +++++++
 mm/page_owner.h      |  3 +++
 mm/page_owner_iter.c | 64 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)
kernel/bpf/helpers.c +9 -0
index 712dca5a2c5b..0ba836f77957 100644
@@ -4946,6 +4946,15 @@ BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
 BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
 BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
 #endif
+#ifdef CONFIG_PAGE_OWNER
+BTF_ID_FLAGS(func, bpf_iter_page_owner_new,
+	     KF_ITER_NEW | KF_SLEEPABLE | KF_PERFMON)
+BTF_ID_FLAGS(func, bpf_iter_page_owner_next,
+	     KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE | KF_PERFMON)
+BTF_ID_FLAGS(func, bpf_iter_page_owner_destroy,
+	     KF_ITER_DESTROY | KF_SLEEPABLE | KF_PERFMON)
+BTF_ID_FLAGS(func, bpf_page_owner_stack_snprint, KF_SLEEPABLE | KF_PERFMON)
+#endif
 BTF_ID_FLAGS(func, __bpf_trap)
 BTF_ID_FLAGS(func, bpf_strcmp, KF_PERFMON);
 BTF_ID_FLAGS(func, bpf_strcasecmp, KF_PERFMON);
mm/page_owner.h +3 -0
index 0aa7695bec82..4ab513f17fc9 100644
@@ -6,6 +6,7 @@
 #ifndef __MM_PAGE_OWNER_H
 #define __MM_PAGE_OWNER_H
 
+#include <linux/jump_label.h>
 #include <linux/page_ext.h>
 #include <linux/sched.h>
 #include <linux/stackdepot.h>
@@ -13,6 +14,8 @@
 /*
  * mm/page_owner.c
  */
+DECLARE_STATIC_KEY_FALSE(page_owner_inited);
+
 struct page_owner {
 	unsigned short order;
 	short last_migrate_reason;
mm/page_owner_iter.c +64 -0

73 lines not on this page · raw message

Signature · 1 line
2.20.1
#8 lore raw

Re: [PATCH v3 4/7] mm/page_owner: add open-coded page_owner iterator

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] mm/page_owner: forged handle and stack UAF in bpf_page_owner_stack_snprint() - [High]…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] mm/page_owner: forged handle and stack UAF in bpf_page_owner_stack_snprint()
- [High] mm/page_owner: bpf_iter_page_owner_next() violates sticky-NULL contract
- [High] mm/page_owner: buf__uninit in bpf_page_owner_stack_snprint() poisons stack --

commit c92f39f3d2ecdefdc5191d503168d482f4fadf55
Author: Zhen Ni <[email protected]>

mm/page_owner: add open-coded page_owner iterator

This commit introduces an open-coded BPF iterator for page_owner using bpf_iter_page_owner_new(), bpf_iter_page_owner_next(), and bpf_iter_page_owner_destroy(), along with the bpf_page_owner_stack_snprint() kfunc to format the current page's allocation stack trace into a buffer.

Quoted 4 lines
mm/page_owner_iter.c +0 -0
index 529b2c4a589a9..4b9741a2de53c 100644

[ ... ]

Quoted 1 line
mm/page_owner_iter.c +0 -0
@@ -116,3 +117,66 @@ int page_owner_iter_register(void)

[ ... ]

mm/page_owner_iter.c int page_owner_iter_register(void) · line 155

Quoted 13 lines
mm/page_owner_iter.c +13 -0
+__bpf_kfunc struct page_owner_scan *
+bpf_iter_page_owner_next(struct bpf_iter_page_owner *it)
+{
+	struct bpf_iter_page_owner_kern *kit = (void *)it;
+
+	if (!static_branch_unlikely(&page_owner_inited))
+		return NULL;
+
+	cond_resched();
+
+	kit->scan.pfn++;
+	if (!page_owner_next_eligible(&kit->scan))
+		return NULL;

[Severity: High]
In kernel/bpf/verifier.c:process_iter_next_call(), once a KF_ITER_NEXT kfunc returns NULL and transitions the iterator to BPF_ITER_STATE_DRAINED, the verifier assumes any subsequent call on the same iterator also returns NULL and marks BPF_REG_0 as constant scalar 0:

	cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
	__mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);

However, mm/page_owner.c:page_owner_next_eligible() iterates using a local pfn variable and only updates scan->pfn when an allocated page is reached:

	unsigned long pfn = scan->pfn;
	...
	for (; pfn < max_pfn; pfn++) {
		...
		scan->pfn = pfn;
		scan->po_snap = *page_owner;

When the loop reaches max_pfn without finding an eligible page, it returns false while leaving kit->scan.pfn at the previously matched PFN plus one (or min_low_pfn if no page matched).

If a BPF program calls bpf_iter_page_owner_next() again after it has returned NULL, kit->scan.pfn++ advances from that earlier PFN and rescans up to max_pfn. Because the scan is lockless and calls cond_resched(), a concurrent allocation in that tail PFN range (or max_pfn increasing via memory hotplug) can cause bpf_iter_page_owner_next() to return a non-NULL &kit->scan pointer at runtime when the verifier tracks R0 as scalar 0.

Should bpf_iter_page_owner_next() latch the drained state (or set kit->scan.pfn to max_pfn on exhaustion and check it before incrementing) so that subsequent calls cannot rescan or return non-NULL?

mm/page_owner_iter.c int page_owner_iter_register(void) · line 165

Quoted 10 lines
mm/page_owner_iter.c +10 -0
+
+	return &kit->scan;
+}
+
+__bpf_kfunc void bpf_iter_page_owner_destroy(struct bpf_iter_page_owner *it)
+{
+}
+
+__bpf_kfunc int bpf_page_owner_stack_snprint(struct page_owner_scan *scan,
+					     char *buf__uninit, u32 buf__sz)

[Severity: High]
In mm/page_owner_iter.c:bpf_page_owner_stack_snprint(), the buffer argument is annotated as buf__uninit paired with buf__sz.

In kernel/bpf/verifier.c:check_kfunc_args(), the __uninit suffix is only handled for KF_ARG_PTR_TO_DYNPTR, whereas a KF_ARG_PTR_TO_MEM and KF_ARG_MEM_SIZE pair is validated with BPF_READ | BPF_WRITE and raw_mode set to false:

	ret = check_mem_size_reg(env, buff_reg, size_reg, buff_argno, argno,
				 BPF_READ | BPF_WRITE, true, meta, &failure);

At the same time, the static stack liveness pass in record_arg_access() -> bpf_kfunc_stack_access_bytes() checks is_kfunc_arg_uninit() and returns -size for buf__uninit:

	if (is_kfunc_arg_uninit(btf, &args[arg]))
		return -size;

In kernel/bpf/liveness.c:record_stack_access_off(), a negative access_bytes marks all covered stack slots as must_write (dead prior to the call), even though bpf_page_owner_stack_snprint() only writes up to len + 1 bytes. At any state-pruning checkpoint before the call (such as inside a bpf_iter_page_owner_next() loop), __clean_func_state() marks those dead stack slots with STACK_POISON, causing check_mem_size_reg() -> check_stack_range_initialized() to reject the BPF program with -EACCES.

Could buf__uninit be renamed to buf (or changed to an interface where the verifier and stack liveness pass agree on initialization semantics) so that passing a stack buffer in an iterator loop is not rejected?

mm/page_owner_iter.c int page_owner_iter_register(void) · line 174

Quoted 9 lines
mm/page_owner_iter.c +9 -0
+{
+	int len = 0;
+
+	if (!buf__uninit || !buf__sz)
+		return 0;
+
+	if (static_branch_unlikely(&page_owner_inited))
+		len = stack_depot_snprint(scan->po_snap.handle, buf__uninit,
+					  buf__sz, 0);

[Severity: Critical]
In mm/page_owner_iter.c:bpf_page_owner_stack_snprint(), scan->po_snap.handle is passed directly to stack_depot_snprint() -> stack_depot_fetch() -> depot_fetch_stack(). In lib/stackdepot.c:depot_fetch_stack(), the handle is assumed to be a valid stackdepot handle:

	if (pool_index >= pools_num_cached) {
		WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n",
		     pool_index, pools_num_cached, handle);
		return NULL;
	}
	...
	stack = pool + offset;
	if (WARN_ON(!refcount_read(&stack->count)))
		return NULL;

An arbitrary handle value can trigger WARN() on an out-of-range pool_index or zero refcount, or point stack at an arbitrary offset inside stack_pools[pool_index] where a forged stack->size causes stack_trace_snprint() to read out of bounds past the end of the pool.

There are two ways a BPF program can supply an arbitrary scan->po_snap.handle and corrupt or retain stack memory:

1. Because struct page_owner_scan and struct page_owner in mm/page_owner.h contain only scalar fields and char comm[TASK_COMM_LEN], __btf_type_is_scalar_struct() returns true. In kernel/bpf/verifier.c:check_kfunc_args(), when a non-BTF pointer such as PTR_TO_STACK or PTR_TO_MAP_VALUE is passed as struct page_owner_scan *scan, the verifier falls back to fixed-size memory and accepts arbitrary BPF-controlled bytes:

	if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
		...
		return -EINVAL;
	}
	kf_arg_type = KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE;
	fallthrough;

2. In mm/page_owner_iter.c:bpf_iter_page_owner_next(), the returned &kit->scan pointer aliases the on-stack struct bpf_iter_page_owner slots as PTR_TO_BTF_ID | PTR_TRUSTED. In kernel/bpf/verifier.c:check_kfunc_call(), regs[BPF_REG_0].parent_id = meta.ref_obj.id is only set in the !__btf_type_is_struct(ptr_type) branch, not in the PTR_TO_BTF_ID branch:

		if (meta.ref_obj.id) {
			err = validate_ref_obj(env, &meta.ref_obj);
			if (err)
				return err;
			regs[BPF_REG_0].parent_id = meta.ref_obj.id;
		}

		if (is_kfunc_rcu_protected(&meta))
			regs[BPF_REG_0].type |= MEM_RCU;
	} else {
		enum bpf_reg_type type = PTR_TO_BTF_ID;

Because parent_id remains 0 on the returned scan pointer, calling bpf_iter_page_owner_destroy() marks the iterator stack slots STACK_INVALID without invalidating registers holding scan. A BPF program can then reuse those stack slots (spilling pointers or writing arbitrary scalars) and read them back through scan or pass the overwritten scan->po_snap.handle to bpf_page_owner_stack_snprint().

Can this be avoided by keeping the stackdepot handle internal to the opaque struct bpf_iter_page_owner state (having bpf_page_owner_stack_snprint() take struct bpf_iter_page_owner *it) and tying the lifetime of the pointer returned by bpf_iter_page_owner_next() to the iterator's ref_obj_id?

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author
#9 lore raw

PATCH v3 5/7 mm/page_owner: record the NUMA node in the scan cursor

In reply to Zhen Ni

Compute the page's NUMA node in page_owner_next_eligible() while the page is still valid, and store it in scan->nid.

The debugfs nid filter reads scan->nid instead of recomputing it, and a BPF program iterating the cursor reads the node straight from scan->nid.

Resolve nid at scan time to avoid dereferencing a possibly-unmapped page. Keep it in the transient scan cursor rather than struct page_owner, so nid does not add per-page memory overhead.

---
Changes in v3:
- Add @scan->nid to the comment describing what is filled.

Changes in v2:
- Removes the v1 get_nid() kfunc. The node is now stored in scan->nid.

v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/

---
 mm/page_owner.c      | 29 +++++++++++++++++------------
 mm/page_owner.h      |  1 +
 mm/page_owner_iter.c |  4 +++-
 3 files changed, 21 insertions(+), 13 deletions(-)
mm/page_owner.c +17 -12
index ece3b69a2579..8bc15bcdc24e 100644
@@ -711,9 +711,22 @@ void __dump_page_owner(const struct page *page)
 	page_ext_put(page_ext);
 }
 
+/*
+ * NUMA node of @page, bypassing PF_POISONED_CHECK() in page_to_nid() to
+ * avoid VM_BUG_ON on poisoned pages; -1 for poisoned pages.
+ */
+static int page_owner_get_nid(struct page *page)
+{
+	memdesc_flags_t page_flags = READ_ONCE(page->flags);
+
+	if (page_flags.f == PAGE_POISON_PATTERN)
+		return -1;
+	return memdesc_nid(&page_flags);
+}
+
 /*
  * Advance @scan to the next page eligible for page_owner output.
- * On a hit, fill scan->pfn/po_snap and return true; the caller
+ * On a hit, fill scan->pfn/po_snap/nid and return true; the caller
  * must advance scan->pfn past the hit before calling again to resume
  * the scan.
  */
@@ -790,6 +803,7 @@ bool page_owner_next_eligible(struct page_owner_scan *scan)
 		if (!scan->po_snap.handle)
 			goto ext_put_continue;
 
+		scan->nid = page_owner_get_nid(page);
 		page_ext_put(page_ext);
 		put_online_mems();
 		return true;
@@ -833,17 +847,8 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos)
 			goto skip_continue;
 
 		if (state->nid_filter_enabled) {
-			int nid;
-			memdesc_flags_t page_flags = READ_ONCE(page->flags);
-
-			/*
-			 * Bypass PF_POISONED_CHECK() in page_to_nid() to avoid
-			 * VM_BUG_ON when accessing poisoned pages.
-			 */
-			if (page_flags.f == PAGE_POISON_PATTERN)
-				goto skip_continue;
-			nid = memdesc_nid(&page_flags);
-			if (!node_isset(nid, state->nid_filter))
+			if (scan.nid < 0 ||
+			    !node_isset(scan.nid, state->nid_filter))
 				goto skip_continue;
 		}
 
mm/page_owner.h +1 -0
index 4ab513f17fc9..934bc9f84d6f 100644
@@ -38,6 +38,7 @@ struct page_owner_scan {
 	/* resume point */
 	unsigned long pfn;
 	struct page_owner po_snap;
+	int nid;
 };
 
 bool page_owner_next_eligible(struct page_owner_scan *scan);
mm/page_owner_iter.c +3 -1
index 4b9741a2de53..b7e0ce1dfa80 100644
@@ -12,10 +12,11 @@ struct bpf_iter__page_owner {
 	__bpf_md_ptr(struct bpf_iter_meta *, meta);
 	__bpf_md_ptr(struct page_owner *, po);
 	u64 pfn;
+	int nid;
 };
 
 DEFINE_BPF_ITER_FUNC(page_owner, struct bpf_iter_meta *meta,
-		     struct page_owner *po, u64 pfn)
+		     struct page_owner *po, u64 pfn, int nid)
 
 static void *page_owner_seq_start(struct seq_file *seq, loff_t *posp)
 {
@@ -52,6 +53,7 @@ static int page_owner_prog_seq_show(struct bpf_prog *prog,
 		/* @po: hit-time snapshot, no RCU window needed. */
 		ctx.po = &scan->po_snap;
 		ctx.pfn = scan->pfn;
+		ctx.nid = scan->nid;
 	}
 
 	return bpf_iter_run_prog(prog, &ctx);
Signature · 1 line
2.20.1
author
#10 lore raw

PATCH v3 6/7 mm/page_owner: add bpf_page_owner_get_memcg_info() kfunc

In reply to Zhen Ni

Expose per-page memcg information to BPF programs, so that a bpf_iter over page_owner can aggregate and filter allocations by memcg. Attributing an allocation to a specific cgroup needs its full cgroup path, not just the memcg name.

Split memcg info collection out of print_page_owner_memcg() into get_page_memcg_info(), which fills struct memcg_info and, when given a caller-provided buffer, writes the full cgroup path via cgroup_path(). print_page_owner_memcg() now consumes the struct; this introduces no functional change to the debugfs output.

The helper takes a single READ_ONCE snapshot within one RCU read-side critical section, so the flags and cgroup path reflect a consistent view of the page. cgroup_path() returns the path relative to the cgroup hierarchy root, so memcg attribution works on both cgroup v1 and v2.

Because scan->pfn is selected by a lockless scan, the kfunc re-resolves it with pfn_to_online_page() under get_online_mems() to avoid accessing a hot-removed section. If the PFN is offline or removed, the kfunc returns -ENODATA instead of dereferencing memcg_data.

---
Changes in v3:
- Check the page_owner_inited static key.
- Resolve the page with pfn_to_online_page() under get_online_mems().

Changes in v2:
- bpf_page_owner_get_memcg_info() now resolves the page with pfn_to_online_page() for an offline or removed pfn.
- Buffer/info args renamed to info__uninit/buf__uninit/buf__sz.

v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/

---
 kernel/bpf/helpers.c |  1 +
 mm/page_owner.c      | 85 ++++++++++++++++++++++++++++++++++----------
 mm/page_owner.h      |  9 +++++
 mm/page_owner_iter.c | 48 +++++++++++++++++++++++++
 4 files changed, 124 insertions(+), 19 deletions(-)
kernel/bpf/helpers.c +1 -0
index 0ba836f77957..bba9eeb076c3 100644
@@ -4954,6 +4954,7 @@ BTF_ID_FLAGS(func, bpf_iter_page_owner_next,
 BTF_ID_FLAGS(func, bpf_iter_page_owner_destroy,
 	     KF_ITER_DESTROY | KF_SLEEPABLE | KF_PERFMON)
 BTF_ID_FLAGS(func, bpf_page_owner_stack_snprint, KF_SLEEPABLE | KF_PERFMON)
+BTF_ID_FLAGS(func, bpf_page_owner_get_memcg_info, KF_SLEEPABLE | KF_PERFMON)
 #endif
 BTF_ID_FLAGS(func, __bpf_trap)
 BTF_ID_FLAGS(func, bpf_strcmp, KF_PERFMON);
mm/page_owner.c +66 -19

141 lines not on this page · raw message

mm/page_owner.h +9 -0
index 934bc9f84d6f..65598415af73 100644
@@ -41,7 +41,16 @@ struct page_owner_scan {
 	int nid;
 };
 
+struct memcg_info {
+	char name[80];
+	bool is_slab;
+	bool is_kmem;
+	bool is_online;
+};
+
 bool page_owner_next_eligible(struct page_owner_scan *scan);
+int get_page_memcg_info(struct page *page, struct memcg_info *info,
+			char *buf, int size);
 
 /*
  * mm/page_owner_iter.c
mm/page_owner_iter.c +48 -0

52 lines not on this page · raw message

Signature · 1 line
2.20.1
#11 lore raw

Re: [PATCH v3 6/7] mm/page_owner: add bpf_page_owner_get_memcg_info() kfunc

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] mm/page_owner: tail page _nr_pages dereferenced as objcg in get_page_memcg_info() - [Medium]…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/page_owner: tail page _nr_pages dereferenced as objcg in get_page_memcg_info()
- [Medium] mm/page_owner: __uninit in bpf_page_owner_get_memcg_info() poisons stack
- [Medium] mm/page_owner: missing __nullable on buf in bpf_page_owner_get_memcg_info() --

commit 6c73498afe7c643ebd28b236a8d852daad168a14
Author: Zhen Ni <[email protected]>

mm/page_owner: add bpf_page_owner_get_memcg_info() kfunc

This commit factors per-page memcg lookup out of print_page_owner_memcg() into get_page_memcg_info() and adds the bpf_page_owner_get_memcg_info() kfunc so BPF page_owner iterators can retrieve memcg metadata and the full cgroup path.

[ ... ]

Quoted 38 lines
mm/page_owner.c +11 -7
index 8bc15bcdc24e9..7470b5d002be4 100644
@@ -538,16 +538,22 @@ void pagetypeinfo_showmixedcount_print(struct seq_file *m,
 
 #ifdef CONFIG_MEMCG
 /*
- * Looking for memcg information and print it out
+ * Collect memcg information of @page into @info; when @buf and a
+ * nonzero @size are given, also copy the full cgroup path of the
+ * charged memcg into @buf.
+ *
+ * Return: 0 if @page is charged to an attributable memcg; -ENODATA
+ * otherwise.
  */
-static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
-					 struct page *page)
+int get_page_memcg_info(struct page *page, struct memcg_info *info,
+			char *buf, int size)
 {
 	unsigned long memcg_data;
 	struct obj_cgroup *objcg;
 	struct mem_cgroup *memcg;
-	bool online;
-	char name[80];
+	int ret = -ENODATA;
+
+	*info = (struct memcg_info){};
 
 	rcu_read_lock();
 	memcg_data = READ_ONCE(page->memcg_data);
@@ -555,31 +561,68 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
 		goto out_unlock;
 
 	if (memcg_data & MEMCG_DATA_OBJEXTS) {
-		ret += scnprintf(kbuf + ret, count - ret,
-				"Slab cache page\n");

mm/page_owner.c static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, · line 572

Quoted 10 lines
mm/page_owner.c +5 -1
+		info->is_slab = true;
 		goto out_unlock;
 	}
 
 	objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK);
-	memcg = objcg ? obj_cgroup_memcg(objcg) : NULL;
+	if (!objcg)
+		goto out_unlock;
+
+	memcg = obj_cgroup_memcg(objcg);

[Severity: High]
Can get_page_memcg_info() in mm/page_owner.c dereference folio->_nr_pages as a struct obj_cgroup pointer when inspecting tail page 1 of a compound folio?

In struct folio (include/linux/mm_types.h), the first tail page (__page_1) places _nr_pages at the same offset as struct page::memcg_data:

include/linux/mm_types.h:struct folio {
    ...
			atomic_t _mapcount_1;
			atomic_t _refcount_1;
#ifdef NR_PAGES_IN_LARGE_FOLIO
			unsigned int _nr_pages;
#endif
		};
		struct page __page_1;
    ...
}

For a compound folio with order >= 3, _nr_pages is 1U << order (for example, 0x8 or 0x200), so its low bits matching OBJEXTS_FLAGS_MASK are zero.

Because bpf_page_owner_get_memcg_info() in mm/page_owner_iter.c calls get_page_memcg_info() on scan->pfn without stabilizing the page reference or folio state (and struct page_owner_scan is a scalar-only struct that a BPF program can also populate on the stack or in a map), page can refer to tail page 1 of a compound folio.

In get_page_memcg_info(), page->memcg_data is read before checking PageTail(page):

	memcg_data = READ_ONCE(page->memcg_data);
	if (!memcg_data || PageTail(page))
		goto out_unlock;

If tail page 1 is concurrently freed via __free_pages_prepare() -> free_tail_page_prepare() -> clear_compound_head() after READ_ONCE() reads _nr_pages into memcg_data, or if the folio is concurrently split to order 0 in mm/huge_memory.c:__split_folio_to_order() where clear_compound_head() clears PageTail(new_head) before new_folio->memcg_data overwrites _nr_pages:

mm/huge_memory.c:__split_folio_to_order() {
    ...
		clear_compound_head(new_head);
		if (new_order) {
			prep_compound_page(new_head, new_order);
			folio_set_large_rmappable(new_folio);
		}
    ...
#ifdef CONFIG_MEMCG
		new_folio->memcg_data = folio->memcg_data;

#endif
}

wouldn't PageTail(page) evaluate to false while memcg_data holds 1UL << order, causing obj_cgroup_memcg(objcg) to dereference that page count as a pointer?

[ ... ]

Quoted 22 lines
mm/page_owner_iter.c +14 -0
index b7e0ce1dfa80a..a649e1207e246 100644
@@ -181,4 +181,52 @@ __bpf_kfunc int bpf_page_owner_stack_snprint(struct page_owner_scan *scan,
 	return len;
 }
 
+/**
+ * bpf_page_owner_get_memcg_info - Report the memcg charged for the current page
+ * @scan: page_owner scan handle.
+ * @info__uninit: output buffer for the memcg_info.
+ * @buf__uninit: optional output buffer for the full cgroup path.
+ * @buf__sz: size of @buf__uninit.
+ *
+ * Fills @info__uninit with the memcg charged for the page backing
+ * @scan->pfn.  If @buf__uninit and @buf__sz are provided, the full
+ * cgroup path is also copied out, so a BPF program can attribute a
+ * page_owner allocation to a specific cgroup.
+ *
+ * @scan->pfn is resolved to a struct page under get_online_mems(),
+ * so the page cannot be accessed after its memory section is offlined

mm/page_owner_iter.c __bpf_kfunc int bpf_page_owner_stack_snprint(struct page_owner_scan *scan, · line 207

Quoted 10 lines
mm/page_owner_iter.c +10 -0
+ * or removed.
+ *
+ * Returns:
+ * * 0 on success
+ * * -ENODATA if the page is not online or not charged to a memcg
+ * * -EOPNOTSUPP if built without CONFIG_MEMCG
+ */
+__bpf_kfunc int bpf_page_owner_get_memcg_info(struct page_owner_scan *scan,
+					      struct memcg_info *info__uninit,
+					      char *buf__uninit, u32 buf__sz)

[Severity: Medium]
Will the __uninit suffix on info__uninit and buf__uninit in bpf_page_owner_get_memcg_info() (mm/page_owner_iter.c) cause the BPF verifier to reject valid programs that pass stack buffers across a pruning checkpoint?

In kernel/bpf/verifier.c:bpf_kfunc_stack_access_bytes(), any kfunc argument matching is_kfunc_arg_uninit() returns a negative size:

	if (is_kfunc_arg_uninit(btf, &args[arg]))
		return -size;
	return size;

That negative size tells record_stack_access_off() in kernel/bpf/liveness.c that the kfunc call overwrites the stack slots without reading them, so static stack liveness analysis marks those slots dead prior to the call. When clean_verifier_state() -> __clean_func_state() in kernel/bpf/states.c runs at an earlier pruning checkpoint (such as the second iteration of bpf_iter_page_owner_next(), or a branch/call after stack allocation), it poisons those dead stack slots with STACK_POISON:

	for (j = start; j < end; j++)
		st->stack[i].slot_type[j] = STACK_POISON;

However, check_kfunc_args() in kernel/bpf/verifier.c only honors is_kfunc_arg_uninit() for KF_ARG_PTR_TO_DYNPTR. For KF_ARG_PTR_TO_MEM (info__uninit) and KF_ARG_MEM_SIZE (buf__uninit), check_kfunc_args() validates the buffers with BPF_READ | BPF_WRITE:

	ret = check_mem_reg(env, reg, argno, type_size, BPF_READ | BPF_WRITE,
			    meta, &known_memory);
    ...
	ret = check_mem_size_reg(env, buff_reg, size_reg, buff_argno, argno,
				 BPF_READ | BPF_WRITE, true, meta, &failure);

When check_stack_range_initialized() in kernel/bpf/verifier.c performs the BPF_READ check with allow_poison == false, it hits *stype == STACK_POISON and fails verification with -EACCES ("slot poisoned by dead code elimination").

Quoted 11 lines
mm/page_owner_iter.c +11 -0
+{
+	struct page *page;
+	int ret = -ENODATA;
+
+	if (!static_branch_unlikely(&page_owner_inited))
+		goto nodata;
+
+	get_online_mems();
+	page = pfn_to_online_page(scan->pfn);
+	if (!page) {
+		put_online_mems();

mm/page_owner_iter.c __bpf_kfunc int bpf_page_owner_stack_snprint(struct page_owner_scan *scan, · line 228

Quoted 10 lines
mm/page_owner_iter.c +10 -0
+		goto nodata;
+	}
+	ret = get_page_memcg_info(page, info__uninit, buf__uninit, buf__sz);
+	put_online_mems();
+	return ret;
+
+nodata:
+	*info__uninit = (struct memcg_info){};
+	if (buf__uninit && buf__sz)
+		buf__uninit[0] = '\0';

[Severity: Medium]
Should the optional cgroup path buffer parameter in bpf_page_owner_get_memcg_info() (mm/page_owner_iter.c) use a __nullable suffix so BPF programs can pass NULL when they only need struct memcg_info?

Both bpf_page_owner_get_memcg_info() and get_page_memcg_info() check whether buf is NULL, and the kernel-doc describes buf__uninit as an optional output buffer. However, in kernel/bpf/verifier.c:get_kfunc_arg_type(), PTR_MAYBE_NULL is only added when is_kfunc_arg_nullable() matches the "__nullable" (or "__arena") suffix:

static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg)

{
	return btf_param_match_suffix(btf, arg, "__nullable") ||
	       btf_param_match_suffix(btf, arg, "__arena");
}

Without a __nullable suffix on the buffer argument, check_kfunc_args() in kernel/bpf/verifier.c rejects a BPF program that passes NULL (or a PTR_MAYBE_NULL register) for arg#3 with -EACCES ("Possibly NULL pointer passed to trusted arg#3").

Quoted 4 lines
mm/page_owner_iter.c +3 -0
+	return ret;
+}
+
 __bpf_kfunc_end_defs();
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author
#12 lore raw

PATCH v3 7/7 selftests/bpf: add tests for the page_owner bpf_iter target

In reply to Zhen Ni

Add a sleepable BPF program and a userspace runner covering the seq_file and open-coded page_owner iterators. The test checks:

- seq_file walk, unfiltered and pid-filtered: records read back from the iterator fd match the lines the program printed.
- open-coded iterator parity with the seq path.
- the kfuncs' contracts: a non-empty stack trace, and memcg classification.

CONFIG_PAGE_OWNER is enabled in the bpf selftests config so the program's page_owner BTF types are available. Skipped when debugfs page_owner is unavailable.

---
Changes in v3:
- Check the ENODATA/EOPNOTSUPP instead of local macros.

Changes in v2:
- Drop the get_nid kfunc coverage.
- Enable CONFIG_PAGE_OWNER in the bpf selftests config.

v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/

---
 tools/testing/selftests/bpf/config            |   2 +
 .../bpf/prog_tests/page_owner_iter.c          | 193 ++++++++++++++++++
 .../selftests/bpf/progs/bpf_iter_page_owner.c | 157 ++++++++++++++
 3 files changed, 352 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/page_owner_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_iter_page_owner.c
tools/testing/selftests/bpf/config +2 -0
index ea7044f30adc..731bc3d79881 100644
@@ -136,3 +136,5 @@ CONFIG_SMC_HS_CTRL_BPF=y
 CONFIG_DIBS=y
 CONFIG_DIBS_LO=y
 CONFIG_PM_WAKELOCKS=y
+CONFIG_DEBUG_KERNEL=y
+CONFIG_PAGE_OWNER=y
tools/testing/selftests/bpf/prog_tests/page_owner_iter.c new file +193 -0

193 lines not on this page · raw message

tools/testing/selftests/bpf/progs/bpf_iter_page_owner.c new file +157 -0

157 lines not on this page · raw message

Signature · 1 line
2.20.1
#13 lore raw

Re: [PATCH v3 7/7] selftests/bpf: add tests for the page_owner bpf_iter target

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] selftests/bpf: ASSERT_OK_PTR() on strstr() and strchr() passes on NULL --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] selftests/bpf: ASSERT_OK_PTR() on strstr() and strchr() passes on NULL --

commit c911a68fd56ba0ae206e2cf17e238a8881f03bd2
Author: Zhen Ni <[email protected]>

selftests/bpf: add tests for the page_owner bpf_iter target

This commit adds a sleepable BPF program and a userspace test runner to exercise the seq_file and open-coded page_owner BPF iterators and their associated kfuncs. It also enables CONFIG_PAGE_OWNER in the BPF selftests configuration.

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/page_owner_iter.c new file +0 -0
new file mode 100644
index 0000000000000..3684ca3089355

[ ... ]

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/page_owner_iter.c new file +3 -0
+/* Validate bpf_page_owner_get_memcg_info() results from the open-coded run. */
+static void check_memcg_skel(struct bpf_iter_page_owner *skel)
+{

[ ... ]

Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/page_owner_iter.c new file +8 -0
+	if (skel->bss->memcg_first_name[0])
+		ASSERT_OK_PTR(strstr(skel->bss->memcg_first_path,
+				    skel->bss->memcg_first_name),
+			      "memcg_name_in_path");
+}
+
+void serial_test_page_owner_iter(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/page_owner_iter.c line 187

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/page_owner_iter.c new file +3 -0
+	ASSERT_GT(skel->bss->stack_bytes, 0, "stack_snprint_ok");
+	ASSERT_OK_PTR(strstr(skel->bss->stack_text, "+0x"), "stack_frames");
+	ASSERT_OK_PTR(strchr(skel->bss->stack_text, '\n'), "stack_lines");

[Severity: Medium]
Will these ASSERT_OK_PTR() checks in serial_test_page_owner_iter() and check_memcg_skel() in tools/testing/selftests/bpf/prog_tests/page_owner_iter.c actually catch a NULL return from strstr() or strchr()?

In tools/testing/selftests/bpf/test_progs.h, ASSERT_OK_PTR() checks the pointer via libbpf_get_error():

    #define ASSERT_OK_PTR(ptr, name) ({                                 \
            static int duration = 0;                                    \
            const void *___res = (ptr);                                 \
            int ___err = libbpf_get_error(___res);                      \
            bool ___ok = ___err == 0;                                   \
            CHECK(!___ok, (name), "unexpected error: %d\n", ___err);    \
            ___ok;                                                      \
    })

And in tools/lib/bpf/libbpf.c, libbpf_get_error() returns -errno when given NULL:

    long libbpf_get_error(const void *ptr)
    {
            if (!IS_ERR_OR_NULL(ptr))
                    return 0;

            if (IS_ERR(ptr))
                    errno = -PTR_ERR(ptr);

            return -errno;
    }

Because strstr() and strchr() do not set errno when returning NULL on a non-match, if errno is 0 after bpf_prog_test_run_opts() succeeds, libbpf_get_error(NULL) returns 0 and ASSERT_OK_PTR() silently passes even when "+0x", '\n', or memcg_first_name is missing.

Should these checks use ASSERT_NEQ(..., NULL, ...) instead?

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/page_owner_iter.c new file +6 -0
+
+	check_memcg_skel(skel);
+
+destroy:
+	bpf_iter_page_owner__destroy(skel);
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
maintainer
#14 lore raw
In reply to Zhen Ni
Quoted 5 lines · On Wed, 21 Oct 2026 19:24:30 +0800 Zhen Ni <[email protected]> wrote:

page_owner records who allocated every page and is a core memory-debug facility, but it currently reaches userspace as the debugfs flat dump:
slicing allocations by pid, comm, NUMA node, cgroup or stack, etc., means streaming and parsing the entire file in userspace, which is expensive on large hosts and hard to query live.

Confused. Your series 8570b65220e8 ("mm/page_owner: add print_mode filter") addressed this significantly?

author
#15 lore raw
Quoted 12 lines · Andrew Morton
Quoted 5 lines · On Wed, 21 Oct 2026 19:24:30 +0800 Zhen Ni <[email protected]> wrote:

page_owner records who allocated every page and is a core memory-debug facility, but it currently reaches userspace as the debugfs flat dump:
slicing allocations by pid, comm, NUMA node, cgroup or stack, etc., means streaming and parsing the entire file in userspace, which is expensive on large hosts and hard to query live.

Confused. Your series 8570b65220e8 ("mm/page_owner: add print_mode filter") addressed this significantly?

Thank you. I'd like to share some background on how this new approach came about.

After series 8570b65220e8 ("mm/page_owner: add print_mode filter"), I attempted to further extend the filter capabilities so that userspace could more selectively capture page owner results. This effort led to the series [PATCH v3 0/8] mm/page_owner: Add PID/TGID/COMM and cgroup filtering.

During that discussion, we gradually reached a consensus that "a BPF iter" might be a better solution to this problem. The main concerns with continuing to extend the kernel-side filter approach were:

1. As filter capabilities grow, the kernel side needs to handle increasingly complex parameter parsing and various combinations of different filter attributes, which is not conducive to maintainability.

2. More fundamentally, kernel-side filters cannot provide truly flexible filtering because the filtering logic is hardcoded in the kernel.

At that point, I did a POC to verify the BPF iter direction:

https://lore.kernel.org/linux-mm/[email protected]/

It seems there is broad agreement on this direction. This new series is the implementation of that approach.

Thanks,
Zhen

#16 lore raw
In reply to Zhen Ni
Quoted 4 lines · Zhen Ni

Exposing the same walk to eBPF lets a program filter and aggregate in the kernel and surface only the records it cares about. This series adds that access as a seq_file bpf_iter target and an open-coded iterator.

This is overkill. Two iterators, two kfuncs that print strings, a rewrite of read_page_owner(), a new file and a new header.

All that the prog needs from the kernel is the page_owner record of a pfn. That's one kfunc. The walk, the eligibility rules and the filters belong in the bpf prog:

  bpf_for(i, 0, ctx->nr) {
          pfn = ctx->start + i;
          if (bpf_page_owner_read(pfn, &po) ||
              pfn & ((1ul << po.order) - 1))
                  continue;
          if (po.pid != pid || bpf_page_cgroup_ino(pfn) != cgrp)
                  continue;
          /* add 1 << po.order to a hash map keyed by po.handle */
  }

Something like this:

mm/bpf_memcontrol.c +15 -0
index d8f579c..3adb716 100644
@@ -161,6 +161,20 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg)
 	mem_cgroup_flush_stats(memcg);
 }
 
+/**
+ * bpf_page_cgroup_ino - Get the cgroup a page is charged to
+ * @pfn: the page
+ *
+ * Return: What /proc/kpagecgroup has for @pfn: the inode number of the closest
+ * online ancestor of the memory cgroup, 0 if the page is not charged.
+ */
+__bpf_kfunc u64 bpf_page_cgroup_ino(unsigned long pfn)
+{
+	struct page *page = pfn_to_online_page(pfn);
+
+	return page ? page_cgroup_ino(page) : 0;
+}
+
 /**
  * bpf_proactive_reclaim - proactively reclaim memory from a memory cgroup
  * @memcg: the target memory cgroup to reclaim from.
@@ -218,6 +232,7 @@ BTF_ID_FLAGS(func, bpf_mem_cgroup_memory_events)
 BTF_ID_FLAGS(func, bpf_mem_cgroup_usage)
 BTF_ID_FLAGS(func, bpf_mem_cgroup_page_state)
 BTF_ID_FLAGS(func, bpf_mem_cgroup_flush_stats, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_page_cgroup_ino, KF_PERFMON)
 
 BTF_KFUNCS_END(bpf_memcontrol_kfuncs)
 
mm/page_owner.c +40 -1

58 lines not on this page · raw message

pw-bot: cr

author
#17 lore raw
Quoted 9 lines · Alexei Starovoitov
Quoted 4 lines · Zhen Ni

Exposing the same walk to eBPF lets a program filter and aggregate in the kernel and surface only the records it cares about. This series adds that access as a seq_file bpf_iter target and an open-coded iterator.

This is overkill. Two iterators, two kfuncs that print strings, a rewrite of read_page_owner(), a new file and a new header.

Thanks for the detailed review and the sketch. I agree with the principle that a thin data kfunc would be ideal if it were sufficient, but I’d like to explain why the current design ended up this way.

On the two iterator surfaces:
They aren’t duplicate implementations. Both are thin shells over the single page_owner_next_eligible() extracted from read_page_owner(), so the eligibility rules exist exactly once. There’s no code redundancy; the two iters only differ in access patterns while sharing the same underlying iteration logic. This shape is also what Documentation/bpf/ bpf_iterators.rst prescribes:

When adding a new BPF iterator program, it is expected that similar
functionality will be added as open-coded iterator for maximum
flexibility.
It's also expected that iteration logic and code will be maximally shared and reused between two iterator API surfaces.

On the two “print strings” kfuncs:
These are things a prog genuinely cannot do on its own. The record only carries an opaque stack-depot handle, and resolving it is not reachable from BPF. Likewise, the cgroup path is built by cgroup_path() walking the kernfs parent chain — a prog has no way to format it. Given the 512- byte BPF stack limit, the caller-provides-buffer pattern (as in bpf_d_path() and bpf_snprintf_btf()) is the standard shape for this kind of kfunc. I agree they could have been mentioned earlier rather than left to the diff.

On “a rewrite of read_page_owner()”:
This is really an extraction of common functions, not a rewrite. Aside from the fix for mem-hotplug locking (which addresses a pre-existing bug in the debugfs interface), there are no substantial logic changes. It’s not a large-scale refactoring.

On the new file and header:
This is for code clarity — to distinguish the traditional interface from the BPF interface and make maintenance easier.

Quoted 4 lines

All that the prog needs from the kernel is the page_owner record of a pfn. That's one kfunc. The walk, the eligibility rules and the filters belong in the bpf prog:

Quoted 10 lines
   bpf_for(i, 0, ctx->nr) {
           pfn = ctx->start + i;
           if (bpf_page_owner_read(pfn, &po) ||
               pfn & ((1ul << po.order) - 1))
                   continue;
           if (po.pid != pid || bpf_page_cgroup_ino(pfn) != cgrp)
                   continue;
           /* add 1 << po.order to a hash map keyed by po.handle */
   }

On why the prog can't actually execute it:

1. What range to scan. Where do ctx->start and ctx->nr come from? No in-tree iterator context provides them, and a prog cannot derive them:
min_low_pfn..max_pfn, memblock holes, and offline sections are all kernel state. The loop bounds alone require either another kfunc or a kernel-side walk — which is exactly what the iter target already is.

2. What makes a page eligible. The alignment test in the sketch is the only part the kernel agrees is expressible in a prog. Everything before it reads state a prog doesn’t have a handle to: pfn_valid() over MAX_ORDER_NR_PAGES areas, pfn_to_online_page() under get_online_mems(), skip_buddy_pages(), the PAGE_EXT_OWNER / PAGE_EXT_OWNER_ALLOCATED flags in page_ext, and taking the record snapshot inside the RCU window so the checked value and the reported value are the same copy.

3. The walk is deliberately lockless, so exposing struct page to the prog is not safe. The pfn validity check and the page_ext dereference must happen under the same kernel-controlled RCU window. Otherwise the prog could observe offlined pages or stale page_ext data, breaking consistency between the check and the snapshot. Keeping the walk in the kernel guarantees that the page_owner record is read atomically with respect to mem-hotplug and RCU protection.

4. One record doesn’t give the prog the useful data. The record holds only opaque handle/free_handle tokens; stack-depot resolution exists solely in the kernel. The memcg is neither in the record nor trivially reachable — the path is built by cgroup_path() walking the kernfs parent chain. The NUMA node comes from page_owner_get_nid(), which bypasses page_to_nid()’s poison check. And so on.

Thanks again for pushing on this. I’m still concerned that the sketch may not be workable in practice, and that it would also lose some useful capabilities. I hope the above clarifies the reasoning. Happy to discuss further.

Thanks,
Zhen

#18 lore raw

On Fri, Oct 23, 2026 at 07:51 AM zhen.ni <[email protected]> wrote:

tldr: it's still nack

Quoted 6 lines

underlying iteration logic. This shape is also what Documentation/bpf/ bpf_iterators.rst prescribes:

When adding a new BPF iterator program, it is expected that similar functionality will be added as open-coded iterator for maximum flexibility.

It says what to do when an iterator is added. It doesn't say that one is needed.
Iterators are for kernel data structures that a prog cannot walk on its own: tasks, cgroups, kmem_caches.
pfn is a number. bpf_for() is the iterator.

Quoted 5 lines

1. What range to scan. Where do ctx->start and ctx->nr come from? No in-tree iterator context provides them, and a prog cannot derive them:
min_low_pfn..max_pfn, memblock holes, and offline sections are all kernel state. The loop bounds alone require either another kfunc or a kernel-side walk — which is exactly what the iter target already is.

From user space. It's a syscall prog, like scan_open_coded() in your patch 7. Its ctx is what user space passes in ctx_in. /proc/zoneinfo has start_pfn and spanned for every zone. In holes and offline sections pfn_to_online_page() returns NULL and bpf_page_owner_read() returns -ENOENT.

Quoted 7 lines

2. What makes a page eligible. The alignment test in the sketch is the only part the kernel agrees is expressible in a prog. Everything before it reads state a prog doesn’t have a handle to: pfn_valid() over MAX_ORDER_NR_PAGES areas, pfn_to_online_page() under get_online_mems(), skip_buddy_pages(), the PAGE_EXT_OWNER / PAGE_EXT_OWNER_ALLOCATED flags in page_ext, and taking the record snapshot inside the RCU window so the checked value and the reported value are the same copy.

bpf_page_owner_read() in the diff does all of that:
pfn_to_online_page(), page_ext_get(), PAGE_EXT_OWNER_ALLOCATED check and the copy of the record before page_ext_put(). PAGE_EXT_OWNER is set together with PAGE_EXT_OWNER_ALLOCATED and it's never cleared. No need to check both.
__reset_page_owner() clears PAGE_EXT_OWNER_ALLOCATED, so skip_buddy_pages() and MAX_ORDER_NR_PAGES steps only make the walk faster. They don't change what is eligible.

Quoted 3 lines

3. The walk is deliberately lockless, so exposing struct page to the prog is not safe. The pfn validity check and the page_ext dereference must happen under the same kernel-controlled RCU window. Otherwise the

The prog doesn't see struct page or page_ext. It passes a pfn and gets a copy of the record. The lookup, the check and the copy are in one RCU section inside the kfunc. offline_page_ext() does synchronize_rcu() before freeing page_ext.
page_ext_from_phys() does the same lookup from any context.

Quoted 6 lines

4. One record doesn’t give the prog the useful data. The record holds only opaque handle/free_handle tokens; stack-depot resolution exists solely in the kernel. The memcg is neither in the record nor trivially reachable — the path is built by cgroup_path() walking the kernfs parent chain. The NUMA node comes from page_owner_get_nid(), which bypasses page_to_nid()’s poison check. And so on.

The handle is the key to filter and aggregate by. page_owner_stacks/show_stacks_handles maps handles to stacks. Same for memcg. bpf_page_cgroup_ino() in the diff returns the id. st_ino of the cgroup directory is that id.

author
#19 lore raw
Quoted 11 lines · Alexei Starovoitov

On Fri, Oct 23, 2026 at 07:51 AM zhen.ni <[email protected]> wrote:

tldr: it's still nack

Quoted 6 lines

underlying iteration logic. This shape is also what Documentation/bpf/ bpf_iterators.rst prescribes:

When adding a new BPF iterator program, it is expected that similar functionality will be added as open-coded iterator for maximum flexibility.

Quoted 5 lines

It says what to do when an iterator is added. It doesn't say that one is needed.
Iterators are for kernel data structures that a prog cannot walk on its own: tasks, cgroups, kmem_caches.
pfn is a number. bpf_for() is the iterator.

I agree the doc is prescribing what to do when an iterator is added, not that every case must have one. But my point is that having both iterators coexist is the convention, not overkill.

Page owner traversal looks like iterating over PFNs, but PFN is just a number; what is actually being traversed is page_owner records, buddy blocks, online/ offline sections, memblock holes—kernel state that a prog cannot see on its own. So it is not a pure numeric-range walk.

Quoted 7 lines
Quoted 6 lines

1. What range to scan. Where do ctx->start and ctx->nr come from? No in-tree iterator context provides them, and a prog cannot derive them:
min_low_pfn..max_pfn, memblock holes, and offline sections are all kernel state. The loop bounds alone require either another kfunc or a kernel-side walk — which is exactly what the iter target already is.

Quoted 6 lines

From user space. It's a syscall prog, like scan_open_coded() in your
patch 7. Its ctx is what user space passes in ctx_in. /proc/zoneinfo has start_pfn and spanned for every zone. In holes and offline sections pfn_to_online_page() returns NULL and bpf_page_owner_read() returns -ENOENT.

I agree the range can be obtained. But being able to obtain a range is not the same as that range being meaningful to the user. Page owner users typically care about a process, a cgroup's leaks, or the global distribution; PFN is just a physical address index, and users almost never know in advance which zone or which PFN range to scan. Letting the prog specify a PFN range raises the bar for writing it while the practical use cases remain limited. This is not a feasibility issue—it is a question of abstraction level.

Quoted 8 lines
Quoted 7 lines

2. What makes a page eligible. The alignment test in the sketch is the only part the kernel agrees is expressible in a prog. Everything before it reads state a prog doesn’t have a handle to: pfn_valid() over MAX_ORDER_NR_PAGES areas, pfn_to_online_page() under get_online_mems(), skip_buddy_pages(), the PAGE_EXT_OWNER / PAGE_EXT_OWNER_ALLOCATED flags in page_ext, and taking the record snapshot inside the RCU window so the checked value and the reported value are the same copy.

Quoted 8 lines

bpf_page_owner_read() in the diff does all of that:
pfn_to_online_page(), page_ext_get(), PAGE_EXT_OWNER_ALLOCATED check and the copy of the record before page_ext_put(). PAGE_EXT_OWNER is set together with PAGE_EXT_OWNER_ALLOCATED and it's never cleared. No need to check both.
__reset_page_owner() clears PAGE_EXT_OWNER_ALLOCATED, so skip_buddy_pages() and MAX_ORDER_NR_PAGES steps only make the walk faster. They don't change what is eligible.

I agree on eligibility. But I'd like to expand on "only faster": what those steps change is the scale of the walk, and under SPARSEMEM that scale is structural, not a constant factor. On a 256G, 8-NUMA machine I have, node0 is:

crash> struct pglist_data ffff0106ffffe580 | grep node_spanned_pages
node_spanned_pages = 17235968,
crash> struct pglist_data ffff0106ffffe580 | grep node_present_pages
node_present_pages = 523024,

For this one node alone, about 97% of spanned is holes. With bpf_for(pfn, start, end) plus a bpf_page_owner_read() per iteration, that is at least 16,712,944 kfunc calls, and bpf_for has no scheduling point inside, and all of them are wasted work. The gap grows with memory size, and I'm concerned "only faster" understates the runtime cost on real machines.

One more structural issue: bpf_for is capped at BPF_MAX_LOOPS = 8 * 1024
* 1024, and this is a runtime check—the verifier will not reject it up front. The spanned of this node is already 17.23M, so a single bpf_for cannot cover it; you have to split it into N chunks and poll, with the chunking state kept in a prog map, i.e. a loop nested in a loop.

And cur/end are stored as BPF_W (32-bit), while the PFN upper bound tracks the physical address width: on x86-64 with 46-bit PA, max_pfn can reach 2^34, and 2^40 with LA57; on arm64, 2^36 for 48-bit PA and 2^40 for 52-bit PA. A 32-bit PFN range will overflow here.

Solving all of this requires yet another, more complex kfunc—and it looks increasingly like bpf iter. So why not just use the iter?

Quoted 10 lines
Quoted 4 lines

3. The walk is deliberately lockless, so exposing struct page to the prog is not safe. The pfn validity check and the page_ext dereference must happen under the same kernel-controlled RCU window. Otherwise the

The prog doesn't see struct page or page_ext. It passes a pfn and gets a copy of the record. The lookup, the check and the copy are in one RCU section inside the kfunc. offline_page_ext() does synchronize_rcu() before freeing page_ext.
page_ext_from_phys() does the same lookup from any context.

I have no doubt about feasibility.

Quoted 8 lines
Quoted 7 lines

4. One record doesn’t give the prog the useful data. The record holds only opaque handle/free_handle tokens; stack-depot resolution exists solely in the kernel. The memcg is neither in the record nor trivially reachable — the path is built by cgroup_path() walking the kernfs parent chain. The NUMA node comes from page_owner_get_nid(), which bypasses page_to_nid()’s poison check. And so on.

Quoted 6 lines

The handle is the key to filter and aggregate by. page_owner_stacks/show_stacks_handles maps handles to stacks. Same for memcg. bpf_page_cgroup_ino() in the diff returns the id. st_ino of the cgroup directory is that id.

Similarly, I have no doubt about feasibility. I do have some concerns about data consistency and performance. It also increases prog complexity.

In summary, while I don't dispute the feasibility of the bpf_for + bpf_page_owner_read approach and fully agree that kernel-side simplicity is paramount, I wonder if we should also weigh the complexity on the program side. If using this kfunc imposes a steep learning curve, I'm concerned it might limit adoption. If a feature lacks practical utility, it becomes difficult to justify the effort.

Thanks for the detailed explanations and for walking me through these points.

Thanks,
Zhen

#20 lore raw
Quoted 4 lines · zhen.ni

For this one node alone, about 97% of spanned is holes. With bpf_for(pfn, start, end) plus a bpf_page_owner_read() per iteration, that is at least 16,712,944 kfunc calls, and bpf_for has no scheduling point inside, and all of them are wasted work. The gap grows with memory

Right. spanned includes the holes. So don't pass them. /sys/devices/system/memory/ has memoryN only for present blocks. /sys/devices/system/node/nodeN/ links to the blocks of that node. /proc/iomem has "System RAM" ranges.
523024 pages in a 32G node looks like 64K pages.
Then the whole 256G box is 4M pfns.
When a hole is passed anyway the call ends at !online_section() in pfn_to_online_page(). /proc/kpageflags does that plus put_user() for every pfn, holes included.
Syscall progs run under migrate_disable() only.

Quoted 3 lines

front. The spanned of this node is already 17.23M, so a single bpf_for cannot cover it; you have to split it into N chunks and poll, with the chunking state kept in a prog map, i.e. a loop nested in a loop.

The loop over chunks is in user space. One bpf_prog_test_run_opts() per range with start and nr in ctx_in. No map, no nested loop.

Quoted 1 line

for 52-bit PA. A 32-bit PFN range will overflow here.

The snippet was:

  bpf_for(i, 0, ctx->nr) {
          pfn = ctx->start + i;

i is an offset within the chunk. start and pfn are 64-bit.

Quoted 2 lines

Solving all of this requires yet another, more complex kfunc—and it looks increasingly like bpf iter. So why not just use the iter?

Nothing of the above needs kernel code.
v3 is +474 -83 lines on the kernel side. The diff I sent is +55 -1 and it doesn't touch the debugfs path.

author
#21 lore raw
Quoted 10 lines · Alexei Starovoitov
Quoted 4 lines · zhen.ni

For this one node alone, about 97% of spanned is holes. With bpf_for(pfn, start, end) plus a bpf_page_owner_read() per iteration, that is at least 16,712,944 kfunc calls, and bpf_for has no scheduling point inside, and all of them are wasted work. The gap grows with memory

Right. spanned includes the holes. So don't pass them. /sys/devices/system/memory/ has memoryN only for present blocks. /sys/devices/system/node/nodeN/ links to the blocks of that node. /proc/iomem has "System RAM" ranges.

Thanks for taking the time to review this.

From /sys/devices/system/memory/ you get a discrete list of memory blocks. The prog (or userspace) still has to walk those blocks, handle boundaries, and track progress. That is traversal logic that an iterator already absorbs into a single next() call. Pushing it out just moves the complexity; it doesn't eliminate it.

Therefore, this only demonstrates feasibility, not superiority.

Quoted 5 lines

523024 pages in a 32G node looks like 64K pages.
Then the whole 256G box is 4M pfns.
When a hole is passed anyway the call ends at !online_section() in pfn_to_online_page(). /proc/kpageflags does that plus put_user() for every pfn, holes included.

True, the kernel safely returns an error for offline PFNs. But paying a kfunc call just to learn that a PFN is ineligible is exactly the waste the iterator eliminates — the kernel-side cursor skips holes and ineligible pages before the prog ever sees them.

Quoted 1 line

Syscall progs run under migrate_disable() only.

Agreed, migrate_disable() pins the task to one CPU without disabling preemption on PREEMPT_RT. But being preemptible doesn't mean the loop is free — an 8M-iteration tight loop still consumes CPU, delays RCU grace periods, and piles up softirq work. Skipping ineligible pages up front reduces that pressure.

Quoted 8 lines
Quoted 4 lines

front. The spanned of this node is already 17.23M, so a single bpf_for cannot cover it; you have to split it into N chunks and poll, with the chunking state kept in a prog map, i.e. a loop nested in a loop.

The loop over chunks is in user space. One bpf_prog_test_run_opts() per range with start and nr in ctx_in. No map, no nested loop.

The outer loop being in userspace doesn't remove the chunking burden — it just relocates it. Userspace still has to parse sysfs, construct ranges, issue one syscall per chunk, read the map, accumulate results into a global aggregate, clear the map, and handle partial failures. And in the snippet, i is an offset within the chunk while start and pfn are 64-bit — the chunking arithmetic itself is non-trivial. An iterator absorbs all of this into next().

Quoted 11 lines
Quoted 1 line

for 52-bit PA. A 32-bit PFN range will overflow here.

The snippet was:

   bpf_for(i, 0, ctx->nr) {
           pfn = ctx->start + i;

i is an offset within the chunk. start and pfn are 64-bit.

Quoted 2 lines

Solving all of this requires yet another, more complex kfunc—and it looks increasingly like bpf iter. So why not just use the iter?

Nothing of the above needs kernel code.

Feasibility isn't in question. The question is whether we want every prog writer to reinvent this traversal plumbing, or let the kernel do what iterators are designed for.

Quoted 3 lines

v3 is +474 -83 lines on the kernel side. The diff I sent is +55 -1 and it doesn't touch the debugfs path.

That diff includes my improvements to the page_owner infrastructure, so the line count doesn't really establish that bpf_for is better than iter. The iter approach only adds an exit on the debugfs path — thin shells over the single page_owner_next_eligible() extracted from read_page_owner(). It changes no logic on the debugfs path (aside from the mem-hotplug locking fix, which addresses a pre-existing bug in the debugfs interface).

And the fact that the bpf_for approach doesn't touch the debugfs path is, if anything, a disadvantage: it means two sets of eligibility rules would have to be maintained going forward.

With the iter approach, any future change only needs to be made in page_owner_next_eligible().

Thanks,
Zhen

#22 lore raw
Quoted 3 lines · zhen.ni

Feasibility isn't in question. The question is whether we want every prog writer to reinvent this traversal plumbing, or let the kernel do what iterators are designed for.

What the prog and user space can do on their own stays out of the kernel. The reason for the nack is the same as before.

I'm done with this thread and all of your future patches. It's auto NACK for all of them. Pls carry it forward, so no one in the kernel community wastes time on this anymore.

pw-bot: cr