mm: improve folio refcount scalability

Some call sites manipulate page refcount directly based on its own assumptions of the refcount int value. Instead of making such assumptions these call sites should use high-level API like set_frozen() or init_refcount().

This patch tries to consolidate all page refcount implementation details inside mm headers. The main reason for this is the following patch that will stop using zero value for frozen refcounts.

Refactor external refcount API:
- Introduce init/init_as_frozen functions and replace low-level refcount manipulations with them where applicable.
- Introduce page/folio_is_frozen().
While page_ref_count() will always return zero for frozen pages, those functions allow more efficient (later) and more obvious code [1].
- Rename _unless_zero() into _unless_frozen() to deprecate zero value assumption.

Introduce debug asserts:
- VM_WARN_ON_ONCE() to prevent following scenarios:

page = alloc_frozen_page() /* page is frozen */ page_ref_inc(page, 1) /* BUG: Increment on frozen page instead of init */

[1]: https://lore.kernel.org/all/aqAIFV4nOGPbWiDS@thinkstation/

Reviewed-by Artem Kuzin <[email protected]>

Co-developed-by Ivan Gorbunov <[email protected]>

Signed-off-by Ivan Gorbunov <[email protected]>

Signed-off-by Ilya Gladyshev <[email protected]>

Acked-by Bjorn Helgaas <[email protected]> # p2pdma.c

---
 drivers/pci/p2pdma.c               |  4 +--
 drivers/virtio/virtio_mem.c        |  2 +-
 include/linux/mm.h                 |  2 +-
 include/linux/page_ref.h           | 43 +++++++++++++++++++++++++-----
 kernel/liveupdate/kexec_handover.c |  6 ++---
 lib/test_hmm.c                     |  4 +--
 mm/hugetlb.c                       |  2 +-
 mm/internal.h                      |  2 +-
 mm/memory-failure.c                |  8 +++---
 mm/memremap.c                      |  4 +--
 mm/mm_init.c                       |  6 ++---
 mm/page_alloc.c                    | 10 +++----
 mm/page_frag_cache.c               |  2 +-
 13 files changed, 62 insertions(+), 33 deletions(-)

drivers/pci/p2pdma.c +2 -2

index 9334eb314663..2213214daa89 100644
@@ -148,7 +148,7 @@ static int p2pmem_alloc_mmap(struct file *filp, struct kobject *kobj,
 		 * using it.
 		 */
 		VM_WARN_ON_ONCE_PAGE(page_ref_count(page), page);
-		set_page_count(page, 1);
+		init_page_count(page);
 		ret = vm_insert_page(vma, vaddr, page);
 		if (ret) {
 			gen_pool_free(p2pdma->pool, (uintptr_t)kaddr, len);

@@ -158,7 +158,7 @@ static int p2pmem_alloc_mmap(struct file *filp, struct kobject *kobj,
 			 * because we don't want to trigger the
 			 * p2pdma_folio_free() path.
 			 */
-			set_page_count(page, 0);
+			set_page_count_frozen(page);
 			percpu_ref_put(ref);
 			return ret;
 		}

drivers/virtio/virtio_mem.c +1 -1

index e18dd736f2ec..e282f5e048b9 100644
@@ -1293,7 +1293,7 @@ static void virtio_mem_fake_offline_cancel_offline(unsigned long pfn,
 	 * when going offline.
 	 */
 	for (i = 0; i < nr_pages; i++)
-		page_ref_inc(pfn_to_page(pfn + i));
+		init_page_count(pfn_to_page(pfn + i));
 }
 
 static void virtio_mem_online_page(struct virtio_mem *vm,

include/linux/mm.h +1 -1

index dd09c438fa23..dab5621ad8f7 100644
@@ -1860,7 +1860,7 @@ static inline int folio_put_testzero(struct folio *folio)
  */
 static inline bool get_page_unless_zero(struct page *page)
 {
-	return page_ref_add_unless_zero(page, 1);
+	return page_ref_add_unless_frozen(page, 1);
 }
 
 static inline struct folio *folio_get_nontail_page(struct page *page)

include/linux/page_ref.h +24 -2

index 9f5c75d06f76..82ff3a99297a 100644
@@ -62,6 +62,21 @@ static inline void __page_ref_unfreeze(struct page *page, int v)
 
 #endif
 
+static inline bool __page_count_is_frozen(int count)
+{
+	return count == 0;
+}
+
+static inline bool page_is_frozen(const struct page *page)
+{
+	return __page_count_is_frozen(atomic_read(&page->_refcount));
+}
+
+static inline bool folio_is_frozen(const struct folio *folio)
+{
+	return page_is_frozen(&folio->page);
+}
+
 static inline int page_ref_count(const struct page *page)
 {
 	return atomic_read(&page->_refcount);

@@ -119,9 +134,9 @@ static inline void set_page_count(struct page *page, int v)
 		__page_ref_set(page, v);
 }
 
-static inline void folio_set_count(struct folio *folio, int v)
+static inline void folio_init_count(struct folio *folio)
 {
-	set_page_count(&folio->page, v);
+	set_page_count(&folio->page, 1);
 }
 
 /*

@@ -133,8 +148,14 @@ static inline void init_page_count(struct page *page)
 	set_page_count(page, 1);
 }
 
+static inline void set_page_count_frozen(struct page *page)
+{
+	set_page_count(page, 0);
+}
+
 static inline void page_ref_add(struct page *page, int nr)
 {
+	VM_WARN_ON_ONCE_PAGE(page_is_frozen(page), page);
 	atomic_add(nr, &page->_refcount);
 	if (page_ref_tracepoint_active(page_ref_mod))
 		__page_ref_mod(page, nr);

@@ -147,6 +168,7 @@ static inline void folio_ref_add(struct folio *folio, int nr)
 
 static inline void page_ref_sub(struct page *page, int nr)
 {
+	VM_WARN_ON_ONCE_PAGE(page_is_frozen(page), page);
 	atomic_sub(nr, &page->_refcount);
 	if (page_ref_tracepoint_active(page_ref_mod))
 		__page_ref_mod(page, -nr);

include/linux/page_ref.h +2 -0

@@ -168,6 +191,7 @@ static inline int folio_ref_sub_return(struct folio *folio, int nr)
 
 static inline void page_ref_inc(struct page *page)
 {
+	VM_WARN_ON_ONCE_PAGE(page_is_frozen(page), page);
 	atomic_inc(&page->_refcount);
 	if (page_ref_tracepoint_active(page_ref_mod))
 		__page_ref_mod(page, 1);

@@ -180,6 +204,7 @@ static inline void folio_ref_inc(struct folio *folio)
 
 static inline void page_ref_dec(struct page *page)
 {
+	VM_WARN_ON_ONCE_PAGE(page_is_frozen(page), page);
 	atomic_dec(&page->_refcount);
 	if (page_ref_tracepoint_active(page_ref_mod))
 		__page_ref_mod(page, -1);

include/linux/page_ref.h +1 -0

@@ -207,6 +233,7 @@ static inline int folio_ref_sub_and_test(struct folio *folio, int nr)
 static inline int page_ref_inc_return(struct page *page)
 {
 	int ret = atomic_inc_return(&page->_refcount);
+	VM_WARN_ON_ONCE_PAGE(__page_count_is_frozen(ret - 1), page);
 
 	if (page_ref_tracepoint_active(page_ref_mod_and_return))
 		__page_ref_mod_and_return(page, 1, ret);

include/linux/page_ref.h +6 -5

 	if (page_ref_tracepoint_active(page_ref_mod_and_test))

@@ -235,6 +263,7 @@ static inline int folio_ref_dec_and_test(struct folio *folio)
 static inline int page_ref_dec_return(struct page *page)
 {
 	int ret = atomic_dec_return(&page->_refcount);
+	VM_WARN_ON_ONCE_PAGE(__page_count_is_frozen(ret + 1), page);
 
 	if (page_ref_tracepoint_active(page_ref_mod_and_return))
 		__page_ref_mod_and_return(page, -1, ret);

@@ -246,7 +275,7 @@ static inline int folio_ref_dec_return(struct folio *folio)
 	return page_ref_dec_return(&folio->page);
 }
 
-static inline bool page_ref_add_unless_zero(struct page *page, int nr)
+static inline bool page_ref_add_unless_frozen(struct page *page, int nr)
 {
 	bool ret = atomic_add_unless(&page->_refcount, nr, 0);
 

@@ -255,9 +284,9 @@ static inline bool page_ref_add_unless_zero(struct page *page, int nr)
 	return ret;
 }
 
-static inline bool folio_ref_add_unless_zero(struct folio *folio, int nr)
+static inline bool folio_ref_add_unless_frozen(struct folio *folio, int nr)
 {
-	return page_ref_add_unless_zero(&folio->page, nr);
+	return page_ref_add_unless_frozen(&folio->page, nr);
 }
 
 /**

@@ -273,12 +302,12 @@ static inline bool folio_ref_add_unless_zero(struct folio *folio, int nr)
  */
 static inline bool folio_try_get(struct folio *folio)
 {
-	return folio_ref_add_unless_zero(folio, 1);
+	return folio_ref_add_unless_frozen(folio, 1);
 }
 
 static inline bool folio_ref_try_add(struct folio *folio, int count)
 {
-	return folio_ref_add_unless_zero(folio, count);
+	return folio_ref_add_unless_frozen(folio, count);
 }
 
 static inline int page_ref_freeze(struct page *page, int count)

kernel/liveupdate/kexec_handover.c +3 -3

index 7c4d86daf86d..b5966715bd95 100644
@@ -487,7 +487,7 @@ EXPORT_SYMBOL_GPL(kho_radix_walk_tree);
 static void kho_init_pages(struct page *page, unsigned long nr_pages)
 {
 	for (unsigned long i = 0; i < nr_pages; i++) {
-		set_page_count(page + i, 1);
+		init_page_count(page + i);
 		/* Clear each page's codetag to avoid accounting mismatch. */
 		clear_page_tag_ref(page + i);
 	}

@@ -498,13 +498,13 @@ static void kho_init_folio(struct page *page, unsigned int order)
 	unsigned long nr_pages = (1 << order);
 
 	/* Head page gets refcount of 1. */
-	set_page_count(page, 1);
+	init_page_count(page);
 	/* Clear head page's codetag to avoid accounting mismatch. */
 	clear_page_tag_ref(page);
 
 	/* For higher order folios, tail pages get a page count of zero. */
 	for (unsigned long i = 1; i < nr_pages; i++)
-		set_page_count(page + i, 0);
+		set_page_count_frozen(page + i);
 
 	if (order > 0)
 		prep_compound_page(page, order);

lib/test_hmm.c +2 -2

index 6911daa9f854..8171711e3e7a 100644
@@ -1844,7 +1844,7 @@ static void dmirror_devmem_folio_split(struct folio *head, struct folio *tail)
 	if (tail == NULL) {
 		folio_reset_order(rfolio);
 		rfolio->mapping = NULL;
-		folio_set_count(rfolio, 1);
+		folio_init_count(rfolio);
 		return;
 	}
 

@@ -1858,7 +1858,7 @@ static void dmirror_devmem_folio_split(struct folio *head, struct folio *tail)
 
 	folio_page(tail, 0)->mapping = folio_page(head, 0)->mapping;
 	tail->pgmap = head->pgmap;
-	folio_set_count(page_folio(rpage_tail), 1);
+	folio_init_count(page_folio(rpage_tail));
 }
 
 static const struct dev_pagemap_ops dmirror_devmem_ops = {

mm/hugetlb.c +1 -1

index 4f6f58bf3db6..ec620e0440d3 100644
@@ -3173,7 +3173,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
 	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
 		__init_single_page(page, pfn, zone, nid);
 		prep_compound_tail(page, &folio->page, order);
-		set_page_count(page, 0);
+		set_page_count_frozen(page);
 	}
 }
 

mm/internal.h +1 -1

index 38b1165212c9..3ed6d747b4e4 100644
@@ -661,7 +661,7 @@ static inline void set_page_refcounted(struct page *page)
 {
 	VM_BUG_ON_PAGE(PageTail(page), page);
 	VM_BUG_ON_PAGE(page_ref_count(page), page);
-	set_page_count(page, 1);
+	init_page_count(page);
 }
 
 static inline void set_pages_refcounted(struct page *page, unsigned long nr_pages)

mm/memory-failure.c +0 -0

index a8b03e2920ba..8d89e01ba105 100644

mm/memory-failure.c +3 -3

@@ -1166,7 +1166,7 @@ static int me_huge_page(struct page_state *ps, struct page *p)
 		 */
 		folio_put(folio);
 		if (__page_handle_poison(p) > 0) {
-			page_ref_inc(p);
+			init_page_count(p);
 			res = MF_RECOVERED;
 		} else {
 			res = MF_FAILED;

@@ -2132,7 +2132,7 @@ static int try_memory_failure_hugetlb(unsigned long pfn, int flags)
 	if (res == MF_HUGETLB_FREED) {
 		folio_unlock(folio);
 		if (__page_handle_poison(p) > 0) {
-			page_ref_inc(p);
+			init_page_count(p);
 			res = MF_RECOVERED;
 		} else {
 			res = MF_FAILED;

@@ -2462,7 +2462,7 @@ int memory_failure(unsigned long pfn, int flags)
 	case 0:
 		if (is_free_buddy_page(p)) {
 			if (take_page_off_buddy(p)) {
-				page_ref_inc(p);
+				init_page_count(p);
 				res = MF_RECOVERED;
 			} else {
 				/* We lost the race, try again */

mm/memremap.c +2 -2

index accba23aef28..e1188122ba80 100644
@@ -463,7 +463,7 @@ void free_zone_device_folio(struct folio *folio)
 		 * Reset the refcount to 1 to prepare for handing out the page
 		 * again.
 		 */
-		folio_set_count(folio, 1);
+		folio_init_count(folio);
 		break;
 
 	case MEMORY_DEVICE_FS_DAX:

@@ -520,7 +520,7 @@ void zone_device_page_init(struct page *page, struct dev_pagemap *pgmap,
 	 * memunmap_pages().
 	 */
 	WARN_ON_ONCE(!percpu_ref_tryget_many(&page_pgmap(page)->ref, 1 << order));
-	set_page_count(page, 1);
+	init_page_count(page);
 	lock_page(page);
 
 	if (order)

mm/mm_init.c +3 -3

index 1533aebafb68..51d29ebe2074 100644
@@ -1013,7 +1013,7 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
 	case MEMORY_DEVICE_PRIVATE:
 	case MEMORY_DEVICE_COHERENT:
 	case MEMORY_DEVICE_PCI_P2PDMA:
-		set_page_count(page, 0);
+		set_page_count_frozen(page);
 		break;
 
 	case MEMORY_DEVICE_GENERIC:

@@ -1066,7 +1066,7 @@ static void __ref memmap_init_compound(struct page *head,
 
 		__init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
 		prep_compound_tail(page, head, order);
-		set_page_count(page, 0);
+		set_page_count_frozen(page);
 	}
 	prep_compound_head(head, order);
 }

@@ -2165,7 +2165,7 @@ void __init init_cma_reserved_pageblock(struct page *page)
 
 	do {
 		__ClearPageReserved(p);
-		set_page_count(p, 0);
+		set_page_count_frozen(p);
 	} while (++p, --i);
 
 	init_pageblock_migratetype(page, MIGRATE_CMA, false);

mm/page_alloc.c +5 -5

index 12fac9084c48..2744e069862f 100644
@@ -1606,8 +1606,8 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 
 	/*
 	 * When initializing the memmap, __init_single_page() sets the refcount
-	 * of all pages to 1 ("allocated"/"not free"). We have to set the
-	 * refcount of all involved pages to 0.
+	 * of all pages to 1 ("allocated"/"not free"). We have to freeze the
+	 * refcount of all involved pages.
 	 *
 	 * Note that hotplugged memory pages are initialized to PageOffline().
 	 * Pages freed from memblock might be marked as reserved.

@@ -1617,14 +1617,14 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 		for (loop = 0; loop < nr_pages; loop++, p++) {
 			VM_WARN_ON_ONCE(PageReserved(p));
 			__ClearPageOffline(p);
-			set_page_count(p, 0);
+			set_page_count_frozen(p);
 		}
 
 		adjust_managed_page_count(page, nr_pages);
 	} else {
 		for (loop = 0; loop < nr_pages; loop++, p++) {
 			__ClearPageReserved(p);
-			set_page_count(p, 0);
+			set_page_count_frozen(p);
 		}
 
 		/* memblock adjusts totalram_pages() manually. */

@@ -6452,7 +6452,7 @@ void free_reserved_pages(struct page *page, unsigned int order)
 
 	for (i = 0; i < nr_pages; i++) {
 		clear_page_tag_ref(page + i);
-		set_page_count(page + i, 0);
+		set_page_count_frozen(page + i);
 		ClearPageReserved(page + i);
 	}
 	adjust_managed_page_count(page, nr_pages);

mm/page_frag_cache.c +1 -1

index e63efe78b7d4..56d19b2dea06 100644
@@ -143,7 +143,7 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc,
 			goto refill;
 		}
 
-		/* OK, page count is 0, we can safely set it */
+		/* OK, page is frozen, we can safely set count */
 		set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1);
 
 		/* reset page count bias and offset to start of new frag */

Signature · 1 line

2.55.0

The current page refcount implementation uses a single counter value (zero) as dead. So, to prevent incrementing a dead refcount in folio_try_get(), it fundamentally requires a CAS loop.

This CAS loop can act as a serialization point and can become a significant bottleneck during high-frequency file read operations [1][2].

This patch reallocates the refcount value range:

(1) refcount < 0 means dead refcount (uninit / frozen)
(2) refcount = 0 allowed only as a temporary state (see below)
(3) refcount > 0 is a regular reference count

In other words, refcount is now split into "dead bit" and a 31-bit counter.

Refcount decrement now works as follows:
1. Counter decrement
2. If it is now zero, try to put it deep inside the dead zone (CAS to INT_MIN). Or you can view it as "set up frozen bit and reset counter".
3. This CAS can fail only if someone grabbed a reference in-between -- that's okay, this page is their problem now.

The size of the dead zone allows performing an optimistic increment inside page_ref_add_unless_frozen(), replacing the previous read + CAS loop with a single RMW operation. This reduces cache line bouncing and improves scalability, especially in NUMA scenarios.

[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/CAHk-=wj00-nGmXEkxY=-=Z_qP6kiGUziSFvxHJ9N-cLWry5zpA@mail.gmail.com/

Reviewed-by Artem Kuzin <[email protected]>

Co-developed-by Ivan Gorbunov <[email protected]>

Signed-off-by Ivan Gorbunov <[email protected]>

Signed-off-by Ilya Gladyshev <[email protected]>

Acked-by Linus Torvalds <[email protected]>

---
 include/linux/page-flags.h | 13 +++++++++++++
 include/linux/page_ref.h   | 30 +++++++++++++++++++++++++-----
 2 files changed, 38 insertions(+), 5 deletions(-)

include/linux/page-flags.h +0 -0

index 7a863572adce..b19721e0e7ca 100644

include/linux/page-flags.h +0 -0

 #ifndef __GENERATING_BOUNDS_H
 
 /*

include/linux/page_ref.h +1 -1

index 82ff3a99297a..cc7a9d7db504 100644
@@ -64,7 +64,7 @@ static inline void __page_ref_unfreeze(struct page *page, int v)
 
 static inline bool __page_count_is_frozen(int count)
 {
-	return count == 0;
+	return count & PAGEREF_FROZEN_BIT;
 }
 
 static inline bool page_is_frozen(const struct page *page)

The current page refcount implementation uses a single counter value (zero) as dead. So, to prevent incrementing a dead refcount in folio_try_get(), it fundamentally requires a CAS loop.

This CAS loop can act as a serialization point and can become a significant bottleneck during high-frequency file read operations [1][2].

This patch reallocates the refcount value range:

(1) refcount < 0 means dead refcount (uninit / frozen)
(2) refcount = 0 allowed only as a temporary state (see below)
(3) refcount > 0 is a regular reference count

In other words, refcount is now split into "dead bit" and a 31-bit counter.

Refcount decrement now works as follows:
1. Counter decrement
2. If it is now zero, try to put it deep inside the dead zone (CAS to INT_MIN). Or you can view it as "set up frozen bit and reset counter".
3. This CAS can fail only if someone grabbed a reference in-between -- that's okay, this page is their problem now.

The size of the dead zone allows performing an optimistic increment inside page_ref_add_unless_frozen(), replacing the previous read + CAS loop with a single RMW operation. This reduces cache line bouncing and improves scalability, especially in NUMA scenarios.

[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/CAHk-=wj00-nGmXEkxY=-=Z_qP6kiGUziSFvxHJ9N-cLWry5zpA@mail.gmail.com/

Reviewed-by Artem Kuzin <[email protected]>

Co-developed-by Ivan Gorbunov <[email protected]>

Signed-off-by Ivan Gorbunov <[email protected]>

Signed-off-by Ilya Gladyshev <[email protected]>

Acked-by Linus Torvalds <[email protected]>

---
 include/linux/page-flags.h | 13 +++++++++++++
 include/linux/page_ref.h   | 30 +++++++++++++++++++++++++-----
 2 files changed, 38 insertions(+), 5 deletions(-)

Quoted 18 lines · Ilya Gladyshev

The current page refcount implementation uses a single counter value
(zero) as dead. So, to prevent incrementing a dead refcount in folio_try_get(), it fundamentally requires a CAS loop.

This CAS loop can act as a serialization point and can become a significant bottleneck during high-frequency file read operations [1][2].

This patch reallocates the refcount value range:

(1) refcount < 0 means dead refcount (uninit / frozen)
(2) refcount = 0 allowed only as a temporary state (see below)
(3) refcount > 0 is a regular reference count

In other words, refcount is now split into "dead bit" and a 31-bit counter.

Quoted 8 lines

Refcount decrement now works as follows:
1. Counter decrement
2. If it is now zero, try to put it deep inside the dead zone (CAS to INT_MIN). Or you can view it as "set up frozen bit and reset counter".
3. This CAS can fail only if someone grabbed a reference in-between -- that's okay, this page is their problem now.

I am not sure this works. David raised a concern on V4[1], where if a folio lands to page_frag_free()'s free_frozen_pages(), it will not be properly handled. "this page is their problem" only works if every folio/page free functions call a generic folio/page free function that checks the page type and does the proper handling.

Quoted 7 lines

Refcount decrement now works as follows:
1. Counter decrement
2. If it is now zero, try to put it deep inside the dead zone (CAS to INT_MIN). Or you can view it as "set up frozen bit and reset counter".
3. This CAS can fail only if someone grabbed a reference in-between -- that's okay, this page is their problem now.

I am not sure this works. David raised a concern on V4[1], where if a folio lands to page_frag_free()'s free_frozen_pages(), it will not be properly handled. "this page is their problem" only works if every folio/page free functions call a generic folio/page free function that checks the page type and does the proper handling.

Before your patches, only a speculative page getter can see any page and folio_put() handles the last reference drop. So it is safe today.

Let me know if I miss anything.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 9 lines

The size of the dead zone allows performing an optimistic increment
inside page_ref_add_unless_frozen(), replacing the previous read + CAS loop with a single RMW operation. This reduces cache line bouncing and improves scalability, especially in NUMA scenarios.

[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/CAHk-=wj00-nGmXEkxY=-=Z_qP6kiGUziSFvxHJ9N-cLWry5zpA@mail.gmail.com/

Quoted 10 lines

Reviewed-by Artem Kuzin <[email protected]>

Co-developed-by Ivan Gorbunov <[email protected]>

Signed-off-by Ivan Gorbunov <[email protected]>

Signed-off-by Ilya Gladyshev <[email protected]>

Acked-by Linus Torvalds <[email protected]>

---

 include/linux/page-flags.h | 13 +++++++++++++
 include/linux/page_ref.h | 30 +++++++++++++++++++++++++-----
 2 files changed, 38 insertions(+), 5 deletions(-)

Signature · 2 lines

Best Regards,
Yan, Zi

Quoted 43 lines · Zi Yan

Quoted 18 lines · Ilya Gladyshev

The current page refcount implementation uses a single counter value
(zero) as dead. So, to prevent incrementing a dead refcount in folio_try_get(), it fundamentally requires a CAS loop.

This CAS loop can act as a serialization point and can become a significant bottleneck during high-frequency file read operations [1][2].

This patch reallocates the refcount value range:

(1) refcount < 0 means dead refcount (uninit / frozen)
(2) refcount = 0 allowed only as a temporary state (see below)
(3) refcount > 0 is a regular reference count

In other words, refcount is now split into "dead bit" and a 31-bit counter.

Quoted 8 lines

Refcount decrement now works as follows:
1. Counter decrement
2. If it is now zero, try to put it deep inside the dead zone (CAS to INT_MIN). Or you can view it as "set up frozen bit and reset counter".
3. This CAS can fail only if someone grabbed a reference in-between -- that's okay, this page is their problem now.

I am not sure this works. David raised a concern on V4[1], where if a folio lands to page_frag_free()'s free_frozen_pages(), it will not be properly handled. "this page is their problem" only works if every folio/page free functions call a generic folio/page free function that checks the page type and does the proper handling.

Quoted 7 lines

Refcount decrement now works as follows:
1. Counter decrement
2. If it is now zero, try to put it deep inside the dead zone (CAS to INT_MIN). Or you can view it as "set up frozen bit and reset counter".
3. This CAS can fail only if someone grabbed a reference in-between -- that's okay, this page is their problem now.

Quoted 5 lines

I am not sure this works. David raised a concern on V4[1], where if a folio lands to page_frag_free()'s free_frozen_pages(), it will not be properly handled. "this page is their problem" only works if every folio/page free functions call a generic folio/page free function that checks the page type and does the proper handling.

After some thinking, this scenario is indeed possible and problematic. The complexity of the required setup give me hope that there are some implicit barriers, however I wasn't able to find them. But I am not an mm guru, so maybe I missed something :shrug:

-- Below is a minimal buggy scenario --

Thread A: network code that calls page_frag_free()
Thread B: calls folio_try_get(). Maybe it is the DAMON paddr scanner,
maybe this folio was previously used in page cache (however this noticeably complicates the scenario)
Thread C: user of folio's reincarnation. It also requires destruction via
folio_put() and not via free_frozen_pages() -- for example hugetlb or memcg charged folio.

Refcount operations as follows (FR = FROZEN):

A: DEC 1 -> 0 [ page_ref_dec_and_test() ]
B: INC 0 -> 1 [ folio_try_get() ] => success
B: DEC 1 -> 0 [ page_ref_dec_and_test() ]
B: CAS 0 -> FR [ page_ref_dec_and_test() ]
B: * deallocates via __folio_put() *

C: * re-allocates page *
C: DEC 1 -> 0 [ page_ref_dec_and_test() ]
A: CAS 0 -> FR [ page_ref_dec_and_test() ] => success
A: * deallocates directly via free_frozen_pages() * => BUG

This can be fixed with a s/page_frag_free/folio_put/-style patch, however it seems unreasonable...

There needs to be a common page freeing function that dipatches stuff.

Quoted 2 lines · David Hildenbrand (Arm)

There needs to be a common page freeing function that dipatches stuff.

I think this all needs to be entirely generic to all pages. Agreed. Everybody needs to do it right.

But I think that in the scenario that Ilya mentioned:

Refcount operations as follows (FR = FROZEN):

  A: DEC 1 -> 0  [ page_ref_dec_and_test() ]
  B: INC 0 -> 1  [ folio_try_get() ] => success
  B: DEC 1 -> 0  [ page_ref_dec_and_test() ]

B: CAS 0 -> FR [ page_ref_dec_and_test() ]
B: * deallocates via __folio_put() *

C: * re-allocates page *
C: DEC 1 -> 0 [ page_ref_dec_and_test() ]
A: CAS 0 -> FR [ page_ref_dec_and_test() ] => success
A: * deallocates directly via free_frozen_pages() * => BUG

the big here is that the de-allocation was done not from the state of the *page*, but from the state of who finished it off.

So that

A: CAS 0 -> FR [ page_ref_dec_and_test() ] => success

is not a problem per se. It's a fine case of "we had one single unique final free".

The failure is that A then goes on to do that

"deallocates directly via free_frozen_pages"

because A mis-judged the page as having the old stale state that it had when *A* was using it.

IOW, I think the soluition is either:

(a) every user of put_page_testzero(page) agrees tro do exactly the
same thing and there is no difference between how that final "0->FROZEN" is dealt with, so it doesn't matter that A is freeing a page that C allocated for something else

or

(b) the CAS 0 -> FR state sequence always looks at the *page* to
decide what to do, never at the caller state (ie A will look at whatever C wrote when it allocated the page to know how to do that final free).

Hmm?

Quoted 9 lines · Linus Torvalds

Quoted 3 lines · David Hildenbrand (Arm)

There needs to be a common page freeing function that dipatches stuff.

I think this all needs to be entirely generic to all pages. Agreed. Everybody needs to do it right.

It assumes all callers of page_ref_dec_and_test() are free page functions, but there are two exceptions:

1. compaction_free() uses folio_put_testzero(), which calls page_ref_dec_and_test() to put not used migration dst page back to isolated freelist instead of freeing them.

2. virtio_mem_fake_offline_going_offline() drops the last reference of fake-offline pages, so that they can be offline.

They will need to be converted to not use page_ref_dec_and_test() in addition to a common page freeing function. And any such future use needs to be banned.

Quoted 42 lines

But I think that in the scenario that Ilya mentioned:

Refcount operations as follows (FR = FROZEN):

A: DEC 1 -> 0 [ page_ref_dec_and_test() ]
B: INC 0 -> 1 [ folio_try_get() ] => success
B: DEC 1 -> 0 [ page_ref_dec_and_test() ]
B: CAS 0 -> FR [ page_ref_dec_and_test() ]
B: * deallocates via __folio_put() *

C: * re-allocates page *
C: DEC 1 -> 0 [ page_ref_dec_and_test() ]
A: CAS 0 -> FR [ page_ref_dec_and_test() ] => success
A: * deallocates directly via free_frozen_pages() * => BUG

the big here is that the de-allocation was done not from the state of the *page*, but from the state of who finished it off.

So that

A: CAS 0 -> FR [ page_ref_dec_and_test() ] => success

is not a problem per se. It's a fine case of "we had one single unique final free".

The failure is that A then goes on to do that

"deallocates directly via free_frozen_pages"

because A mis-judged the page as having the old stale state that it had when *A* was using it.

IOW, I think the soluition is either:

(a) every user of put_page_testzero(page) agrees tro do exactly the same thing and there is no difference between how that final "0->FROZEN" is dealt with, so it doesn't matter that A is freeing a page that C allocated for something else

or

Quoted 7 lines

(b) the CAS 0 -> FR state sequence always looks at the *page* to decide what to do, never at the caller state (ie A will look at whatever C wrote when it allocated the page to know how to do that final free).

Hmm?

Just wonder if we could make 1 -> FROZEN atomic to get rid of the gap.

Quoted 69 lines · Zi Yan

Quoted 5 lines · Linus Torvalds

I think this all needs to be entirely generic to all pages. Agreed. Everybody needs to do it right.

It assumes all callers of page_ref_dec_and_test() are free page functions, but there are two exceptions:

1. compaction_free() uses folio_put_testzero(), which calls page_ref_dec_and_test() to put not used migration dst page back to isolated freelist instead of freeing them.

2. virtio_mem_fake_offline_going_offline() drops the last reference of fake-offline pages, so that they can be offline.

They will need to be converted to not use page_ref_dec_and_test() in addition to a common page freeing function. And any such future use needs to be banned.

Quoted 47 lines

But I think that in the scenario that Ilya mentioned:

Refcount operations as follows (FR = FROZEN):

A: DEC 1 -> 0 [ page_ref_dec_and_test() ]
B: INC 0 -> 1 [ folio_try_get() ] => success
B: DEC 1 -> 0 [ page_ref_dec_and_test() ]
B: CAS 0 -> FR [ page_ref_dec_and_test() ]
B: * deallocates via __folio_put() *

C: * re-allocates page *
C: DEC 1 -> 0 [ page_ref_dec_and_test() ]
A: CAS 0 -> FR [ page_ref_dec_and_test() ] => success
A: * deallocates directly via free_frozen_pages() * => BUG

the big here is that the de-allocation was done not from the state of the *page*, but from the state of who finished it off.

So that

A: CAS 0 -> FR [ page_ref_dec_and_test() ] => success

is not a problem per se. It's a fine case of "we had one single unique final free".

The failure is that A then goes on to do that

"deallocates directly via free_frozen_pages"

because A mis-judged the page as having the old stale state that it had when *A* was using it.

IOW, I think the soluition is either:

(a) every user of put_page_testzero(page) agrees tro do exactly the same thing and there is no difference between how that final "0->FROZEN" is dealt with, so it doesn't matter that A is freeing a page that C allocated for something else

or

(b) the CAS 0 -> FR state sequence always looks at the *page* to decide what to do, never at the caller state (ie A will look at whatever C wrote when it allocated the page to know how to do that final free).

Quoted 3 lines

Quoted 2 lines

Hmm?

Just wonder if we could make 1 -> FROZEN atomic to get rid of the gap.

Hmmm, I’m afraid that since you need CAS for a safe 1->FR transition, it will result in a CAS loop for the decrement itself (like in atomic_sub_unless).

Hmmm, I’m afraid that since you need CAS for a safe 1->FR transition, it will result in a CAS loop for the decrement itself (like in atomic_sub_unless). And this will introduce scalability issues just like in folio_try_get(), but this time for everyone...

Quoted 5 lines · Ilya Gladyshev

Hmmm, I’m afraid that since you need CAS for a safe 1->FR transition, it will result in a CAS loop for the decrement itself (like in atomic_sub_unless). And this will introduce scalability issues just like in folio_try_get(), but this time for everyone...

Note that we could make that CAS case be the thing that only the special cases do.

IOW, maybe only do that slow sequence in compaction_free() and the memory offlining.

So we'd have two different cases:

- the high-performance case is ready and willing to accept the "sees
zero" window and the extra 0->FR state that can race with somebody else taking an optimistic ref

- the unusual slow cases that are *not* willing to deal with
optimistic ref takers do the "CAS 1 -> FR" state atomically and always use compare-and-exchange for their freeing path

That actually sounds like a good approach to me.

Quoted 6 lines · Linus Torvalds

Quoted 2 lines · David Hildenbrand (Arm)

There needs to be a common page freeing function that dipatches stuff.

I think this all needs to be entirely generic to all pages. Agreed. Everybody needs to do it right.

It assumes all callers of page_ref_dec_and_test() are free page functions, but there are two exceptions:

1. compaction_free() uses folio_put_testzero(), which calls page_ref_dec_and_test() to put not used migration dst page back to isolated freelist instead of freeing them.

2. virtio_mem_fake_offline_going_offline() drops the last reference of fake-offline pages, so that they can be offline.

This patch reallocates the refcount value range:

(1) refcount < 0 means dead refcount (uninit / frozen)
(2) refcount = 0 allowed only as a temporary state (see below)
(3) refcount > 0 is a regular reference count

In other words, refcount is now split into "dead bit" and a 31-bit counter.

Quoted 6 lines · Ilya Gladyshev

This patch reallocates the refcount value range:

(1) refcount < 0 means dead refcount (uninit / frozen)
(2) refcount = 0 allowed only as a temporary state (see below)
(3) refcount > 0 is a regular reference count

Quoted 2 lines

In other words, refcount is now split into "dead bit" and a 31-bit counter.

It seems to me that refcount overflow is now a problem.

We can deliberately increase the refcount on a page by stuffing it into a pipe. Over and over again. See merge commit 6b3a70773630 and the four commits on that branch: f958d7b528b1 88b1a17dfc3e 8fde12ca79af 15fab63e1e57

Unfortunately, I think the discussion that led to those commits was conducted off-list because security. I wish we had a way to declassify thse emails after the fact.

include/linux/page-flags.h enum pageflags { · line 207

Quoted 9 lines

include/linux/page-flags.h +9 -0

+/* Most significant bit in page refcount */
+#define PAGEREF_FROZEN_BIT BIT(31)
+
+/* Page reference counter can be in 3 logical states,
+ * which are described below with their value representation
+ *        state              |         value
+ * (1)  safe with  owners    |   1...INT_MAX
+ * (2)  safe with no owners  |         0
+ * (3)  frozen               |  INT_MIN....-1

I think we need four states. The first two are the same.

(3) frozen: 0xc000'0000 - 0xffff'ffff
(4) temporarily overflown: 0x8000'0000 - 0xbfff'ffff

(we don't really need that much space for temporary overflow; we could have something like 0x8100'0000 as the boundary if that works out better)

include/linux/page_ref.h static inline void __page_ref_unfreeze(struct page *page, int v) · line 67

Quoted 4 lines

  static inline bool __page_count_is_frozen(int count)
  {

include/linux/page_ref.h +1 -1

-	return count == 0;
+	return count & PAGEREF_FROZEN_BIT;

This probably becomes '(unsigned)count >> 30 == 3'. If we choose a different boundary then something like (unsigned)count >> 24 >= 0x81.

include/linux/page_ref.h static inline bool folio_is_frozen(const struct folio *folio) · line 85

Quoted 7 lines

static inline int page_ref_count(const struct page *page) {

include/linux/page_ref.h +4 -1

-	return atomic_read(&page->_refcount);
+	int val = atomic_read(&page->_refcount);
+
+	if (unlikely(val & PAGEREF_FROZEN_BIT))
+		return 0;

... use __page_count_is_frozen() here?

and you need to adjust folio_ref_zero_or_close_to_overflow(). try_get_page() can stay as it is.

(Thanks to Pedro for asking me annoying questions about mapcount overflow which prompted me to look at this again)

From: Gladyshev Ilya <[email protected]>

Recap
-----

This patchset addresses a scalability issue of a folio's add_unless() operation, noticeable during contended IO reads from the same page [folio_try_get()]. The main idea is to replace CAS loop with optimistic increment (and deal with failure later). This requires splitting refcount into counter and separate "dead/frozen" bit.

To allow for such modification, this patchset also slightly refactors page_ref API, consolidating all implementation logic inside mm headers. For more information, check individual commit messages. The original performance issue and previous attempts by other people can be found in [1][2].

Performance
-----------

To my regret, I don't have any access to high-core CPUs that I can benchmark on, and a 12 vcpu laptop isn't really a scalability test. So, here I can only paste my previous measurements on Linux 6.15. To be fair, none of the related code paths really changed, so I don't expect any changes in the numbers here.

[1]: https://lore.kernel.org/linux-mm/CAHk-=wj00-nGmXEkxY=-=Z_qP6kiGUziSFvxHJ9N-cLWry5zpA@mail.gmail.com/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/all/aqAIFV4nOGPbWiDS@thinkstation/

Quoted 16 lines · On Sat, 12 Sep 2026 22:50:07 +0300 Ilya Gladyshev <[email protected]>…

Quoted 1 line

From: Gladyshev Ilya <[email protected]>

Recap
-----

This patchset addresses a scalability issue of a folio's add_unless() operation, noticeable during contended IO reads from the same page [folio_try_get()]. The main idea is to replace CAS loop with optimistic increment (and deal with failure later). This requires splitting refcount into counter and separate "dead/frozen" bit.

To allow for such modification, this patchset also slightly refactors page_ref API, consolidating all implementation logic inside mm headers. For more information, check individual commit messages. The original performance issue and previous attempts by other people can be found in [1][2].

Quoted 8 lines

Performance
-----------

To my regret, I don't have any access to high-core CPUs that I can benchmark on, and a 12 vcpu laptop isn't really a scalability test. So, here I can only paste my previous measurements on Linux 6.15. To be fair, none of the related code paths really changed, so I don't expect any changes in the numbers here.

From a quick eyeballing, this patchset seems to provide performance gains which are similar to Kiryl's ill-fated "mm/filemap: Implement fast short reads" [2]. Is that accurate?

Quoted 3 lines

[1]: https://lore.kernel.org/linux-mm/CAHk-=wj00-nGmXEkxY=-=Z_qP6kiGUziSFvxHJ9N-cLWry5zpA@mail.gmail.com/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/all/aqAIFV4nOGPbWiDS@thinkstation/