kho: support preserving high-order non-compound pages
Subsystems (3)
v3 cover kho: support preserving high-order non-compound pages
Introduction
============
This series is required for the ongoing effort to preserve DMA allocations across KHO [1]. It addresses a fundamental mismatch between the current KHO restoration logic and the physical reality of high-order buddy allocations.
The Problem
===========
The current KHO restore implementation treats all multi-page blocks as split pages during restoration. Specifically, kho_restore_pages() initializes every 4KB sub-page with a refcount of 1.
However, many kernel subsystems, most notably the DMA allocator (via dma_alloc_coherent), frequently return high-order non-compound pages. In this state, only the head page carries a refcount of 1, while all tail pages have a refcount of 0.
Consequently, when these contiguous blocks are restored by KHO in the new kernel, the forced reference count of 1 on tail pages causes some trouble with the buddy allocator. Downstream of the eventual free path, __free_pages_prepare() [2] ends up calling page_expected_state() [3] when is_check_pages_enabled() returns true (triggered when CONFIG_DEBUG_VM is enabled or debug_pagealloc=on).
This detects the unexpected non-zero reference counts on tail pages [4] and incorrectly taints the kernel while leaking the physical pages in question.
Proposed Solution
=================
Following feedback on the v1 RFC, this series moves away from auto type detection and instead introduces explicit preserve / restore APIs for high-order pages.
Callers now explicitly preserve these high-order blocks as a single unit by using kho_preserve_page() and kho_restore_page(). These functions apply a refcount of 1 to the head page while leaving tail pages at 0.
The existing APIs (kho_preserve_pages / kho_restore_pages) remain as is for ranges of independent 4KB pages, continuing to use the split refcount.
The internal initialization logic is refactored to provide a helper:kho_init_high_order_page(), which is shared between folios and high-order page restore APIs. We also consolidate the common metadata validation and state clearing into __kho_restore_page() to avoid duplication.
[v3]
- Renamed "unsplit" terminology to "high-order".
- Consolidated the common restoration code (magic checks, private clearing etc.) into the internal __kho_restore_page() helper.
[v2]
- https://lore.kernel.org/all/[email protected]/
- Dropped automatic type detection via higher bits in Radix key.
- Introduced explicit kho_preserve_page and kho_restore_page helpers.
- Refactored internal init logic to share code between folios & high-order pages.
[v1] https://lore.kernel.org/all/[email protected]/
Thanks,
Praan
[1] https://lore.kernel.org/all/[email protected]/
[2] https://elixir.bootlin.com/linux/v7.1.1/source/mm/page_alloc.c#L1370
[3] https://elixir.bootlin.com/linux/v7.1.1/source/mm/page_alloc.c#L1027
[4] https://elixir.bootlin.com/linux/v7.1.1/source/mm/page_alloc.c#L1034
Pranjal Shrivastava (2):
kho: Introduce a helper to init high order pages
kho: Introduce preserve/restore APIs for high-order pages
include/linux/kexec_handover.h | 10 +++ kernel/liveupdate/kexec_handover.c | 130 ++++++++++++++++++++++++----- 2 files changed, 117 insertions(+), 23 deletions(-)
Signature · 1 line
2.55.0.229.g6434b31f56-goog
PATCH v3 1/2 kho: Introduce a helper to init high order pages
The current KHO restoration logic assumes all multi-page blocks are split into independent 4KB pages. Break out a helper to prepare for supporting high-order non-compound pages.
Extract kho_init_high_order_page() to handle the refcount pattern where only the head page is refcounted. Use the helper for folio restoration that requires a similar refcount logic.
--- kernel/liveupdate/kexec_handover.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
kernel/liveupdate/kexec_handover.c +19 -10
@@ -357,6 +357,24 @@ int kho_radix_walk_tree(struct kho_radix_tree *tree, }EXPORT_SYMBOL_GPL (kho_radix_walk_tree); +/* For physically contiguous pages. */ +static void kho_init_high_order_page (struct page *page,unsigned int order) +{ +unsigned long nr_pages = (1UL << order); + +/* Head page gets refcount of 1. */ +set_page_count (page,1 ); +/* Clear head page's codetag to avoid accounting mismatch. */ +clear_page_tag_ref (page); + +/* For high-order blocks, tail pages get a page count of zero. */ +for (unsigned long i =1 ; i < nr_pages; i++) { +set_page_count (page + i,0 ); +/* Clear each page's codetag to avoid accounting mismatch. */ +clear_page_tag_ref (page + i); + } +} +/* For physically contiguous 0-order pages. */ static void kho_init_pages (struct page *page,unsigned long nr_pages) {
@@ -369,16 +387,7 @@ static void kho_init_pages(struct page *page, unsigned long nr_pages)static void kho_init_folio (struct page *page,unsigned int order) { -unsigned long nr_pages = (1 << order); - -/* Head page gets refcount of 1. */ -set_page_count (page,1 ); -/* Clear head page's codetag to avoid accounting mismatch. */ -clear_page_tag_ref (page); - -/* For higher order folios, tail pages get a page count of zero. */ -for (unsigned long i =1 ; i < nr_pages; i++) -set_page_count (page + i,0 ); +kho_init_high_order_page (page, order);if (order >0 )prep_compound_page (page, order);
Signature · 1 line
2.55.0.229.g6434b31f56-goog
PATCH v3 2/2 kho: Introduce preserve/restore APIs for high-order pages
In reply to Pranjal ShrivastavaThe current KHO page preservation APIs (e.g. kho_preserve_pages) assume that multi-page blocks are split into independent 4KB pages during restoration. This is incompatible with high-order non-compound pages, such as DMA buffers, which must be restored with tail pages having a zero reference count.
Introduce explicit preserve and restore APIs for high-order pages, which preserve and restore a high-order page block as a single unit, applying a refcount of 1 to the head page while leaving tail pages at 0. Rename the existing internal helper to __kho_restore_page() and consolidate the common restoration code into it.
--- include/linux/kexec_handover.h | 10 +++ kernel/liveupdate/kexec_handover.c | 101 +++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 13 deletions(-)
include/linux/kexec_handover.h +10 -0
@@ -22,6 +22,8 @@ bool is_kho_boot(void);int kho_preserve_folio (struct folio *folio);void kho_unpreserve_folio (struct folio *folio); +int kho_preserve_page (struct page *page,unsigned int order); +void kho_unpreserve_page (struct page *page,unsigned int order);int kho_preserve_pages (struct page *page,unsigned long nr_pages);void kho_unpreserve_pages (struct page *page,unsigned long nr_pages);int kho_preserve_vmalloc (void *ptr,struct kho_vmalloc *preservation);
@@ -30,6 +32,7 @@ void *kho_alloc_preserve(size_t size);void kho_unpreserve_free (void *mem);void kho_restore_free (void *mem);struct folio *kho_restore_folio (phys_addr_t phys); +struct page *kho_restore_page (phys_addr_t phys);struct page *kho_restore_pages (phys_addr_t phys,unsigned long nr_pages);void *kho_restore_vmalloc (const struct kho_vmalloc *preservation);int kho_add_subtree (const char *name,void *blob,size_t size);
@@ -65,6 +68,13 @@ static inline int kho_preserve_pages(struct page *page, unsigned int nr_pages)static inline void kho_unpreserve_pages (struct page *page,unsigned int nr_pages) { } +static inline int kho_preserve_page (struct page *page,unsigned int order) +{ +return -EOPNOTSUPP ; +} + +static inline void kho_unpreserve_page (struct page *page,unsigned int order) { } +static inline int kho_preserve_vmalloc (void *ptr,struct kho_vmalloc *preservation) {
kernel/liveupdate/kexec_handover.c +88 -13
150 lines not on this page · raw message
Signature · 1 line
2.55.0.229.g6434b31f56-goog
Mike Rapoport
maintainerRe: [PATCH v3 2/2] kho: Introduce preserve/restore APIs for high-order pages
Hi Pranjal,
Quoted 17 lines
The current KHO page preservation APIs (e.g.
kho_preserve_pages) assume that multi-page blocks are split into independent 4KB pages during restoration. This is incompatible with high-order non-compound pages, such as DMA buffers, which must be restored with tail pages having a zero reference count.Introduce explicit preserve and restore APIs for high-order pages, which preserve and restore a high-order page block as a single unit, applying a refcount of 1 to the head page while leaving tail pages at 0. Rename the existing internal helper to
__kho_restore_page()and consolidate the common restoration code into it.Signed-off-by Pranjal Shrivastava <[email protected]>kernel/liveupdate/kexec_handover.c +0
-0@@ -457,10 +482,14 @@ struct page *kho_restore_pages(phys_addr_t phys, unsigned long nr_pages)while (pfn < end_pfn) { const unsigned int order =
kernel/liveupdate/kexec_handover.c struct page *kho_restore_pages(phys_addr_t phys, unsigned long nr_pages) · line 492
Quoted 10 lines
kernel/liveupdate/kexec_handover.c +5
-1min (count_trailing_zeros (pfn),ilog2 (end_pfn - pfn)); -struct page *page =kho_restore_page (PFN_PHYS (pfn), false); +unsigned int info_order; +struct page *page =__kho_restore_page (PFN_PHYS (pfn), &info_order);if (!page)return NULL ; + +kho_init_pages (page,1UL << info_order); +adjust_managed_page_count (page,1UL << info_order);
Seeing
+ adjust_managed_page_count(page, 1UL << info_order);
for the third time hinted that something is off ;-)
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v3 2/2] kho: Introduce preserve/restore APIs for high-order pages
Quoted 34 lines · Mike Rapoport
Hi Pranjal,
Quoted 17 lines
The current KHO page preservation APIs (e.g.
kho_preserve_pages) assume that multi-page blocks are split into independent 4KB pages during restoration. This is incompatible with high-order non-compound pages, such as DMA buffers, which must be restored with tail pages having a zero reference count.Introduce explicit preserve and restore APIs for high-order pages, which preserve and restore a high-order page block as a single unit, applying a refcount of 1 to the head page while leaving tail pages at 0. Rename the existing internal helper to
__kho_restore_page()and consolidate the common restoration code into it.Signed-off-by Pranjal Shrivastava <[email protected]>kernel/liveupdate/kexec_handover.c +0
-0@@ -457,10 +482,14 @@ struct page *kho_restore_pages(phys_addr_t phys, unsigned long nr_pages)while (pfn < end_pfn) { const unsigned int order =kernel/liveupdate/kexec_handover.c struct page *kho_restore_pages(phys_addr_t phys, unsigned long nr_pages) · line 492
Quoted 10 lines
kernel/liveupdate/kexec_handover.c +5
-1min (count_trailing_zeros (pfn),ilog2 (end_pfn - pfn)); -struct page *page =kho_restore_page (PFN_PHYS (pfn), false); +unsigned int info_order; +struct page *page =__kho_restore_page (PFN_PHYS (pfn), &info_order);if (!page)return NULL ; + +kho_init_pages (page,1UL << info_order); +adjust_managed_page_count (page,1UL << info_order);Seeing
kernel/liveupdate/kexec_handover.c +1
-0+adjust_managed_page_count (page,1UL << info_order);
Quoted 2 lines
for the third time hinted that something is off ;-)
Ack. I'll fold it into the __kho_restore_page() helper
Thanks,
Praan