Make KHO Stateless
Subsystems (5)
cover Make KHO Stateless
This series transitions KHO from an xarray-based metadata tracking system with serialization to using page table like data structures that can be passed directly to the next kernel.
The key motivations for this change are to:
- Eliminate the need for data serialization before kexec.
- Remove the former KHO state machine by deprecating the finalize and abort states.
- Pass preservation metadata more directly to the next kernel via the FDT.
The new approach uses a per-order page table structure (kho_order_table, kho_page_table, kho_bitmap_table) to mark preserved pages. The physical address of the root kho_order_table is passed in the FDT, allowing the next kernel to reconstruct the preserved memory map.
The series includes the following changes:
1. Introduce the KHO page table data structures.
2. Adopt the KHO page tables, remove the xarray-based tracking and the serialization/finalization code.
3. Update memblock to use direct KHO API calls, and adjust KHO FDT completion timing.
4. Remove the KHO notifier system infrastructure.
Jason Miu (4):
kho: Introduce KHO page table data structures
kho: Adopt KHO page tables and remove serialization
memblock: Remove KHO notifier usage
kho: Remove notifier system infrastructure
include/linux/kexec_handover.h | 44 +- kernel/kexec_core.c | 4 + kernel/kexec_handover.c | 821 ++++++++++++++++----------------- kernel/kexec_internal.h | 2 + mm/memblock.c | 46 +- 5 files changed, 404 insertions(+), 513 deletions(-)
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
RFC 1/4 kho: Introduce KHO page table data structures
Introduce a page-table-like data structure for tracking preserved memory pages, which will replace the current xarray-based implementation.
The primary motivation for this change is to eliminate the need for serialization. By marking preserved pages directly in these new tables and passing them to the next kernel, the entire serialization process can be removed. This ultimately allows for the removal of the KHO finalize and abort states, simplifying the overall design.
The new KHO page table is a hierarchical structure that maps physical addresses to preservation metadata. It begins with a root kho_order_table that contains an entry for each page order. Each entry points to a separate, multi-level tree of kho_page_tables that splits a physical address into indices. The traversal terminates at a kho_bitmap_table, where each bit represents a single preserved page.
This commit adds the core data structures for this hierarchy:
- kho_order_table: The root table, indexed by page order.
- kho_page_table: Intermediate-level tables.
- kho_bitmap_table: The lowest-level table where individual pages
are marked.
The new functions are not yet integrated with the public kho_preserve_* APIs and are marked __maybe_unused. The full integration and the removal of the old xarray code will follow in a subsequent commit.
--- kernel/kexec_handover.c | 344 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 344 insertions(+)
kernel/kexec_handover.c +344 -0
@@ -46,6 +46,350 @@ static int __init kho_parse_enable(char *p) }early_param ("kho" , kho_parse_enable); +/* +* KHO page tables provide a page-table-like data structure for tracking +* preserved memory pages. It is a hierarchical structure that starts with a +* `struct kho_order_table`. Each entry in this table points to the root of a +* `struct kho_page_table` tree, which tracks the preserved memory pages for a +* specific page order. +* +* Each entry in a `struct kho_page_table` points to the next level page table, +* until level 2, which points to a `struct kho_bitmap_table`. The lowest level +* (level 1) is a bitmap table where each bit represents a preserved page. +* +* The table hierarchy is shown as below. +* +* kho_order_table +* +-------------------------------+--------------------+ +* | 0 order| 1 order| 2 order ... | HUGETLB_PAGE_ORDER | +* ++------------------------------+--------------------+ +* | +* | +* v +* ++------+ +* | Lv6 | kho_page_table +* ++------+ +* | +* | +* | +-------+ +* +-> | Lv5 | kho_page_table +* ++------+ +* | +* | +* | +-------+ +* +-> | Lv4 | kho_page_table +* ++------+ +* | +* | +* | +-------+ +* +-> | Lv3 | kho_page_table +* ++------+ +* | +* | +* | +-------+ +* +> | Lv2 | kho_page_table +* ++------+ +* | +* | +* | +-------+ +* +-> | Lv1 | kho_bitmap_table +* +-------+ +* +* The depth of the KHO page tables depends on the system's page size and the +* page order. Both larger page sizes and higher page orders result in +* shallower KHO page tables. For example, on a system with a 4KB native +* page size, 0-order tables have a depth of 6 levels. +* +* The following diagram illustrates how a physical address is split into +* indices for the different KHO page table levels and the final bitmap. +* +* 63 62:54 53:45 44:36 35:27 26:0 +* +--------+--------+--------+--------+--------+-----------------+ +* | Lv 6 | Lv 5 | Lv 4 | Lv 3 | Lv 2 | Lv 1 (bitmap) | +* +--------+--------+--------+--------+--------+-----------------+ +* +* For higher order pages, the bit fields for each level shift to the left by +* the page order. +* +* Each KHO page table and bitmap table is PAGE_SIZE in size. For 0-order +* pages, the bitmap table contains (PAGE_SIZE * 8) bits, covering a +* (PAGE_SIZE * 8 * PAGE_SIZE) memory range. For example, on a system with a +* 4KB native page size, the bitmap table contains 32768 bits and covers a +* 128MB memory range. +* +* Each KHO page table contains (PAGE_SIZE / 8) entries, where each entry is a +* descriptor (a physical address) pointing to the next level table. +* For example, with a 4KB page size, each page table holds 512 entries. +* The level 2 KHO page table is an exception, where each entry points to a +* KHO bitmap table instead. +* +* An entry of a KHO page table of a 4KB page system is shown as below as an +* example. +* +* 63:12 11:0 +* +------------------------------+--------------+ +* | descriptor to next table | zeros | +* +------------------------------+--------------+ +*/ + +#define BITMAP_TABLE_SHIFT (_order) (PAGE_SHIFT + PAGE_SHIFT + 3 + (_order)) +#define BITMAP_TABLE_MASK (_order) ((1ULL << BITMAP_TABLE_SHIFT(_order)) - 1) +#define PRESERVED_PAGE_OFFSET_SHIFT (_order) (PAGE_SHIFT + (_order)) +#define PAGE_TABLE_SHIFT_PER_LEVEL (ilog2(PAGE_SIZE / sizeof(unsigned long))) +#define PAGE_TABLE_LEVEL_MASK ((1ULL << PAGE_TABLE_SHIFT_PER_LEVEL) - 1) +#define PTR_PER_LEVEL (PAGE_SIZE / sizeof(unsigned long)) + +typedef int (*kho_walk_callback_t )(phys_addr_t pa,int order); + +struct kho_bitmap_table { +unsigned long bitmaps[PAGE_SIZE /sizeof (unsigned long )]; +}; + +struct kho_page_table { +unsigned long tables[PTR_PER_LEVEL ]; +}; + +struct kho_order_table { +unsigned long orders[HUGETLB_PAGE_ORDER +1 ]; +}; + +/* +* `kho_order_table` points to a page that serves as the root of the KHO page +* table hierarchy. This page is allocated during KHO module initialization. +* Its physical address is written to the FDT and passed to the next kernel +* during kexec. +*/ +static struct kho_order_table *kho_order_table; + +static unsigned long kho_page_table_level_shift (int level,int order) +{ +/* +* Calculate the cumulative bit shift required to extract the page table +* index for a given physical address at a specific `level` and `order`. +* +* - Level 1 is the bitmap table, which has its own indexing logic, so +* the shift is 0. +* - Level 2 and above: The base shift is `BITMAP_TABLE_SHIFT(order)`, +* which corresponds to the entire address space covered by a single +* level 1 bitmap table. +* - Each subsequent level adds `PAGE_TABLE_SHIFT_PER_LEVEL` to the +* total shift amount. +*/ +return level <=1 ?0 : +BITMAP_TABLE_SHIFT (order) +PAGE_TABLE_SHIFT_PER_LEVEL * (level -2 ); +} + +static int kho_get_bitmap_table_index (unsigned long pa,int order) +{ +/* 4KB (12bits of addr) + 8B per entries (6bits of addr) + order bits */ +unsigned long idx = pa >> (PAGE_SHIFT +6 + order); + +return idx; +} + +static int kho_get_page_table_index (unsigned long pa,int order,int level) +{ +unsigned long high_addr; +unsigned long page_table_offset; +unsigned long shift; + +if (level ==1 ) +return kho_get_bitmap_table_index (pa, order); + + shift =kho_page_table_level_shift (level, order); + high_addr = pa >> shift; + + page_table_offset = high_addr &PAGE_TABLE_LEVEL_MASK ; +return page_table_offset; +} + +static int kho_table_level (int order) +{ +unsigned long bits_to_resolve; +int page_table_num; + +/* We just need 1 bitmap table to cover all addresses */ +if (BITMAP_TABLE_SHIFT (order) >=64 ) +return 1 ; + + bits_to_resolve =64 -BITMAP_TABLE_SHIFT (order); + +/* +* The level we need is the bits to resolve over the bits a page tabel +* can resolve. Get the ceiling as ceil(a/b) = (a + b - 1) / b. +* Total level is the all table levels plus the buttom +* bitmap level. +*/ + page_table_num = (bits_to_resolve +PAGE_TABLE_SHIFT_PER_LEVEL -1 ) + /PAGE_TABLE_SHIFT_PER_LEVEL ; +return page_table_num +1 ; +} + +static struct kho_page_table *kho_alloc_page_table (void ) +{ +return (struct kho_page_table *)get_zeroed_page (GFP_KERNEL ); +} + +static void kho_set_preserved_page_bit (struct kho_bitmap_table *bitmap_table, +unsigned long pa,int order) +{ +int bitmap_table_index =kho_get_bitmap_table_index (pa, order); +int offset; + +/* Get the bit offset in a 64bits bitmap entry */ + offset = (pa >>PRESERVED_PAGE_OFFSET_SHIFT (order)) &0x3f ; + +set_bit (offset, + (unsigned long *)&bitmap_table->bitmaps[bitmap_table_index]); +} + +static unsigned long kho_pgt_desc (struct kho_page_table *va) +{ +return (unsigned long )virt_to_phys (va); +} + +static struct kho_page_table *kho_page_table (unsigned long desc) +{ +return (struct kho_page_table *)phys_to_virt (desc); +} + +static int __kho_preserve_page_table (unsigned long pa,int order) +{ +int num_table_level =kho_table_level (order); +struct kho_page_table *cur; +struct kho_page_table *next; +struct kho_bitmap_table *bitmap_table; +int i, page_table_index; +unsigned long page_table_desc; + +if (!kho_order_table->orders[order]) { + cur =kho_alloc_page_table (); +if (!cur) +return -ENOMEM ; + page_table_desc =kho_pgt_desc (cur); + kho_order_table->orders[order] = page_table_desc; + } + + cur =kho_page_table (kho_order_table->orders[order]); + +/* Go from high level tables to low level tables */ +for (i = num_table_level; i >1 ; i--) { + page_table_index =kho_get_page_table_index (pa, order, i); + +if (!cur->tables[page_table_index]) { + next =kho_alloc_page_table (); +if (!next) +return -ENOMEM ; + cur->tables[page_table_index] =kho_pgt_desc (next); + }else { + next =kho_page_table (cur->tables[page_table_index]); + } + + cur = next; + } + +/* Cur is now pointing to the level 1 bitmap table */ + bitmap_table = (struct kho_bitmap_table *)cur; +kho_set_preserved_page_bit (bitmap_table, + pa &BITMAP_TABLE_MASK (order), + order); + +return 0 ; +} + +/* +* TODO: __maybe_unused is added to the functions: +* kho_preserve_page_table() +* kho_walk_tables() +* kho_memblock_reserve() +* since they are not actually being called in this change. +* __maybe_unused will be removed in the next patch. +*/ +static __maybe_unused int kho_preserve_page_table (unsigned long pfn,int order) +{ +unsigned long pa =PFN_PHYS (pfn); + +might_sleep (); + +return __kho_preserve_page_table (pa, order); +} + +static int __kho_walk_bitmap_table (int order, +struct kho_bitmap_table *bitmap_table, +unsigned long pa, +kho_walk_callback_t cb) +{ +int i; +unsigned long offset; +int ret =0 ; +int order_factor =1 << order; +unsigned long *bitmap = (unsigned long *)bitmap_table; + +for_each_set_bit (i, bitmap,PAGE_SIZE *BITS_PER_BYTE ) { + offset = (unsigned long )PAGE_SIZE * order_factor * i; + ret =cb (offset + pa, order); +if (ret) +return ret; + } + +return 0 ; +} + +static int __kho_walk_page_tables (int order,int level, +struct kho_page_table *cur,unsigned long pa, +kho_walk_callback_t cb) +{ +struct kho_page_table *next; +struct kho_bitmap_table *bitmap_table; +int i; +unsigned long offset; +int ret =0 ; + +if (level ==1 ) { + bitmap_table = (struct kho_bitmap_table *)cur; +return __kho_walk_bitmap_table (order, bitmap_table, pa, cb); + } + +for (i =0 ; i <PTR_PER_LEVEL ; i++) { +if (cur->tables[i]) { + next =kho_page_table (cur->tables[i]); + offset = i; + offset <<=kho_page_table_level_shift (level, order); + ret =__kho_walk_page_tables (order, level -1 , + next, offset + pa, cb); +if (ret <0 ) +return ret; + } + } + +return 0 ; +} + +static __maybe_unused int kho_walk_page_tables (struct kho_page_table *top,int order, +kho_walk_callback_t cb) +{ +int num_table_level; + +if (top) { + num_table_level =kho_table_level (order); +return __kho_walk_page_tables (order, num_table_level, top,0 , cb); + } + +return 0 ; +} + +static __maybe_unused int kho_memblock_reserve (phys_addr_t pa,int order) +{ +int sz =1 << (order +PAGE_SHIFT ); +struct page *page =phys_to_page (pa); + +memblock_reserve (pa, sz); +memblock_reserved_mark_noinit (pa, sz); + page->private = order; + +return 0 ; +} +/* * Keep track of memory that is to be preserved across KHO. *
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC v1 1/4] kho: Introduce KHO page table data structures
Quoted 10 lines · Jason Miu
+ *
kho_order_table
+ * +-------------------------------+--------------------+
+ * | 0 order| 1 order| 2 order ... |HUGETLB_PAGE_ORDER|
+ * ++------------------------------+--------------------++ * | + * | + * v+ * ++------+
+ * | Lv6 |kho_page_table
+ * ++------+
I seem to remember suggesting this could be simplified without the special case 7h level table table for order.
Encode the phys address as:
(order << 51) | (phys >> (PAGE_SHIFT + order))
Then you don't need another table for order, the 64 bits encode everything consistently. Order can't be > 52 so it is only 6 bits, meaning the result fits into at most 57 bits.
Quoted 4 lines
+ * 63 62:54 53:45 44:36 35:27 26:0
+ * +--------+--------+--------+--------+--------+-----------------+
+ * | Lv 6 | Lv 5 | Lv 4 | Lv 3 | Lv 2 | Lv 1 (bitmap) |
+ * +--------+--------+--------+--------+--------+-----------------+
This isn't quite right, the 11:0 bits are must be zero and not used to index anything.
Adjusting to reflect the above math, it would be like this:
63:60 59:51 50:42 41:33 32:34 23:15 14:0
+-----+--------+--------+--------+--------+--------+-----------------+ | 0 | Lv 6 | Lv 5 | Lv 4 | Lv 3 | Lv 2 | Lv 1 (bitmap) | +-----+--------+--------+--------+--------+--------+-----------------+
The order level is just folded into lv 6
Quoted 2 lines
+ * For higher order pages, the bit fields for each level shift to the left by
+ * the page order.
This is probably an unncessary complexity. The table levels cost only 64 bytes, it isn't so valuable to eliminate them. So with the above math it shifts right not left. Level 1 is always the bitmap and it doesn't move around. I'd label this 0 in the code.
If you also fix the sizes to be 64 bytes and 4096 bytes regardless of PAGE_SIZE then everything is easy and fixed, while still efficient on higher PAGE_SIZE architectures.
Fruther, changing the formula to this:
(1 << (63 - PAGE_SHIFT - order)) | (phys >> (PAGE_SHIFT + order))
Will shift the overhead levels to the top of the radix tree and share them across all orders, higher PAGE_SIZE arches will just get a single lvl 5 and an unecessary lvl 6 - cost 64 extra bytes who cares.
kernel/kexec_handover.c static int __init kho_parse_enable(char *p) · line 140
Quoted 6 lines
kernel/kexec_handover.c +6
-0+#define BITMAP_TABLE_SHIFT (_order) (PAGE_SHIFT + PAGE_SHIFT + 3 + (_order)) +#define BITMAP_TABLE_MASK (_order) ((1ULL << BITMAP_TABLE_SHIFT(_order)) - 1) +#define PRESERVED_PAGE_OFFSET_SHIFT (_order) (PAGE_SHIFT + (_order)) +#define PAGE_TABLE_SHIFT_PER_LEVEL (ilog2(PAGE_SIZE / sizeof(unsigned long))) +#define PAGE_TABLE_LEVEL_MASK ((1ULL << PAGE_TABLE_SHIFT_PER_LEVEL) - 1) +#define PTR_PER_LEVEL (PAGE_SIZE / sizeof(unsigned long))
please use inlines and enums :(
It looks like if you make the above algorithm changes most of the this code is deleted.
Jason
Pasha Tatashin
maintainerRe: [RFC v1 1/4] kho: Introduce KHO page table data structures
Quoted 13 lines · Jason Gunthorpe
Quoted 10 lines · Jason Miu
+ *
kho_order_table
+ * +-------------------------------+--------------------+
+ * | 0 order| 1 order| 2 order ... |HUGETLB_PAGE_ORDER|
+ * ++------------------------------+--------------------++ * | + * | + * v+ * ++------+
+ * | Lv6 |kho_page_table
+ * ++------+
Quoted 6 lines
I seem to remember suggesting this could be simplified without the special case 7h level table table for order.
Encode the phys address as:
(order << 51) | (phys >> (
PAGE_SHIFT+ order))
Why 51 and not 52, this limits to 63bit address space, is it not?
Quoted 5 lines
Then you don't need another table for order, the 64 bits encode everything consistently. Order can't be > 52 so it is only 6 bits, meaning the result fits into at most 57 bits.
Hi Jason,
Nice packing. That's a really clever bit-packing scheme to create a unified address space.
I like the idea, but I'm trying to find the benefits compared to the current per-order tree approach.
1. Packing adds a slight performance overhead for higher orders. With the current approach, preserving higher order pages only requires a 3/4-level page table. With bit-packing proposal we will always have extra loads during preserve/unpreserve operations.
2. It also adds insignificant memory overhead, as extra levels will have a couple extra pages.
3. It slightly complicates the logic in the new kernel. Instead of simply iterating a known tree for a specific order, the boot-time walker would need to reconstruct the per-order subtrees, and walk them.
Perhaps I'm missing a key benefit of the unified tree? The current approach might not be as elegant as having everything packed into the same page table but it seems to be OK to me, and easy to understand.
Pasha
Re: [RFC v1 1/4] kho: Introduce KHO page table data structures
Quoted 14 lines · Pasha Tatashin
Quoted 13 lines · Jason Gunthorpe
Quoted 10 lines · Jason Miu
+ *
kho_order_table
+ * +-------------------------------+--------------------+
+ * | 0 order| 1 order| 2 order ... |HUGETLB_PAGE_ORDER|
+ * ++------------------------------+--------------------++ * | + * | + * v+ * ++------+
+ * | Lv6 |kho_page_table
+ * ++------+
Quoted 8 lines
Quoted 6 lines
I seem to remember suggesting this could be simplified without the special case 7h level table table for order.
Encode the phys address as:
(order << 51) | (phys >> (
PAGE_SHIFT+ order))Why 51 and not 52, this limits to 63bit address space, is it not?
Yeah, might have got the math off
Quoted 2 lines
I like the idea, but I'm trying to find the benefits compared to the current per-order tree approach.
It is probably about half the code compared to what I see here because everything is agressively simplified.
Quoted 4 lines
3. It slightly complicates the logic in the new kernel. Instead of simply iterating a known tree for a specific order, the boot-time walker would need to reconstruct the per-order subtrees, and walk them.
The core walker just runs over a range, it is easy to compute the range.
Jason
Re: [RFC v1 1/4] kho: Introduce KHO page table data structures
Hi Jason,
Quoted 25 lines · Jason Gunthorpe
Quoted 14 lines · Pasha Tatashin
Quoted 13 lines · Jason Gunthorpe
Quoted 10 lines · Jason Miu
+ *
kho_order_table
+ * +-------------------------------+--------------------+
+ * | 0 order| 1 order| 2 order ... |HUGETLB_PAGE_ORDER|
+ * ++------------------------------+--------------------++ * | + * | + * v+ * ++------+
+ * | Lv6 |kho_page_table
+ * ++------+Quoted 8 lines
Quoted 6 lines
I seem to remember suggesting this could be simplified without the special case 7h level table table for order.
Encode the phys address as:
(order << 51) | (phys >> (
PAGE_SHIFT+ order))Why 51 and not 52, this limits to 63bit address space, is it not?
Quoted 7 lines
Yeah, might have got the math off
Quoted 2 lines
I like the idea, but I'm trying to find the benefits compared to the current per-order tree approach.
It is probably about half the code compared to what I see here because everything is agressively simplified.
Thank you very much for providing feedback to me, and I think this is a very smart idea.
Quoted 7 lines
Quoted 4 lines
3. It slightly complicates the logic in the new kernel. Instead of simply iterating a known tree for a specific order, the boot-time walker would need to reconstruct the per-order subtrees, and walk them.
The core walker just runs over a range, it is easy to compute the range.
I believe the "range" here refers to the specific portion of the tree relevant to the target_order being restored, while the target_order is the variable from 0 to MAX_PAGE_ORDER to be used in the tree walk in the new kernel.
My current understanding of the walker for a given target_order:
1. Find the start_level from the target_order. (for example,target_order = 10, start_level = 4)
2. The path from the root down to the level above start_level is
fixed (index 0 at each of these levels).
3. At start_level, the index is also fixed, by (1 << (63 -PAGE_SHIFT - order)) in a 9 bit slice.
4. Then, for all levels *below* order_level, the walker iterates
through all 512 table entries, until the bitmap level.
so the "range" is the subtrees under the start_level, is my understanding correct?
--
Jason Miu
Re: [RFC v1 1/4] kho: Introduce KHO page table data structures
Quoted 8 lines
1. Find the
start_levelfrom thetarget_order. (for example,target_order= 10,start_level= 4)
2. The path from the root down to the level abovestart_levelis
fixed (index 0 at each of these levels).
3. Atstart_level, the index is also fixed, by (1 << (63 -PAGE_SHIFT- order)) in a 9 bit slice.
4. Then, for all levels *below*order_level, the walker iterates
through all 512 table entries, until the bitmap level.
You don't need any special logic like that, that is my point, the whole thing is very simple:
static int get_index(unsigned int level, u64 pos)
{
return (pos / (level * ITEMS_PER_TABLE * ITEMS_PER_BITMAP)) %
ITEMS_PER_TABLE;
}walk_table(u64 *table, unsigned int level, u64 start, u64 last)
{
unsigned int index = get_index(level, start);
unsigned int last_index = get_index(level, last);
do {
if (table[index]) {
u64 *next_table = phys_to_virt(table[index]);
if (level == 1)
walk_bitmap(next_table);
else
walk_table(next_table, level - 1, start, last);
}
index++;
} while (index <= last_index);
}insert_table(u64 *table, unsigned int level, u64 pos)
{
unsigned int index = get_index(level, start);
u64 *next_table;
if (!table[index]) {
// allocate table[index]
}
else
next_table = phys_to_virt(table[index]);
if (level == 1)
insert_bitmap(next_table, pos);
else
insert_table(next_table, level - 1, pos);
}That's it.. No special cases requried.
The above is very limited, it only works with certain formulations of start/last:
start has only one bit set
start & last == true,
last ^ start has bits 0 -> N set N > log2(ITEMS_PER_BITMAP)
Which align to my suggestion for encoding.
Jason
RFC 2/4 kho: Adopt KHO page tables and remove serialization
In reply to Jason MiuTransition the KHO system to use the new page table data structures for managing preserved memory, replacing the previous xarray-based approach. Remove the serialization process and the associated finalization and abort logic.
Update the methods for marking memory to be preserved to use the KHO page table hierarchy. Remove the former system of tracking preserved pages using an xarray-based structure.
Change the method of passing preserved memory information to the next kernel to be direct. Instead of serializing the memory map, place the physical address of the kho_order_table, which holds the roots of the KHO page tables for each order, in the FDT. Remove the explicit kho_finalize() and kho_abort() functions and the logic supporting the finalize and abort states, as they are no longer needed. This simplifies the KHO lifecycle.
Enable the next kernel's initialization process to read the kho_order_table address from the FDT. The kernel will then traverse the KHO page table structures to discover all preserved memory regions, reserving them to prevent early boot-time allocators from overwriting them.
This architectural shift to using a shared page table structure simplifies the KHO design and eliminates the overhead of serializing and deserializing the preserved memory map.
--- include/linux/kexec_handover.h | 17 -- kernel/kexec_handover.c | 532 +++++---------------------------- 2 files changed, 71 insertions(+), 478 deletions(-)
include/linux/kexec_handover.h +0 -17
@@ -19,23 +19,6 @@ enum kho_event {struct folio ;struct notifier_block ; -#define DECLARE_KHOSER_PTR (name, type) \ - union { \ - phys_addr_t phys; \ - type ptr; \ - } name -#define KHOSER_STORE_PTR (dest, val) \ - ({ \ - typeof(val) v = val; \ - typecheck(typeof((dest).ptr), v); \ - (dest).phys = virt_to_phys(v); \ - }) -#define KHOSER_LOAD_PTR (src) \ - ({ \ - typeof(src) s = src; \ - (typeof((s).ptr))((s).phys ? phys_to_virt((s).phys) : NULL); \ - }) -struct kho_serialization ;#ifdef CONFIG_KEXEC_HANDOVER
kernel/kexec_handover.c +71 -461
666 lines not on this page · raw message
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Mike Rapoport
maintainerRe: [RFC v1 2/4] kho: Adopt KHO page tables and remove serialization
Hi Jason,
Quoted 33 lines · Jason Miu
Transition the KHO system to use the new page table data structures for managing preserved memory, replacing the previous xarray-based approach. Remove the serialization process and the associated finalization and abort logic.
Update the methods for marking memory to be preserved to use the KHO page table hierarchy. Remove the former system of tracking preserved pages using an xarray-based structure.
Change the method of passing preserved memory information to the next kernel to be direct. Instead of serializing the memory map, place the physical address of the
kho_order_table, which holds the roots of the KHO page tables for each order, in the FDT. Remove the explicitkho_finalize()andkho_abort()functions and the logic supporting the finalize and abort states, as they are no longer needed. This simplifies the KHO lifecycle.Enable the next kernel's initialization process to read the
kho_order_tableaddress from the FDT. The kernel will then traverse the KHO page table structures to discover all preserved memory regions, reserving them to prevent early boot-time allocators from overwriting them.This architectural shift to using a shared page table structure simplifies the KHO design and eliminates the overhead of serializing and deserializing the preserved memory map.
Signed-off-by Jason Miu <[email protected]>--- include/linux/kexec_handover.h | 17 -- kernel/kexec_handover.c | 532 +++++---------------------------- 2 files changed, 71 insertions(+), 478 deletions(-)
kernel/kexec_handover.c static int __kho_preserve_page_table(unsigned long pa, int order) · line 300
Quoted 10 lines
kernel/kexec_handover.c +1
-9-/* -* TODO: __maybe_unused is added to the functions: -* kho_preserve_page_table() -* kho_walk_tables() -* kho_memblock_reserve() -* since they are not actually being called in this change. -* __maybe_unused will be removed in the next patch. -*/ -static __maybe_unused int kho_preserve_page_table (unsigned long pfn,int order) +static int kho_preserve_page_table (unsigned long pfn,int order)
Just merge this and the previous patch so that the patch will replace the current preservation mechanism with a new one.
Quoted 7 lines
{
kernel/kexec_handover.c +0
-0unsigned long pa =PFN_PHYS (pfn);@@ -365,8 +357,8 @@ static int __kho_walk_page_tables(int order, int level,return 0 ; }
...
kernel/kexec_handover.c int kho_preserve_folio(struct folio *folio) · line 756
Quoted 11 lines
kernel/kexec_handover.c +1
-5@@ -1023,12 +752,8 @@ int kho_preserve_folio(struct folio *folio) {const unsigned long pfn =folio_pfn (folio);const unsigned int order =folio_order (folio); -struct kho_mem_track *track = &kho_out.ser.track; - -if (kho_out.finalized) -return -EBUSY ; -return __kho_preserve_order (track, pfn, order); +return kho_preserve_page_table (pfn, order);
I don't think we should "rename" __kho_preserve_order() to kho_preserve_page_table(). __kho_preserve_order() could use the new data structure, or call the new implementation, but I don't see a reason to replace it.
Signature · 2 lines
Sincerely yours, Mike.
Re: [RFC v1 2/4] kho: Adopt KHO page tables and remove serialization
Hi Mike,
Thank you very much for your comments!
Quoted 5 lines · Mike Rapoport
Quoted 5 lines
-/*
- * TODO:__maybe_unusedis added to the functions:
- *kho_preserve_page_table()
- *kho_walk_tables()
- *kho_memblock_reserve()
Quoted 8 lines
kernel/kexec_handover.c static int __kho_preserve_page_table(unsigned long pa, int order) · line 300
Quoted 5 lines
- * since they are not actually being called in this change.
- *__maybe_unusedwill be removed in the next patch.
- */kernel/kexec_handover.c +1
-1-static __maybe_unused int kho_preserve_page_table (unsigned long pfn,int order) +static int kho_preserve_page_table (unsigned long pfn,int order)Just merge this and the previous patch so that the patch will replace the current preservation mechanism with a new one.
Sure I can do this.
Quoted 9 lines
Quoted 9 lines
kernel/kexec_handover.c +0
-4@@ -1023,12 +752,8 @@ int kho_preserve_folio(struct folio *folio) {const unsigned long pfn =folio_pfn (folio);const unsigned int order =folio_order (folio); -struct kho_mem_track *track = &kho_out.ser.track; - -if (kho_out.finalized) -return -EBUSY ;
Quoted 8 lines
kernel/kexec_handover.c int kho_preserve_folio(struct folio *folio) · line 756
Quoted 2 lines
kernel/kexec_handover.c +1
-1-return __kho_preserve_order (track, pfn, order); +return kho_preserve_page_table (pfn, order);I don't think we should "rename"
__kho_preserve_order()tokho_preserve_page_table().__kho_preserve_order()could use the new data structure, or call the new implementation, but I don't see a reason to replace it.
Ok, I prefer calling the new implementation, so it will look like:
kho_preserve_folio() -> __kho_preserve_order() -> __kho_preserve_page_table()
__kho_preserve_page_table() is the internal implementation ofkho_preserve_page_table() and we can remove the kho_preserve_page_table().
--
Jason Miu
kernel test robot
build botRe: [RFC v1 2/4] kho: Adopt KHO page tables and remove serialization
In reply to Jason Miukernel test robot noticed the following build warnings:
Hi Jason,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on rppt-memblock/for-next]
[also build test WARNING on linus/master v6.17-rc6]
[cannot apply to rppt-memblock/fixes akpm-mm/mm-everything next-20250918] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Jason-Miu/kho-Introduce-KHO-page-table-data-structures/20250917-105117 base: https://git.kernel.org/pub/scm/linux/kernel/git/rppt/memblock.git for-next patch link: https://lore.kernel.org/r/20250917025019.1585041-3-jasonmiu%40google.com patch subject: [RFC v1 2/4] kho: Adopt KHO page tables and remove serialization config: x86_64-buildonly-randconfig-003-20250919 (https://download.01.org/0day-ci/archive/20250919/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250919/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>, old ones prefixed by <<):
>> WARNING: modpost: vmlinux: section mismatch in reference: kho_memblock_reserve+0x3d (section: .text) -> __memblock_reserve (section: .init.text) >> WARNING: modpost: vmlinux: section mismatch in reference: kho_memblock_reserve+0x49 (section: .text) -> memblock_reserved_mark_noinit (section: .init.text)
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC 3/4 memblock: Remove KHO notifier usage
In reply to Jason MiuUpdate memblock to use direct KHO API calls for memory preservation.
Remove the KHO notifier registration and callbacks from the memblock subsystem. These notifiers were tied to the former KHO finalize and abort events, which are no longer used.
Memblock now preserves its reserve_mem regions and registers its metadata by calling kho_preserve_phys(), kho_preserve_folio(), and kho_add_subtree() directly within its initialization function. This is made possible by changes in the KHO core to complete the FDT at a later stage, during kexec.
--- include/linux/kexec_handover.h | 7 ++---- kernel/kexec_core.c | 4 +++ kernel/kexec_handover.c | 46 +++++++++++++++++++++------------- kernel/kexec_internal.h | 2 ++ mm/memblock.c | 46 ++++++++-------------------------- 5 files changed, 47 insertions(+), 58 deletions(-)
include/linux/kexec_handover.h +2 -5
@@ -19,15 +19,13 @@ enum kho_event {struct folio ;struct notifier_block ; -struct kho_serialization ; -#ifdef CONFIG_KEXEC_HANDOVER bool kho_is_enabled (void );int kho_preserve_folio (struct folio *folio);int kho_preserve_phys (phys_addr_t phys,size_t size);struct folio *kho_restore_folio (phys_addr_t phys); -int kho_add_subtree (struct kho_serialization *ser,const char *name,void *fdt); +int kho_add_subtree (const char *name,void *fdt);int kho_retrieve_subtree (const char *name,phys_addr_t *phys);int register_kho_notifier (struct notifier_block *nb);
@@ -58,8 +56,7 @@ static inline struct folio *kho_restore_folio(phys_addr_t phys)return NULL ; } -static inline int kho_add_subtree (struct kho_serialization *ser, -const char *name,void *fdt) +static inline int kho_add_subtree (const char *name,void *fdt) {return -EOPNOTSUPP ; }
kernel/kexec_core.c +4 -0
@@ -1147,6 +1147,10 @@ int kernel_kexec(void) goto Unlock; } + error =kho_commit_fdt (); +if (error) + goto Unlock; +#ifdef CONFIG_KEXEC_JUMP if (kexec_image->preserve_context) {/*
kernel/kexec_handover.c +28 -18
85 lines not on this page · raw message
kernel/kexec_internal.h +2 -0
@@ -46,6 +46,7 @@ struct kexec_buf;int kho_locate_mem_hole (struct kexec_buf *kbuf,int (*func)(struct resource *,void *));int kho_fill_kimage (struct kimage *image); +int kho_commit_fdt (void );#else static inline int kho_locate_mem_hole (struct kexec_buf *kbuf,int (*func)(struct resource *,void *))
@@ -54,5 +55,6 @@ static inline int kho_locate_mem_hole(struct kexec_buf *kbuf, }static inline int kho_fill_kimage (struct kimage *image) {return 0 ; } +static inline int kho_commit_fdt (void ) {return 0 ; }#endif /* CONFIG_KEXEC_HANDOVER */ #endif /* LINUX_KEXEC_INTERNAL_H */
mm/memblock.c +11 -35
70 lines not on this page · raw message
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
kernel test robot
build botRe: [RFC v1 3/4] memblock: Remove KHO notifier usage
kernel test robot noticed the following build warnings:
Hi Jason,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on rppt-memblock/for-next]
[also build test WARNING on linus/master v6.17-rc6 next-20250917] [cannot apply to rppt-memblock/fixes akpm-mm/mm-everything] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Jason-Miu/kho-Introduce-KHO-page-table-data-structures/20250917-105117 base: https://git.kernel.org/pub/scm/linux/kernel/git/rppt/memblock.git for-next patch link: https://lore.kernel.org/r/20250917025019.1585041-4-jasonmiu%40google.com patch subject: [RFC v1 3/4] memblock: Remove KHO notifier usage config: riscv-randconfig-002-20250918 (https://download.01.org/0day-ci/archive/20250918/[email protected]/config) compiler: riscv32-linux-gcc (GCC) 9.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250918/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from arch/riscv/include/asm/thread_info.h:11,
from include/linux/thread_info.h:60,
from include/asm-generic/preempt.h:5,
from ./arch/riscv/include/generated/asm/preempt.h:1,
from include/linux/preempt.h:79,
from include/linux/spinlock.h:56,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/slab.h:16,
from mm/memblock.c:11:
>> arch/riscv/include/asm/page.h:99: warning: "ARCH_PFN_OFFSET" redefined
99 | #define ARCH_PFN_OFFSET (PFN_DOWN((unsigned long)phys_ram_base))
|
In file included from mm/memblock.c:9:
include/asm-generic/memory_model.h:15: note: this is the location of the previous definition
15 | #define ARCH_PFN_OFFSET (0UL)
| vim +/ARCH_PFN_OFFSET +99 arch/riscv/include/asm/page.h
3335068f87217ea Alexandre Ghiti 2023-03-24 98a7407a1318a9d7b Alexandre Ghiti 2023-03-24 @99 #define ARCH_PFN_OFFSET (PFN_DOWN((unsigned long)phys_ram_base))07037db5d479f90 Palmer Dabbelt 2017-07-10 100
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC 4/4 kho: Remove notifier system infrastructure
In reply to Jason MiuRemove the KHO notifier system.
Eliminate the core KHO notifier API
functions (register_kho_notifier, unregister_kho_notifier), the kho_event enum, and the notifier chain head from KHO internal structures.
This infrastructure was used to support the now-removed finalize and abort states and is no longer required. Client subsystems now interact with KHO through direct API calls.
--- include/linux/kexec_handover.h | 20 -------------------- kernel/kexec_handover.c | 15 --------------- 2 files changed, 35 deletions(-)
include/linux/kexec_handover.h +0 -20
@@ -10,14 +10,7 @@ struct kho_scratch {phys_addr_t size; }; -/* KHO Notifier index */ -enum kho_event { -KEXEC_KHO_FINALIZE =0 , -KEXEC_KHO_ABORT =1 , -}; -struct folio ; -struct notifier_block ;#ifdef CONFIG_KEXEC_HANDOVER bool kho_is_enabled (void );
@@ -28,9 +21,6 @@ struct folio *kho_restore_folio(phys_addr_t phys);int kho_add_subtree (const char *name,void *fdt);int kho_retrieve_subtree (const char *name,phys_addr_t *phys); -int register_kho_notifier (struct notifier_block *nb); -int unregister_kho_notifier (struct notifier_block *nb); -void kho_memory_init (void ); void kho_populate(phys_addr_t fdt_phys, u64 fdt_len, phys_addr_t scratch_phys,
@@ -66,16 +56,6 @@ static inline int kho_retrieve_subtree(const char *name, phys_addr_t *phys)return -EOPNOTSUPP ; } -static inline int register_kho_notifier (struct notifier_block *nb) -{ -return -EOPNOTSUPP ; -} - -static inline int unregister_kho_notifier (struct notifier_block *nb) -{ -return -EOPNOTSUPP ; -} -static inline void kho_memory_init (void ) { }
kernel/kexec_handover.c +0 -15
@@ -16,7 +16,6 @@#include <linux/libfdt.h> #include <linux/list.h> #include <linux/memblock.h> -#include <linux/notifier.h> #include <linux/page-isolation.h> #include <asm/early_ioremap.h>
@@ -683,13 +682,11 @@ static int kho_debugfs_fdt_add(struct list_head *list, struct dentry *dir, }struct kho_out { -struct blocking_notifier_head chain_head;struct dentry *dir;struct kho_serialization ser; };static struct kho_out kho_out = { - .chain_head =BLOCKING_NOTIFIER_INIT (kho_out.chain_head), .ser = { .fdt_list =LIST_HEAD_INIT (kho_out.ser.fdt_list), },
@@ -727,18 +724,6 @@ int kho_add_subtree(const char *name, void *fdt) }EXPORT_SYMBOL_GPL (kho_add_subtree); -int register_kho_notifier (struct notifier_block *nb) -{ -return blocking_notifier_chain_register (&kho_out.chain_head, nb); -} -EXPORT_SYMBOL_GPL (register_kho_notifier); - -int unregister_kho_notifier (struct notifier_block *nb) -{ -return blocking_notifier_chain_unregister (&kho_out.chain_head, nb); -} -EXPORT_SYMBOL_GPL (unregister_kho_notifier); -/** * kho_preserve_folio - preserve a folio across kexec. * @folio: folio to preserve.
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Quoted 10 lines · Jason Miu
This series transitions KHO from an xarray-based metadata tracking system with serialization to using page table like data structures that can be passed directly to the next kernel.
The key motivations for this change are to:
- Eliminate the need for data serialization before kexec.
- Remove the former KHO state machine by deprecating the finalize and abort states.
- Pass preservation metadata more directly to the next kernel via the FDT.
Quoted 4 lines
The new approach uses a per-order page table structure (
kho_order_table,kho_page_table,kho_bitmap_table) to mark preserved pages. The physical address of the rootkho_order_tableis passed in the FDT, allowing the next kernel to reconstruct the preserved memory map.
It is not a "page table" structure, it is just a radix tree with bits as the leaf.
Jason
Pasha Tatashin
maintainerQuoted 12 lines · Jason Gunthorpe
Quoted 10 lines · Jason Miu
This series transitions KHO from an xarray-based metadata tracking system with serialization to using page table like data structures that can be passed directly to the next kernel.
The key motivations for this change are to:
- Eliminate the need for data serialization before kexec.
- Remove the former KHO state machine by deprecating the finalize and abort states.
- Pass preservation metadata more directly to the next kernel via the FDT.
Quoted 7 lines
Quoted 4 lines
The new approach uses a per-order page table structure (
kho_order_table,kho_page_table,kho_bitmap_table) to mark preserved pages. The physical address of the rootkho_order_tableis passed in the FDT, allowing the next kernel to reconstruct the preserved memory map.It is not a "page table" structure, it is just a radix tree with bits as the leaf.
To be fair above it is referred to as a page table *like* data structure, but I agree kho radix tree sounds like a good overall name for this, and it might make sense to rename from kho_page_table to kho_radix_tree in other places.
Quoted 2 lines
Jason
Quoted 11 lines · Jason Gunthorpe
Quoted 10 lines · Jason Miu
This series transitions KHO from an xarray-based metadata tracking system with serialization to using page table like data structures that can be passed directly to the next kernel.
The key motivations for this change are to:
- Eliminate the need for data serialization before kexec.
- Remove the former KHO state machine by deprecating the finalize and abort states.
- Pass preservation metadata more directly to the next kernel via the FDT.
Quoted 7 lines
Quoted 4 lines
The new approach uses a per-order page table structure (
kho_order_table,kho_page_table,kho_bitmap_table) to mark preserved pages. The physical address of the rootkho_order_tableis passed in the FDT, allowing the next kernel to reconstruct the preserved memory map.It is not a "page table" structure, it is just a radix tree with bits as the leaf.
Sounds like the IDA data structure. Maybe that API needs to be enhanced for this use case, but surely using the same data structure would be a good thing?
Pasha Tatashin
maintainerQuoted 18 lines · Matthew Wilcox
Quoted 16 lines · Jason Gunthorpe
Quoted 14 lines · Jason Miu
This series transitions KHO from an xarray-based metadata tracking system with serialization to using page table like data structures that can be passed directly to the next kernel.
The key motivations for this change are to:
- Eliminate the need for data serialization before kexec.
- Remove the former KHO state machine by deprecating the finalize and abort states.
- Pass preservation metadata more directly to the next kernel via the FDT.The new approach uses a per-order page table structure (
kho_order_table,kho_page_table,kho_bitmap_table) to mark preserved pages. The physical address of the rootkho_order_tableis passed in the FDT, allowing the next kernel to reconstruct the preserved memory map.
Quoted 6 lines
Quoted 2 lines
It is not a "page table" structure, it is just a radix tree with bits as the leaf.
Sounds like the IDA data structure. Maybe that API needs to be enhanced for this use case, but surely using the same data structure would be a good thing?
Normally, I would agree, but in this case, this has to be a simple data structure that, in the long run, is going to be stable between different kernel versions: the old and the next kernel must understand it. Therefore, relying on any external data structure would require the maintainers and other developers to be aware of this rather unusual kernel requirement. So, I think it is much better to keep this implementation private to KHO, whose only responsibility is reliably passing memory pages from the old kernel to the next kernel.
Mike Rapoport
maintainerHi Jason,
Quoted 9 lines · Jason Miu
This series transitions KHO from an xarray-based metadata tracking system with serialization to using page table like data structures that can be passed directly to the next kernel.
The key motivations for this change are to:
- Eliminate the need for data serialization before kexec.
- Remove the former KHO state machine by deprecating the finalize and abort states.
- Pass preservation metadata more directly to the next kernel via the FDT.
If we pass the preservation metadata directly between the kernels, it means that any change to that data structure will break compatibility between the new and old kernels. With serialization this is less severe because a more recent kernel can relatively easy have backward compatible deserialization.
I'm all for removing KHO state machine, but that does not necessarily mean we must remove the serialization of memory persistence metadata?
For example, we can do the serialization at kernel_kexec() time and if we want to avoid memory allocations there we might preallocate pages for khoser_mem_chunk's as amount of bitmaps grow.
It also would be interesting to see how much time is saved if we remove the serialization.
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerQuoted 13 lines · Mike Rapoport
Hi Jason,
Quoted 9 lines · Jason Miu
This series transitions KHO from an xarray-based metadata tracking system with serialization to using page table like data structures that can be passed directly to the next kernel.
The key motivations for this change are to:
- Eliminate the need for data serialization before kexec.
- Remove the former KHO state machine by deprecating the finalize and abort states.
- Pass preservation metadata more directly to the next kernel via the FDT.
Quoted 7 lines
If we pass the preservation metadata directly between the kernels, it means that any change to that data structure will break compatibility between the new and old kernels. With serialization this is less severe because a more recent kernel can relatively easy have backward compatible deserialization.
I'm all for removing KHO state machine, but that does not necessarily mean we must remove the serialization of memory persistence metadata?
I think the tables should be treated as the final serialized data structure, and should get all the same properties that other KHO serialization formats have like stable binary format, versioning, etc.
It just so happens that the table format lends itself very well to being serialized on-the-go. When a page is marked as preserved during normal operation, it is very simple to just allocate all the intermediate levels and mark the page as reserved. There is no further processing needed to "serialize" it -- like we need to do with the bitmaps today.
So I don't really see why we should introduce an intermediate processing step when it is easy to just directly build the serialized data structure during normal operation.
Quoted 7 lines
For example, we can do the serialization at
kernel_kexec()time and if we want to avoid memory allocations there we might preallocate pages for khoser_mem_chunk's as amount of bitmaps grow.It also would be interesting to see how much time is saved if we remove the serialization.
Signature · 2 lines
Regards, Pratyush Yadav
Quoted 3 lines · Pratyush Yadav
I think the tables should be treated as the final serialized data structure, and should get all the same properties that other KHO serialization formats have like stable binary format, versioning, etc.
Right, that's how I see it too
Jason