kho: make boot time huge page allocation work nicely with KHO
Subsystems (5)
Pratyush Yadav
maintainerv3 cover kho: make boot time huge page allocation work nicely with KHO
From: "Pratyush Yadav (Google)" <[email protected]>
Hi,
Gigantic huge page allocation is somewhat broken currently with KHO.
First, they break scratch size accounting. Since they are allocated using the memblock alloc APIs, they count towards RSRV_KERN, and this scratch size when using scratch_scale. This means if huge pages take a large enough chunk of system memory scratch size will blow up and fail to allocate.
Second, scratch can not contain preserved memory, and if hugepages are allocated from scratch, they will fail to be preserved with the upcoming hugetlb preservation series [0].
Fix this by introducing the concept of extended scratch areas. They are areas that the kernel discovers on boot by walking the radix tree and finding free memory ranges. See patch 10 for more details.
Discovering the scratch areas needs some preparatory changes to KHO, the radix tree APIs, and to memblock. Patches 1-17 do that.
Patch 18 adds the scratch discovery logic.
Patch 19 adds the dedicated memblock hugetlb allocator.
Patch 20-21 fix the scratch size calculation with using scratch_scale.
This series is based on top of the liveupdate/kho-scratch branch.
[0] https://lore.kernel.org/linux-mm/[email protected]/T/#u
Changes in v3:
More notable changes:
- Refactor how key width and table depth is calculated.
- I noticed that extended scratch breaks migrate type init during testing. Below two changes fix it:
- Remove memblock_is_kho_scratch_memory() and kho_scratch_migratetype(). Instead, add kho_init_scratch_migratetype() and rework how migrate type for scratch pages is initialized.
- Move migrate type initialization from deferred_free_pages() to deferred_init_pages().
Other changes:
- Use kho_get_mem_map() directly in kho_populate() and add a comment on why it is okay to use.
- Don't say "export kho_scratch_overlap()".
- Add a comment in kho_extend_scratch() that it should always be called after efi_init()::reserve_regions().
- Get rid of memblock_reserved_size_flags(). Use memblock_reserved_hugetlb_size() instead.
- Define KHO_EXT_BLKSIZE to SZ_1G and add a comment explaining why the size was picked.
- Fix memblock tests build.
- Rename kho_ext_walk_key() to kho_ext_walk_leaf().
- Add some comments around the kho_ext tree callbacks to clarify which tree the function is called for and which one it uses.
- Move initialization of kho_in.radix_tree to kho_memory_init_early().
- Do not check if the tree is already initialized in kho_radix_init_tree().
Changes in v2:
Detailed changelog below.
At a high level, the major change in this version is to remove MEMBLOCK_KHO_SCRATCH_EXT. Keep MEMBLOCK_KHO_SCRATCH as the only memory type and mark the discovered areas with it. For HugeTLB, add a dedicated allocation routine and if allocated memory lands in scratch, do a retry. Also introduce MEMBLOCK_RSRV_HUGETLB to help with accounting of scratch area sizes.
- Fixup commit message in patch 1 to make namespacing change clearer.
- Use @key in kernel-doc for radix functions.
- Add a runtime check on key width.
- Move all mem retrieval logic to kho_mem_retrieve().
- Add a comment in kho_mem_retrieve() explaining why mem_map won't be NULL.
- Rename callbacks to ->leaf() and ->node().
- Fixup commit messages.
- Clear tree->root in kho_radix_destroy_tree(). This lets the tree be re-initialized by calling kho_radix_init_tree()
- Add kho_get_mem_map() earlier in the series.
- Export kho_scratch_overlap() and use it in memblock_is_kho_scratch_memory().
- Get rid of MEMBLOCK_KHO_SCRATCH_EXT.
- Introduce MEMBLOCK_RSRV_HUGETLB.
- Introduce memblock_alloc_hugetlb() for hugetlb bootmem allocations.
- Refactor memblock_reserved_kern_size() to allow calculating size by flags.
- Exclude hugetlb memory from scratch size calculation.
- Collect R-bys.
Regards,
Pratyush Yadav
Pratyush Yadav (Google) (21):
kho: generalize radix tree APIs
kho: make radix max key width more obvious
kho: disallow wide keys in radix tree
kho: return virtual address of mem_map
kho: store incoming radix tree in kho_in
kho: move all memory retrieval logic to kho_mem_retrieve()
kho: add a struct for radix callbacks
kho: add callback for table pages
kho: add data argument to radix walk callback
kho: allow early-boot usage of the KHO radix tree
kho: allow destroying KHO radix tree
kho: add kho_radix_init_tree()
kho: expose kho_scratch_overlap() to kexec_handover.h
kho: initialize kho_scratch pointer earlier in boot
kho: initialize preserved memory map radix tree earlier mm/mm_init: init deferred page migratetype in deferred_init_pages() mm/mm_init: don't rely on memblock to get KHO scratch migratetype
kho: extend scratch
memblock: make HugeTLB bootmem allocation work with KHO
memblock: add memblock_reserved_hugetlb_size()
kho: exclude hugetlb memory from scratch size calculation
include/linux/kexec_handover.h | 16 + include/linux/kho/abi/kexec_handover.h | 9 +- include/linux/kho_radix_tree.h | 44 +- include/linux/memblock.h | 23 +- kernel/liveupdate/Makefile | 1 - kernel/liveupdate/kexec_handover.c | 549 +++++++++++++++----- kernel/liveupdate/kexec_handover_debug.c | 25 - kernel/liveupdate/kexec_handover_internal.h | 9 - mm/hugetlb.c | 22 +- mm/memblock.c | 144 +++-- mm/mm_init.c | 59 ++- tools/testing/memblock/internal.h | 5 + 12 files changed, 645 insertions(+), 261 deletions(-) delete mode 100644 kernel/liveupdate/kexec_handover_debug.c
base-commit: 08412b8c707fdbccb7bf2116f0554fe09113cd53
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 1/21 kho: generalize radix tree APIs
From: "Pratyush Yadav (Google)" <[email protected]>
The KHO radix tree is a data structure that can track the presence or absence of an arbitrary key, with nothing inherently tied to KHO memory preservation tracking. This was one of the design goals of the radix tree. This was done to enable it to be re-used by other users of KHO.
Despite that, the radix tree APIs are very closely tied to KHO memory preservation tracking. Adding a key is done by kho_radix_add_page(), which encodes it as a page tracking operation and takes in PFN and order. kho_radix_del_page() does the same. These functions encode the key internally that goes into the radix tree. kho_radix_walk_tree() does the same by baking the PFN and order into the callback arguments.
Generalize the APIs by taking the key directly and doing the encoding at the callers. Rename the functions to kho_radix_add_key() and kho_radix_del_key(). In practice, this removes a line each from the functions and moves the encoding function call to the callers. Similarly, update kho_radix_tree_walk_callback_t to take the key directly.
Now that key encoding is no longer an inherent part of the radix tree and can be decided by the user, rename kho_radix_{encode,decode}_key() to kho_{encode,decode}_radix_key(). This moves them out of the "kho_radix_" name space into the "kho_" namespace. This emphasizes that this is KHO's way of encoding the key for its radix tree.
--- include/linux/kho_radix_tree.h | 18 +++---- kernel/liveupdate/kexec_handover.c | 76 ++++++++++++++---------------- 2 files changed, 42 insertions(+), 52 deletions(-)
include/linux/kho_radix_tree.h +6 -12
@@ -34,30 +34,24 @@ struct kho_radix_tree {struct mutex lock;/* protects the tree's structure and root pointer */ }; -typedef int (*kho_radix_tree_walk_callback_t )(phys_addr_t phys, -unsigned int order); +typedef int (*kho_radix_tree_walk_callback_t )(unsigned long key);#ifdef CONFIG_KEXEC_HANDOVER -int kho_radix_add_page (struct kho_radix_tree *tree,unsigned long pfn, -unsigned int order); - -void kho_radix_del_page (struct kho_radix_tree *tree,unsigned long pfn, -unsigned int order); - +int kho_radix_add_key (struct kho_radix_tree *tree,unsigned long key); +void kho_radix_del_key (struct kho_radix_tree *tree,unsigned long key);int kho_radix_walk_tree (struct kho_radix_tree *tree,kho_radix_tree_walk_callback_t cb);#else /* #ifdef CONFIG_KEXEC_HANDOVER */ -static inline int kho_radix_add_page (struct kho_radix_tree *tree,long pfn, -unsigned int order) +static inline int kho_radix_add_key (struct kho_radix_tree *tree,unsigned long key) {return -EOPNOTSUPP ; } -static inline void kho_radix_del_page (struct kho_radix_tree *tree, -unsigned long pfn,unsigned int order) { } +static inline void kho_radix_del_key (struct kho_radix_tree *tree, +unsigned long key) { }static inline int kho_radix_walk_tree (struct kho_radix_tree *tree,kho_radix_tree_walk_callback_t cb)
kernel/liveupdate/kexec_handover.c +36 -40
190 lines not on this page · raw message
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 2/21 kho: make radix max key width more obvious
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
The KHO radix tree constants are somewhat hard to understand. The tree depth essentially comes from the max key width. The max key width comes from the need to store a 52-bit PFN plus one more bit for the order.
All this is very obscure with the corrent code. The PFN width is defined as KHO_ORDER_0_LOG2, which makes very little sense to a new reader not already familiar with what the value means. Then the fact that an extra bit is needed is hidden in the KHO_TREE_MAX_DEPTH calculation.
Simplify this by removing KHO_ORDER_0_LOG2 and replace it with KHO_RADIX_KEY_WIDTH. Update the comment to explain why this value is used. This moves the +1 from KHO_TREE_MAX_DEPTH to KHO_RADIX_KEY_WIDTH, making things clearer.
Update kho_{encode,decode}_radix_key() to not use KHO_ORDER_0_LOG2. Instead, refactor the code and comments to make it clearer how the encoding and decoding is done.
In kho_encode_radix_key(), add a new variable for the shift for phys. Use that in calculating where the order bit goes and in calculating the shifted PFN. Update comments to make this clearer.
In kho_radix_decode_key(), turn order_bit to 0-indexed to simplify the eventual calculation for order. Touch up comments to make the computation clearer.
--- include/linux/kho/abi/kexec_handover.h | 9 +++------ kernel/liveupdate/kexec_handover.c | 19 +++++++++++-------- 2 files changed, 14 insertions(+), 14 deletions(-)
include/linux/kho/abi/kexec_handover.h +3 -6
@@ -257,11 +257,8 @@ struct kho_vmalloc {* memory. These constants govern the indexing, sizing, and depth of the tree. */ enum kho_radix_consts { -/* -* The bit position of the order bit (and also the length of the -* shifted physical address) for an order-0 page. -*/ -KHO_ORDER_0_LOG2 =64 -PAGE_SHIFT , +/* Need to store the PFN, plus one bit for order. */ +KHO_RADIX_KEY_WIDTH =64 -PAGE_SHIFT +1 ,/* Size of the table in kho_radix_node, in log2 */ KHO_TABLE_SIZE_LOG2 =const_ilog2 (PAGE_SIZE /sizeof (phys_addr_t)),
@@ -274,7 +271,7 @@ enum kho_radix_consts {* and 1 bitmap level. */ KHO_TREE_MAX_DEPTH = -DIV_ROUND_UP (KHO_ORDER_0_LOG2 -KHO_BITMAP_SIZE_LOG2 +1 , +DIV_ROUND_UP (KHO_RADIX_KEY_WIDTH -KHO_BITMAP_SIZE_LOG2 , KHO_TABLE_SIZE_LOG2) + 1, };
kernel/liveupdate/kexec_handover.c +11 -8
@@ -97,10 +97,12 @@ static struct kho_out kho_out = {*/ static unsigned long kho_encode_radix_key (phys_addr_t phys,unsigned int order) { -/* Order bits part */ -unsigned long h =1UL << (KHO_ORDER_0_LOG2 - order); -/* Shifted physical address part */ -unsigned long l = phys >> (PAGE_SHIFT + order); +/* The physical address is encoded by shifting the PFN by its order. */ +unsigned long shift =PAGE_SHIFT + order; +/* Order bit goes right before the shifted PFN. */ +unsigned long h =1UL << (64 - shift); +/* Shifted PFN. */ +unsigned long l = phys >> shift;return h | l; }
@@ -118,12 +120,13 @@ static unsigned long kho_encode_radix_key(phys_addr_t phys, unsigned int order)*/ static phys_addr_t kho_decode_radix_key (unsigned long key,unsigned int *order) { -unsigned int order_bit =fls64 (key); +/* fls64() indexes starting from 1. */ +unsigned int order_bit =fls64 (key) -1 ;phys_addr_t phys; -/* order_bit is numbered starting at 1 from fls64 */ - *order =KHO_ORDER_0_LOG2 - order_bit +1 ; -/* The order is discarded by the shift */ +/* order bit goes right before the shifted PFN. */ + *order =64 - (PAGE_SHIFT + order_bit); +/* The order bit is discarded by the shift */ phys = key << (PAGE_SHIFT + *order);return phys;
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Mike Rapoport
maintainerRe: [PATCH v3 02/21] kho: make radix max key width more obvious
Quoted 14 lines
The KHO radix tree constants are somewhat hard to understand. The tree depth essentially comes from the max key width. The max key width comes from the need to store a 52-bit PFN plus one more bit for the order.
All this is very obscure with the corrent code. The PFN width is defined as
KHO_ORDER_0_LOG2, which makes very little sense to a new reader not already familiar with what the value means. Then the fact that an extra bit is needed is hidden in theKHO_TREE_MAX_DEPTHcalculation.Simplify this by removing
KHO_ORDER_0_LOG2and replace it withKHO_RADIX_KEY_WIDTH. Update the comment to explain why this value is used. This moves the +1 fromKHO_TREE_MAX_DEPTHtoKHO_RADIX_KEY_WIDTH, making things clearer.
Quoted 5 lines
Update kho_{encode,decode}_radix_key() to not use
KHO_ORDER_0_LOG2. Instead, refactor the code and comments to make it clearer how the encoding and decoding is done.In
kho_encode_radix_key(), add a new variable for the shift for phys.
Nit: physical address ^
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerPATCH v3 3/21 kho: disallow wide keys in radix tree
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
The KHO radix tree was designed to track preserved pages. So it does not provide the capability to track any 64-bit key. Instead, it limits the key width to how much it needs for tracking PFNs and their orders. Limiting the width reduces the number of levels in the tree.
KHO is not expected to be the only user of the radix tree. With the API generalized to allow other users, now it is possible to add any key to the tree.
Check the key width at kho_radix_add_key(), and error out if it exceeds what the tree can handle. Do this instead of increasing the tree depth since right now there are no users that need to use wider keys, so this avoids memory overhead and ABI breakage.
--- kernel/liveupdate/kexec_handover.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/liveupdate/kexec_handover.c +12 -0
@@ -156,6 +156,11 @@ static unsigned long kho_radix_get_table_index(unsigned long key,* intermediate nodes do not exist along the path, they are allocated and added * to the tree. * +* NOTE: Currently only keys of width up to %KHO_RADIX_KEY_WIDTH are supported. +* This limit only exists because current users of the radix tree don't use more +* than that. Changing the maximum width requires changing the tree depth, which +* needs bumping the ABI version. +* * Return: 0 on success, or a negative error code on failure. */ int kho_radix_add_key (struct kho_radix_tree *tree,unsigned long key)
@@ -172,6 +177,9 @@ int kho_radix_add_key(struct kho_radix_tree *tree, unsigned long key)if (WARN_ON_ONCE (!tree->root))return -EINVAL ; +if (unlikely (fls64 (key) >KHO_RADIX_KEY_WIDTH )) +return -ERANGE ; +might_sleep ();guard (mutex)(&tree->lock);
@@ -244,6 +252,10 @@ void kho_radix_del_key(struct kho_radix_tree *tree, unsigned long key)if (WARN_ON_ONCE (!tree->root))return ; +/* Keys wider than KHO_RADIX_KEY_WIDTH are not allowed to be added. */ +if (unlikely (fls64 (key) >KHO_RADIX_KEY_WIDTH )) +return ; +might_sleep ();guard (mutex)(&tree->lock);
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 4/21 kho: return virtual address of mem_map
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Currently it is only used by kho_populate(), which doesn't care whether the address is virtual or physical and only cares that it exists and is valid. In coming patches, more callers will be added, all of which will need the virtual address.
Make things simpler by directly returning the virtual address. Rename kho_get_mem_map_phys() to kho_get_mem_map() to accurately reflect what it returns.
--- kernel/liveupdate/kexec_handover.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)
kernel/liveupdate/kexec_handover.c +17 -8
@@ -512,19 +512,24 @@ static int __init kho_preserved_memory_reserve(unsigned long key)return 0 ; } -/* Returns physical address of the preserved memory map from FDT */ -static phys_addr_t __init kho_get_mem_map_phys (const void *fdt) +/* Returns virtual address of the preserved memory map from FDT */ +static __init void *kho_get_mem_map (const void *fdt) {const void *mem_ptr; +phys_addr_t mem_map_phys;int len; mem_ptr =fdt_getprop (fdt,0 ,KHO_FDT_MEMORY_MAP_PROP_NAME , &len);if (!mem_ptr || len !=sizeof (u64)) {pr_err ("failed to get preserved memory map\n" ); -return 0 ; +return NULL ; } -return get_unaligned ((const u64 *)mem_ptr); + mem_map_phys =get_unaligned ((const u64 *)mem_ptr); +if (!mem_map_phys) +return NULL ; + +return phys_to_virt (mem_map_phys); }/*
@@ -1647,9 +1652,8 @@ void __init kho_populate(phys_addr_t fdt_phys, u64 fdt_len, {unsigned int scratch_cnt = scratch_len /sizeof (*kho_scratch);struct kho_scratch *scratch =NULL ; -phys_addr_t mem_map_phys; -void *fdt =NULL ;bool populated = false; +void *fdt =NULL ;int err;/* Validate the input FDT */
@@ -1671,8 +1675,13 @@ void __init kho_populate(phys_addr_t fdt_phys, u64 fdt_len, goto unmap_fdt; } - mem_map_phys =kho_get_mem_map_phys (fdt); -if (!mem_map_phys) +/* +* At this point phys_to_virt() doesn't work properly and so +* kho_get_mem_map() can return a pre-KASLR virtual address. But here we +* only want to make sure the mem_map is valid so the actual value +* doesn't matter as long as it isn't NULL. +*/ +if (!kho_get_mem_map (fdt)) goto unmap_fdt; scratch =early_memremap (scratch_phys, scratch_len);
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Mike Rapoport
maintainerRe: [PATCH v3 04/21] kho: return virtual address of mem_map
Quoted 1 line
kho: return virtual address of
mem_map
It's not really clear from where ...
Quoted 1 line
Currently it is only used by
kho_populate(), which doesn't care whether
... and where it is used ;-)
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerRe: [PATCH v3 04/21] kho: return virtual address of mem_map
Quoted 7 lines · Mike Rapoport
Quoted 1 line
kho: return virtual address of
mem_mapIt's not really clear from where ...
Quoted 1 line
Currently it is only used by
kho_populate(), which doesn't care whether... and where it is used ;-)
Here's what I changed the whole thing to:
kho: return virtual address of mem_map from kho_get_mem_map()
Currently the preserved memory map address is returned by kho_get_mem_map_phys(). It is only used by kho_populate(). kho_populate() doesn't use the actual value. It only cares that the address exists and is valid.
In coming patches, more callers will be added, all of which will need the virtual address of the preserved memory map. Since kho_populate() doesn't care about the actual value and only cares about validity, it can also use the virtual address returned by kho_get_mem_map(). It only needs to make sure the returned value is not NULL.
Rename kho_get_mem_map_phys() to kho_get_mem_map() and return the virtual address of the preserved memory map.
Signature · 2 lines
Regards, Pratyush Yadav
Pratyush Yadav
maintainerPATCH v3 5/21 kho: store incoming radix tree in kho_in
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
This allows other functions to also use the radix tree. While at it, also use kho_get_mem_map_phys() instead of duplicating the code to get the radix tree root from the FDT.
--- kernel/liveupdate/kexec_handover.c | 32 ++++++++++++------------------ 1 file changed, 13 insertions(+), 19 deletions(-)
kernel/liveupdate/kexec_handover.c +13 -19
@@ -1337,6 +1337,7 @@ struct kho_in {char previous_release[__NEW_UTS_LEN +1 ];u32 kexec_count;struct kho_debugfs dbg; +struct kho_radix_tree radix_tree; };static struct kho_in kho_in = {
@@ -1416,24 +1417,15 @@ EXPORT_SYMBOL_GPL(kho_retrieve_subtree);static int __init kho_mem_retrieve (const void *fdt) { -struct kho_radix_tree tree; -const phys_addr_t *mem; -int len; - -/* Retrieve the KHO radix tree from passed-in FDT. */ - mem =fdt_getprop (fdt,0 ,KHO_FDT_MEMORY_MAP_PROP_NAME , &len); - -if (!mem || len !=sizeof (*mem)) { -pr_err ("failed to get preserved KHO memory tree\n" ); -return -ENOENT ; - } - -if (!*mem) -return -EINVAL ; - - tree.root =phys_to_virt (*mem); -mutex_init (&tree.lock); -return kho_radix_walk_tree (&tree, kho_preserved_memory_reserve); +/* +* kho_get_mem_map() should always succeed. If it fails, kho_populate() +* catches that and never sets kho_in.scratch_phys, which stops memory +* retrieval. +*/ + kho_in.radix_tree.root =kho_get_mem_map (fdt); +mutex_init (&kho_in.radix_tree.lock); +return kho_radix_walk_tree (&kho_in.radix_tree, + kho_preserved_memory_reserve); }static __init int kho_out_fdt_setup (void )
@@ -1640,8 +1632,10 @@ void __init kho_memory_init(void)if (kho_in.scratch_phys) { kho_scratch =phys_to_virt (kho_in.scratch_phys); -if (kho_mem_retrieve (kho_get_fdt ())) +if (kho_mem_retrieve (kho_get_fdt ())) { kho_in.fdt_phys =0 ; + kho_in.radix_tree.root =NULL ; + } }else {kho_reserve_scratch (); }
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 6/21 kho: move all memory retrieval logic to kho_mem_retrieve()
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
The memory retrieval logic is spread out across kho_mem_retrieve() and kho_memory_init(). The incoming scratch area is initialized at kho_memory_init(), and the error handling is done there too.
Consolidate all this logic into kho_mem_retrieve() to make the code cleaner.
--- kernel/liveupdate/kexec_handover.c | 31 ++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-)
kernel/liveupdate/kexec_handover.c +19 -12
@@ -1415,8 +1415,13 @@ int kho_retrieve_subtree(const char *name, phys_addr_t *phys, size_t *size) }EXPORT_SYMBOL_GPL (kho_retrieve_subtree); -static int __init kho_mem_retrieve (const void *fdt) +static void __init kho_mem_retrieve (void ) { +const void *fdt =kho_get_fdt (); +int err; + + kho_scratch =phys_to_virt (kho_in.scratch_phys); +/* * kho_get_mem_map() should always succeed. If it fails, kho_populate() * catches that and never sets kho_in.scratch_phys, which stops memory
@@ -1424,8 +1429,16 @@ static int __init kho_mem_retrieve(const void *fdt)*/ kho_in.radix_tree.root =kho_get_mem_map (fdt);mutex_init (&kho_in.radix_tree.lock); -return kho_radix_walk_tree (&kho_in.radix_tree, - kho_preserved_memory_reserve); + + err =kho_radix_walk_tree (&kho_in.radix_tree, kho_preserved_memory_reserve); +if (err) { +/* +* Failed to initialize preserved memory. Clear FDT and radix +* so KHO users don't treat it as a KHO boot. +*/ + kho_in.fdt_phys =0 ; + kho_in.radix_tree.root =NULL ; + } }static __init int kho_out_fdt_setup (void )
@@ -1629,16 +1642,10 @@ fs_initcall(kho_init);void __init kho_memory_init (void ) { -if (kho_in.scratch_phys) { - kho_scratch =phys_to_virt (kho_in.scratch_phys); - -if (kho_mem_retrieve (kho_get_fdt ())) { - kho_in.fdt_phys =0 ; - kho_in.radix_tree.root =NULL ; - } - }else { +if (kho_in.scratch_phys) +kho_mem_retrieve (); +else kho_reserve_scratch (); - } } void __init kho_populate(phys_addr_t fdt_phys, u64 fdt_len,
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 7/21 kho: add a struct for radix callbacks
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
A future commit will add more callbacks for the KHO radix tree. Add a struct for collecting the callbacks.
--- include/linux/kho_radix_tree.h | 15 ++++++++++++--- kernel/liveupdate/kexec_handover.c | 27 +++++++++++++++------------ 2 files changed, 27 insertions(+), 15 deletions(-)
include/linux/kho_radix_tree.h +12 -3
@@ -34,14 +34,23 @@ struct kho_radix_tree {struct mutex lock;/* protects the tree's structure and root pointer */ }; -typedef int (*kho_radix_tree_walk_callback_t )(unsigned long key); +/** +* struct kho_radix_walk_cb - Callbacks for KHO radix tree walk. +* @leaf: Called on each present key in the radix tree. +* +* For each callback, a return value of 0 continues the walk and a non-zero +* return value is directly returned to the caller. +*/ +struct kho_radix_walk_cb { +int (*leaf)(unsigned long key); +};#ifdef CONFIG_KEXEC_HANDOVER int kho_radix_add_key (struct kho_radix_tree *tree,unsigned long key);void kho_radix_del_key (struct kho_radix_tree *tree,unsigned long key);int kho_radix_walk_tree (struct kho_radix_tree *tree, -kho_radix_tree_walk_callback_t cb); +const struct kho_radix_walk_cb *cb);#else /* #ifdef CONFIG_KEXEC_HANDOVER */
@@ -54,7 +63,7 @@ static inline void kho_radix_del_key(struct kho_radix_tree *tree, unsigned long key) { }static inline int kho_radix_walk_tree (struct kho_radix_tree *tree, -kho_radix_tree_walk_callback_t cb) +const struct kho_radix_walk_cb *cb) {return -EOPNOTSUPP ; }
kernel/liveupdate/kexec_handover.c +15 -12
@@ -281,16 +281,18 @@ void kho_radix_del_key(struct kho_radix_tree *tree, unsigned long key) }EXPORT_SYMBOL_GPL (kho_radix_del_key); -static int kho_radix_walk_leaf (struct kho_radix_leaf *leaf, -unsigned long key, -kho_radix_tree_walk_callback_t cb) +static int kho_radix_walk_leaf (struct kho_radix_leaf *leaf,unsigned long key, +const struct kho_radix_walk_cb *cb) {unsigned long *bitmap = (unsigned long *)leaf;unsigned int i;int err; +if (!cb->leaf) +return 0 ; +for_each_set_bit (i, bitmap,PAGE_SIZE *BITS_PER_BYTE ) { - err =cb (key | i); + err = cb->leaf (key | i);if (err)return err; }
@@ -300,7 +302,7 @@ static int kho_radix_walk_leaf(struct kho_radix_leaf *leaf,static int __kho_radix_walk_tree (struct kho_radix_node *root,unsigned int level,unsigned long start, -kho_radix_tree_walk_callback_t cb) +const struct kho_radix_walk_cb *cb) {struct kho_radix_node *node;struct kho_radix_leaf *leaf;
@@ -340,18 +342,16 @@ static int __kho_radix_walk_tree(struct kho_radix_node *root,/** * kho_radix_walk_tree - Traverses the radix tree and calls a callback for each key. * @tree: A pointer to the KHO radix tree to walk. -* @cb: A callback function of type kho_radix_tree_walk_callback_t that will be -* invoked for each key in the tree. +* @cb: Set of callbacks to be invoked during the tree walk. * -* This function walks the radix tree, searching from the specified top level -* down to the lowest level (level 0). For each key found, it invokes the -* provided callback. +* This function walks the radix tree, searching from the top level down to the +* lowest level (level 0), invoking the appropriate callbacks. * * Return: 0 if the walk completed the specified tree, or the non-zero return * value from the callback that stopped the walk. */ int kho_radix_walk_tree (struct kho_radix_tree *tree, -kho_radix_tree_walk_callback_t cb) +const struct kho_radix_walk_cb *cb) {if (WARN_ON_ONCE (!tree->root))return -EINVAL ;
@@ -1417,6 +1417,9 @@ EXPORT_SYMBOL_GPL(kho_retrieve_subtree);static void __init kho_mem_retrieve (void ) { +const struct kho_radix_walk_cb cb = { + .leaf = kho_preserved_memory_reserve, + };const void *fdt =kho_get_fdt ();int err;
@@ -1430,7 +1433,7 @@ static void __init kho_mem_retrieve(void) kho_in.radix_tree.root =kho_get_mem_map (fdt);mutex_init (&kho_in.radix_tree.lock); - err =kho_radix_walk_tree (&kho_in.radix_tree, kho_preserved_memory_reserve); + err =kho_radix_walk_tree (&kho_in.radix_tree, &cb);if (err) {/* * Failed to initialize preserved memory. Clear FDT and radix
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 8/21 kho: add callback for table pages
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
The KHO memory preservation radix tree does not mark the table pages themselves as preserved. This is done to avoid a circular dependency where preserving a page can lead of allocating other preserved pages. This means any walker looking for free ranges of memory outside of scratch areas will ignore the table
Add a table callback that is invoked for each table page. The callback is given the physical address of the table page.
This is useful for the upcoming mechanism that discovers blocks of memory with no preserved pages and lets them be used for boot memory.
Another use case is for users of the radix tree other than KHO itself. The radix tree does not preserve its own pages due to the circular dependency described above. But external users of the radix tree would need to preserve and restore their pages for the radix tree to survive past early boot. They can use this callback to do so.
--- include/linux/kho_radix_tree.h | 3 +++ kernel/liveupdate/kexec_handover.c | 12 ++++++++++++ 2 files changed, 15 insertions(+)
include/linux/kho_radix_tree.h +3 -0
@@ -37,12 +37,15 @@ struct kho_radix_tree {/** * struct kho_radix_walk_cb - Callbacks for KHO radix tree walk. * @leaf: Called on each present key in the radix tree. +* @node: Called on each node of the radix tree itself. Receives the +* physical address of the page containing the node. * * For each callback, a return value of 0 continues the walk and a non-zero * return value is directly returned to the caller. */ struct kho_radix_walk_cb {int (*leaf)(unsigned long key); +int (*node)(phys_addr_t phys); };#ifdef CONFIG_KEXEC_HANDOVER
kernel/liveupdate/kexec_handover.c +12 -0
@@ -288,6 +288,12 @@ static int kho_radix_walk_leaf(struct kho_radix_leaf *leaf, unsigned long key,unsigned int i;int err; +if (cb->node) { + err = cb->node (virt_to_phys (leaf)); +if (err) +return err; + } +if (!cb->leaf)return 0 ;
@@ -310,6 +316,12 @@ static int __kho_radix_walk_tree(struct kho_radix_node *root,unsigned int shift;int err; +if (cb->node) { + err = cb->node (virt_to_phys (root)); +if (err) +return err; + } +for (i =0 ; i <PAGE_SIZE /sizeof (phys_addr_t); i++) {if (!root->table[i])continue ;
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 9/21 kho: add data argument to radix walk callback
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Add an opaque data pointer argument to kho_radix_walk_cb_t. This can be used by callers to pass extra information to the callback.
--- include/linux/kho_radix_tree.h | 8 ++++---- kernel/liveupdate/kexec_handover.c | 24 +++++++++++++----------- 2 files changed, 17 insertions(+), 15 deletions(-)
include/linux/kho_radix_tree.h +4 -4
@@ -44,8 +44,8 @@ struct kho_radix_tree {* return value is directly returned to the caller. */ struct kho_radix_walk_cb { -int (*leaf)(unsigned long key); -int (*node)(phys_addr_t phys); +int (*leaf)(unsigned long key,void *data); +int (*node)(phys_addr_t phys,void *data); };#ifdef CONFIG_KEXEC_HANDOVER
@@ -53,7 +53,7 @@ struct kho_radix_walk_cb {int kho_radix_add_key (struct kho_radix_tree *tree,unsigned long key);void kho_radix_del_key (struct kho_radix_tree *tree,unsigned long key);int kho_radix_walk_tree (struct kho_radix_tree *tree, -const struct kho_radix_walk_cb *cb); +const struct kho_radix_walk_cb *cb,void *data);#else /* #ifdef CONFIG_KEXEC_HANDOVER */
@@ -66,7 +66,7 @@ static inline void kho_radix_del_key(struct kho_radix_tree *tree, unsigned long key) { }static inline int kho_radix_walk_tree (struct kho_radix_tree *tree, -const struct kho_radix_walk_cb *cb) +const struct kho_radix_walk_cb *cb,void *data) {return -EOPNOTSUPP ; }
kernel/liveupdate/kexec_handover.c +13 -11
@@ -282,14 +282,14 @@ void kho_radix_del_key(struct kho_radix_tree *tree, unsigned long key)EXPORT_SYMBOL_GPL (kho_radix_del_key);static int kho_radix_walk_leaf (struct kho_radix_leaf *leaf,unsigned long key, -const struct kho_radix_walk_cb *cb) +const struct kho_radix_walk_cb *cb,void *data) {unsigned long *bitmap = (unsigned long *)leaf;unsigned int i;int err;if (cb->node) { - err = cb->node (virt_to_phys (leaf)); + err = cb->node (virt_to_phys (leaf), data);if (err)return err; }
@@ -298,7 +298,7 @@ static int kho_radix_walk_leaf(struct kho_radix_leaf *leaf, unsigned long key,return 0 ;for_each_set_bit (i, bitmap,PAGE_SIZE *BITS_PER_BYTE ) { - err = cb->leaf (key | i); + err = cb->leaf (key | i, data);if (err)return err; }
@@ -308,7 +308,7 @@ static int kho_radix_walk_leaf(struct kho_radix_leaf *leaf, unsigned long key,static int __kho_radix_walk_tree (struct kho_radix_node *root,unsigned int level,unsigned long start, -const struct kho_radix_walk_cb *cb) +const struct kho_radix_walk_cb *cb,void *data) {struct kho_radix_node *node;struct kho_radix_leaf *leaf;
@@ -317,7 +317,7 @@ static int __kho_radix_walk_tree(struct kho_radix_node *root,int err;if (cb->node) { - err = cb->node (virt_to_phys (root)); + err = cb->node (virt_to_phys (root), data);if (err)return err; }
@@ -338,10 +338,10 @@ static int __kho_radix_walk_tree(struct kho_radix_node *root,* node is pointing to the level 0 bitmap. */ leaf = (struct kho_radix_leaf *)node; - err =kho_radix_walk_leaf (leaf, key, cb); + err =kho_radix_walk_leaf (leaf, key, cb, data); }else { err =__kho_radix_walk_tree (node, level -1 , - key, cb); + key, cb, data); }if (err)
@@ -355,6 +355,7 @@ static int __kho_radix_walk_tree(struct kho_radix_node *root,* kho_radix_walk_tree - Traverses the radix tree and calls a callback for each key. * @tree: A pointer to the KHO radix tree to walk. * @cb: Set of callbacks to be invoked during the tree walk. +* @data: Opaque data pointer passed to each callback in @cb. * * This function walks the radix tree, searching from the top level down to the * lowest level (level 0), invoking the appropriate callbacks.
@@ -363,14 +364,15 @@ static int __kho_radix_walk_tree(struct kho_radix_node *root,* value from the callback that stopped the walk. */ int kho_radix_walk_tree (struct kho_radix_tree *tree, -const struct kho_radix_walk_cb *cb) +const struct kho_radix_walk_cb *cb,void *data) {if (WARN_ON_ONCE (!tree->root))return -EINVAL ;guard (mutex)(&tree->lock); -return __kho_radix_walk_tree (tree->root,KHO_TREE_MAX_DEPTH -1 ,0 , cb); +return __kho_radix_walk_tree (tree->root,KHO_TREE_MAX_DEPTH -1 ,0 , cb, + data); }EXPORT_SYMBOL_GPL (kho_radix_walk_tree);
@@ -501,7 +503,7 @@ static struct page *__init kho_get_preserved_page(phys_addr_t phys,return pfn_to_page (pfn); } -static int __init kho_preserved_memory_reserve (unsigned long key) +static int __init kho_preserved_memory_reserve (unsigned long key,void *data) {union kho_page_info info;struct page *page;
@@ -1445,7 +1447,7 @@ static void __init kho_mem_retrieve(void) kho_in.radix_tree.root =kho_get_mem_map (fdt);mutex_init (&kho_in.radix_tree.lock); - err =kho_radix_walk_tree (&kho_in.radix_tree, &cb); + err =kho_radix_walk_tree (&kho_in.radix_tree, &cb,NULL );if (err) {/* * Failed to initialize preserved memory. Clear FDT and radix
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 10/21 kho: allow early-boot usage of the KHO radix tree
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
The KHO radix tree allocates memory for table pages from the buddy allocator using get_zeroed_page(). This is not available in early boot when memblock is still active.
Using the radix tree in early boot is useful for KHO to track metadata about its memory. One such example is for tracking free blocks for memory allocation when scratch runs out of space. This feature will be added in the following commits.
Add kho_radix_{alloc,free}_node() which allocate and free the table pages. They use slab_is_available() to decide which allocator to use. While slab_is_available() indicates availability of the slab allocator, it gets initialized right after buddy so it serves the same practical purpose.
--- kernel/liveupdate/kexec_handover.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
kernel/liveupdate/kexec_handover.c +22 -2
@@ -146,6 +146,26 @@ static unsigned long kho_radix_get_table_index(unsigned long key,return (key >> s) % (1 <<KHO_TABLE_SIZE_LOG2 ); } +static void __ref *kho_radix_alloc_node (void ) +{ +struct kho_radix_node *node; + +if (slab_is_available ()) + node = (struct kho_radix_node *)get_zeroed_page (GFP_KERNEL ); +else + node =memblock_alloc (PAGE_SIZE ,PAGE_SIZE ); + +return node; +} + +static void __ref kho_radix_free_node (struct kho_radix_node *node) +{ +if (slab_is_available ()) +free_page ((unsigned long )node); +else +memblock_free (node,PAGE_SIZE ); +} +/** * kho_radix_add_key - Add a key to the radix tree. * @tree: The KHO radix tree.
@@ -194,7 +214,7 @@ int kho_radix_add_key(struct kho_radix_tree *tree, unsigned long key) }/* Next node is empty, create a new node for it */ - new_node = (struct kho_radix_node *)get_zeroed_page (GFP_KERNEL ); + new_node =kho_radix_alloc_node ();if (!new_node) { err = -ENOMEM ; goto err_free_nodes;
@@ -225,7 +245,7 @@ int kho_radix_add_key(struct kho_radix_tree *tree, unsigned long key) err_free_nodes:for (i =KHO_TREE_MAX_DEPTH -1 ; i >0 ; i--) {if (intermediate_nodes[i]) -free_page ((unsigned long )intermediate_nodes[i]); +kho_radix_free_node (intermediate_nodes[i]); }if (anchor_node) anchor_node->table[anchor_idx] =0 ;
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 11/21 kho: allow destroying KHO radix tree
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Add kho_radix_destroy_tree() which allows destroying the radix tree and freeing all its pages.
This is will be used by the upcoming scratch extension mechanism. It creates a radix tree to track free blocks and then frees them after telling memblock about them.
--- include/linux/kho_radix_tree.h | 3 +++ kernel/liveupdate/kexec_handover.c | 35 ++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+)
include/linux/kho_radix_tree.h +3 -0
@@ -54,6 +54,7 @@ int kho_radix_add_key(struct kho_radix_tree *tree, unsigned long key);void kho_radix_del_key (struct kho_radix_tree *tree,unsigned long key);int kho_radix_walk_tree (struct kho_radix_tree *tree,const struct kho_radix_walk_cb *cb,void *data); +void kho_radix_destroy_tree (struct kho_radix_tree *tree);#else /* #ifdef CONFIG_KEXEC_HANDOVER */
@@ -71,6 +72,8 @@ static inline int kho_radix_walk_tree(struct kho_radix_tree *tree,return -EOPNOTSUPP ; } +static inline void kho_radix_destroy_tree (struct kho_radix_tree *tree) { } +#endif /* #ifdef CONFIG_KEXEC_HANDOVER */ #endif /* _LINUX_KHO_RADIX_TREE_H */
kernel/liveupdate/kexec_handover.c +35 -0
@@ -301,6 +301,41 @@ void kho_radix_del_key(struct kho_radix_tree *tree, unsigned long key) }EXPORT_SYMBOL_GPL (kho_radix_del_key); +static void __kho_radix_destroy_tree (struct kho_radix_node *root, +unsigned int level) +{ +unsigned long i; + +if (level ==0 ) { +kho_radix_free_node (root); +return ; + } + +for (i =0 ; i <PAGE_SIZE /sizeof (phys_addr_t); i++) { +if (root->table[i]) +__kho_radix_destroy_tree (phys_to_virt (root->table[i]), + level -1 ); + } + +kho_radix_free_node (root); +} + +/** +* kho_radix_destroy_tree - Destroy the radix tree +* @tree: The radix tree to destroy +* +* Walk @tree and free all its nodes. +*/ +void kho_radix_destroy_tree (struct kho_radix_tree *tree) +{ +if (!tree->root) +return ; + +__kho_radix_destroy_tree (tree->root,KHO_TREE_MAX_DEPTH -1 ); + tree->root =NULL ; +} +EXPORT_SYMBOL_GPL (kho_radix_destroy_tree); +static int kho_radix_walk_leaf (struct kho_radix_leaf *leaf,unsigned long key,const struct kho_radix_walk_cb *cb,void *data) {
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 12/21 kho: add kho_radix_init_tree()
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Move the initialization logic of the radix tree into kho_radix_init_tree() instead of having users open-code it. Makes the boundaries cleaner and reduces code duplication when a new user of the radix tree will be added in a future commit.
--- include/linux/kho_radix_tree.h | 7 ++++ kernel/liveupdate/kexec_handover.c | 62 +++++++++++++++++++++--------- 2 files changed, 51 insertions(+), 18 deletions(-)
include/linux/kho_radix_tree.h +7 -0
@@ -54,6 +54,7 @@ int kho_radix_add_key(struct kho_radix_tree *tree, unsigned long key);void kho_radix_del_key (struct kho_radix_tree *tree,unsigned long key);int kho_radix_walk_tree (struct kho_radix_tree *tree,const struct kho_radix_walk_cb *cb,void *data); +int kho_radix_init_tree (struct kho_radix_tree *tree,struct kho_radix_node *root);void kho_radix_destroy_tree (struct kho_radix_tree *tree);#else /* #ifdef CONFIG_KEXEC_HANDOVER */
@@ -72,6 +73,12 @@ static inline int kho_radix_walk_tree(struct kho_radix_tree *tree, return -EOPNOTSUPP; } +static inline int kho_radix_init_tree (struct kho_radix_tree *tree, +struct kho_radix_node *root) +{ +return 0 ; +} +static inline void kho_radix_destroy_tree (struct kho_radix_tree *tree) { }#endif /* #ifdef CONFIG_KEXEC_HANDOVER */
kernel/liveupdate/kexec_handover.c +44 -18
93 lines not on this page · raw message
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 13/21 kho: expose kho_scratch_overlap() to kexec_handover.h
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Support for discovering memory blocks with no preserved memory will be added in coming patches. These areas will also be marked as scratch to allow allocations from them. Memblock will switch to looking through the scratch array to decide the right migratetype.
Expose kho_scratch_overlap() to KHO users. Since it is now used by non-debug code, move it out of kexec_handover_debug.c and into kexec_handover.c. Gate the overlap checks in kho_preserve_folio() and kho_preserve_pages() by IS_ENABLED(CONFIG_KEXEC_HANDOVER_DEBUG) instead.
Since kexec_handover_debug.c is now empty, delete it.
No functional changes.
--- include/linux/kexec_handover.h | 7 ++++++ kernel/liveupdate/Makefile | 1 - kernel/liveupdate/kexec_handover.c | 22 ++++++++++++++++-- kernel/liveupdate/kexec_handover_debug.c | 25 --------------------- kernel/liveupdate/kexec_handover_internal.h | 9 -------- 5 files changed, 27 insertions(+), 37 deletions(-) delete mode 100644 kernel/liveupdate/kexec_handover_debug.c
include/linux/kexec_handover.h +7 -0
@@ -36,6 +36,8 @@ void kho_memory_init(void);void kho_populate (phys_addr_t fdt_phys,u64 fdt_len,phys_addr_t scratch_phys,u64 scratch_len); + +bool kho_scratch_overlap (phys_addr_t phys,size_t size);#else static inline bool kho_is_enabled (void ) {
@@ -112,6 +114,11 @@ static inline void kho_populate(phys_addr_t fdt_phys, u64 fdt_len, phys_addr_t scratch_phys, u64 scratch_len) { } + +static inline bool kho_scratch_overlap (phys_addr_t phys,size_t size) +{ +return false; +}#endif /* CONFIG_KEXEC_HANDOVER */ #endif /* LINUX_KEXEC_HANDOVER_H */
kernel/liveupdate/Makefile +0 -1
@@ -8,7 +8,6 @@ luo-y := \ luo_session.o obj-$(CONFIG_KEXEC_HANDOVER) += kexec_handover.o -obj-$(CONFIG_KEXEC_HANDOVER_DEBUG) += kexec_handover_debug.o obj-$(CONFIG_KEXEC_HANDOVER_DEBUGFS) += kexec_handover_debugfs.o obj-$(CONFIG_LIVEUPDATE) += luo.o
kernel/liveupdate/kexec_handover.c +20 -2
@@ -765,6 +765,22 @@ static phys_addr_t __init scratch_size_node(int nid)return round_up (size,CMA_MIN_ALIGNMENT_BYTES ); } +bool kho_scratch_overlap (phys_addr_t phys,size_t size) +{ +phys_addr_t scratch_start, scratch_end; +unsigned int i; + +for (i =0 ; i < kho_scratch_cnt; i++) { + scratch_start = kho_scratch[i].addr; + scratch_end = kho_scratch[i].addr + kho_scratch[i].size; + +if (phys < scratch_end && (phys + size) > scratch_start) +return true; + } + +return false; +} +/** * kho_reserve_scratch - Reserve a contiguous chunk of memory for kexec *
@@ -962,7 +978,8 @@ int kho_preserve_folio(struct folio *folio)const unsigned long pfn =folio_pfn (folio);const unsigned int order =folio_order (folio); -if (WARN_ON (kho_scratch_overlap (pfn <<PAGE_SHIFT ,PAGE_SIZE << order))) +if (IS_ENABLED (CONFIG_KEXEC_HANDOVER_DEBUG ) && +WARN_ON (kho_scratch_overlap (pfn <<PAGE_SHIFT ,PAGE_SIZE << order)))return -EINVAL ; return kho_radix_add_key(tree, kho_encode_radix_key(PFN_PHYS(pfn),
@@ -1039,7 +1056,8 @@ int kho_preserve_pages(struct page *page, unsigned long nr_pages)unsigned long failed_pfn =0 ;int err =0 ; -if (WARN_ON (kho_scratch_overlap (start_pfn <<PAGE_SHIFT , +if (IS_ENABLED (CONFIG_KEXEC_HANDOVER_DEBUG ) && +WARN_ON (kho_scratch_overlap (start_pfn <<PAGE_SHIFT , nr_pages <<PAGE_SHIFT ))) {return -EINVAL ; }
kernel/liveupdate/kexec_handover_debug.c deleted +0 -25
@@ -1,25 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* -* kexec_handover_debug.c - kexec handover optional debug functionality -* Copyright (C) 2025 Google LLC, Pasha Tatashin <[email protected]> -*/ - -#define pr_fmt (fmt) "KHO: " fmt - -#include "kexec_handover_internal.h" - -bool kho_scratch_overlap (phys_addr_t phys,size_t size) -{ -phys_addr_t scratch_start, scratch_end; -unsigned int i; - -for (i =0 ; i < kho_scratch_cnt; i++) { - scratch_start = kho_scratch[i].addr; - scratch_end = kho_scratch[i].addr + kho_scratch[i].size; - -if (phys < scratch_end && (phys + size) > scratch_start) -return true; - } - -return false; -}
kernel/liveupdate/kexec_handover_internal.h +0 -9
@@ -41,13 +41,4 @@ static inline void kho_debugfs_blob_remove(struct kho_debugfs *dbg, void *blob) { }#endif /* CONFIG_KEXEC_HANDOVER_DEBUGFS */ -#ifdef CONFIG_KEXEC_HANDOVER_DEBUG -bool kho_scratch_overlap (phys_addr_t phys,size_t size); -#else -static inline bool kho_scratch_overlap (phys_addr_t phys,size_t size) -{ -return false; -} -#endif /* CONFIG_KEXEC_HANDOVER_DEBUG */ -#endif /* LINUX_KEXEC_HANDOVER_INTERNAL_H */
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 14/21 kho: initialize kho_scratch pointer earlier in boot
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
In a future patch, mm init will use kho_scratch_overlap() for deciding the migrate type of pageblocks it initializes. The earliest user currently is free_area_init(). kho_scratch_overlap()
relies on kho_scratch pointer being initialized. Introduce kho_memory_init_early() to do this.
kho_populate() would normally be a good place to do this, but unfortunately, phys_to_virt() does not work at that point on ARM64. So we need yet another initialization function.
--- include/linux/kexec_handover.h | 3 +++ kernel/liveupdate/kexec_handover.c | 15 +++++++++++++-- mm/mm_init.c | 1 + 3 files changed, 17 insertions(+), 2 deletions(-)
include/linux/kexec_handover.h +3 -0
@@ -33,6 +33,7 @@ void kho_remove_subtree(void *blob);int kho_retrieve_subtree (const char *name,phys_addr_t *phys,size_t *size);void kho_memory_init (void ); +void kho_memory_init_early (void );void kho_populate (phys_addr_t fdt_phys,u64 fdt_len,phys_addr_t scratch_phys,u64 scratch_len);
@@ -110,6 +111,8 @@ static inline int kho_retrieve_subtree(const char *name, phys_addr_t *phys,static inline void kho_memory_init (void ) { } +static inline void kho_memory_init_early (void ) { } +static inline void kho_populate (phys_addr_t fdt_phys,u64 fdt_len,phys_addr_t scratch_phys,u64 scratch_len) {
kernel/liveupdate/kexec_handover.c +13 -2
@@ -1534,8 +1534,6 @@ static void __init kho_mem_retrieve(void)const void *fdt =kho_get_fdt ();int err; - kho_scratch =phys_to_virt (kho_in.scratch_phys); -/* * kho_get_mem_map() should always succeed. If it fails, kho_populate() * catches that and never sets kho_in.scratch_phys, which stops memory
@@ -1756,6 +1754,19 @@ static __init int kho_init(void) }fs_initcall (kho_init); +void __init kho_memory_init_early (void ) +{ +if (!is_kho_boot ()) +return ; + +/* +* kho_scratch_overlap() needs kho_scratch to be initialized. It +* is used by free_area_init() on KHO boots, so initialize it +* early. +*/ + kho_scratch =phys_to_virt (kho_in.scratch_phys); +} +void __init kho_memory_init (void ) {if (kho_in.scratch_phys)
mm/mm_init.c +1 -0
@@ -2687,6 +2687,7 @@ void __init __weak mem_init(void)void __init mm_core_init_early (void ) { +kho_memory_init_early ();hugetlb_cma_reserve ();hugetlb_bootmem_alloc ();
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 15/21 kho: initialize preserved memory map radix tree earlier
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Currently the preserved memory radix tree is initialized from kho_memory_init(), which happens relatively late in MM init. In a coming patch, the tree will be used from kho_memory_init_early(). Move the tree initialization there.
Simplify some of the code in kho_mem_retrieve() by getting rid of the err variable and jumping to err directly.
--- kernel/liveupdate/kexec_handover.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-)
kernel/liveupdate/kexec_handover.c +17 -13
@@ -1531,20 +1531,8 @@ static void __init kho_mem_retrieve(void)const struct kho_radix_walk_cb cb = { .leaf = kho_preserved_memory_reserve, }; -const void *fdt =kho_get_fdt (); -int err; -/* -* kho_get_mem_map() should always succeed. If it fails, kho_populate() -* catches that and never sets kho_in.scratch_phys, which stops memory -* retrieval. -*/ - err =kho_radix_init_tree (&kho_in.radix_tree,kho_get_mem_map (fdt)); -if (err) - goto err; - - err =kho_radix_walk_tree (&kho_in.radix_tree, &cb,NULL ); -if (err) +if (kho_radix_walk_tree (&kho_in.radix_tree, &cb,NULL )) goto err;return ;
@@ -1756,6 +1744,8 @@ fs_initcall(kho_init);void __init kho_memory_init_early (void ) { +const void *fdt =kho_get_fdt (); +if (!is_kho_boot ())return ;
@@ -1765,6 +1755,20 @@ void __init kho_memory_init_early(void)* early. */ kho_scratch =phys_to_virt (kho_in.scratch_phys); + +/* +* kho_get_mem_map() should always succeed. If it fails, kho_populate() +* catches that and never sets kho_in.fdt_phys. +*/ +if (kho_radix_init_tree (&kho_in.radix_tree,kho_get_mem_map (fdt))) { +/* +* Failed to initialize preserved memory radix tree. Clear FDT +* and scratch so KHO users don't treat it as a KHO boot. +*/ + kho_in.fdt_phys =0 ; + kho_in.scratch_phys =0 ; +return ; + } }void __init kho_memory_init (void )
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 16/21 mm/mm_init: init deferred page migratetype in deferred_init_pages()
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
The deferred pages are initialized in deferred_init_pages(), but their migratetype is initialized later when freeing them in deferred_free_pages(). Setting the migrate type is part of initializing the pages, so move it to deferred_init_pages().
This simplifies the logic in deferred_free_pages(), which now only does the freeing. For non-max-order-aligned pages, it also drops the if (pageblock_aligned) for each freed page, which in theory should be a tiny bit faster.
Another motivation for this change is that in a coming patch, KHO will need to update the migrate type for its scratch pages. Moving all the pageblock init in one place makes that simpler.
--- mm/mm_init.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
mm/mm_init.c +12 -9
@@ -1985,8 +1985,6 @@ static void __init deferred_free_pages(unsigned long pfn,/* Free a large naturally-aligned chunk if possible */ if (nr_pages ==MAX_ORDER_NR_PAGES &&IS_MAX_ORDER_ALIGNED (pfn)) { -for (i =0 ; i < nr_pages; i += pageblock_nr_pages) -init_pageblock_migratetype (page + i, mt, false);__free_pages_core (page,MAX_PAGE_ORDER ,MEMINIT_EARLY );return ; }
@@ -1994,11 +1992,8 @@ static void __init deferred_free_pages(unsigned long pfn,/* Accept chunks smaller than MAX_PAGE_ORDER upfront */ accept_memory (PFN_PHYS (pfn), nr_pages *PAGE_SIZE ); -for (i =0 ; i < nr_pages; i++, page++, pfn++) { -if (pageblock_aligned (pfn)) -init_pageblock_migratetype (page, mt, false); +for (i =0 ; i < nr_pages; i++, page++, pfn++)__free_pages_core (page,0 ,MEMINIT_EARLY ); - } }/* Completion tracking for deferred_init_memmap() threads */
@@ -2017,15 +2012,23 @@ static inline void __init pgdat_init_report_one_done(void)* Return number of pages initialized. */ static unsigned long __init deferred_init_pages (struct zone *zone, -unsigned long pfn,unsigned long end_pfn) +unsigned long start_pfn,unsigned long end_pfn, +enum migratetype mt) {int nid =zone_to_nid (zone); -unsigned long nr_pages = end_pfn - pfn; +unsigned long nr_pages = end_pfn - start_pfn, pfn = start_pfn;int zid =zone_idx (zone);struct page *page =pfn_to_page (pfn);for (; pfn < end_pfn; pfn++, page++)__init_single_page (page, pfn, zid, nid); + +/* Now initialize migrate types for these pages. */ + pfn =pageblock_align (start_pfn); + page =pfn_to_page (pfn); +for (; pfn < end_pfn; pfn += pageblock_nr_pages, page += pageblock_nr_pages) +init_pageblock_migratetype (page, mt, false); +return nr_pages; }
@@ -2067,7 +2070,7 @@ deferred_init_memmap_chunk(unsigned long start_pfn, unsigned long end_pfn,unsigned long mo_pfn =ALIGN (spfn +1 ,MAX_ORDER_NR_PAGES );unsigned long chunk_end =min (mo_pfn, epfn); - nr_pages +=deferred_init_pages (zone, spfn, chunk_end); + nr_pages +=deferred_init_pages (zone, spfn, chunk_end, mt);deferred_free_pages (spfn, chunk_end - spfn, mt); spfn = chunk_end;
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Mike Rapoport
maintainerRe: [PATCH v3 16/21] mm/mm_init: init deferred page migratetype in deferred_init_pages()
mm/mm_init.c static void __init deferred_free_pages(unsigned long pfn,
Quoted 21 lines
The deferred pages are initialized in
deferred_init_pages(), but their migratetype is initialized later when freeing them indeferred_free_pages(). Setting the migrate type is part of initializing the pages, so move it todeferred_init_pages().This simplifies the logic in
deferred_free_pages(), which now only does the freeing. For non-max-order-aligned pages, it also drops the if (pageblock_aligned) for each freed page, which in theory should be a tiny bit faster.Another motivation for this change is that in a coming patch, KHO will need to update the migrate type for its scratch pages. Moving all the pageblock init in one place makes that simpler.
Signed-off-by Pratyush Yadav (Google) <[email protected]>mm/mm_init.c +0
-0@@ -1985,8 +1985,6 @@ static void __init deferred_free_pages(unsigned long pfn,
There's migratetype parameter in deferred_free_pages(), looks like it's not needed anymore.
Quoted 21 lines
mm/mm_init.c +1
-6/* Free a large naturally-aligned chunk if possible */ if (nr_pages ==MAX_ORDER_NR_PAGES &&IS_MAX_ORDER_ALIGNED (pfn)) { -for (i =0 ; i < nr_pages; i += pageblock_nr_pages) -init_pageblock_migratetype (page + i, mt, false);__free_pages_core (page,MAX_PAGE_ORDER ,MEMINIT_EARLY );return ; }@@ -1994,11 +1992,8 @@ static void __init deferred_free_pages(unsigned long pfn,/* Accept chunks smaller than MAX_PAGE_ORDER upfront */ accept_memory (PFN_PHYS (pfn), nr_pages *PAGE_SIZE ); -for (i =0 ; i < nr_pages; i++, page++, pfn++) { -if (pageblock_aligned (pfn)) -init_pageblock_migratetype (page, mt, false); +for (i =0 ; i < nr_pages; i++, page++, pfn++)__free_pages_core (page,0 ,MEMINIT_EARLY ); - } }/* Completion tracking for deferred_init_memmap() threads */
mm/mm_init.c static inline void __init pgdat_init_report_one_done(void) · line 2019
Quoted 10 lines
mm/mm_init.c +3
-2@@ -2017,15 +2012,23 @@ static inline void __init pgdat_init_report_one_done(void)* Return number of pages initialized. */ static unsigned long __init deferred_init_pages (struct zone *zone, -unsigned long pfn,unsigned long end_pfn) +unsigned long start_pfn,unsigned long end_pfn, +enum migratetype mt) {int nid =zone_to_nid (zone); -unsigned long nr_pages = end_pfn - pfn; +unsigned long nr_pages = end_pfn - start_pfn, pfn = start_pfn;
Let's declare pfn on a separate line please.
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerPATCH v3 17/21 mm/mm_init: don't rely on memblock to get KHO scratch migratetype
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Currently struct page init via memmap_init() or deferred_init_memmap() only queries the migrate type from KHO for each discrete memory range. That works currently since KHO scratch memory has a different memory type so it is always it its own region.
An upcoming patch will add support for discovering blocks of memory with no preservations and it will mark it as MEMBLOCK_KHO_SCRATCH to allow allocations from them. This can lead to the bootmem KHO scratch areas to be merged into larger free ranges. This merging breaks the selection of migrate type.
Get rid of kho_scratch_migratetype(). Instead, let the struct page init logic set the migratetype to MIGRATE_MOVABLE for all pages, and then give KHO a chance to update the migrate type of its bootmem scratch areas to MIGRATE_CMA. Since scratch areas should be a few order of magnitues smaller than the total system memory, doing it this way is more efficient than searching the kho_scratch array for every pageblock.
Since kho_scratch_migratetype() no longer exists, drop the migratetype argument to memmap_init_zone_range() and deferred_init_pages().
--- include/linux/kexec_handover.h | 6 ++++ include/linux/memblock.h | 19 ------------ kernel/liveupdate/kexec_handover.c | 27 +++++++++++++++++ mm/memblock.c | 10 ------- mm/mm_init.c | 47 +++++++++++++++++++----------- 5 files changed, 63 insertions(+), 46 deletions(-)
include/linux/kexec_handover.h +6 -0
@@ -39,6 +39,7 @@ void kho_populate(phys_addr_t fdt_phys, u64 fdt_len, phys_addr_t scratch_phys, u64 scratch_len);bool kho_scratch_overlap (phys_addr_t phys,size_t size); +void kho_init_scratch_migratetype (unsigned long start_pfn,unsigned long end_pfn);#else static inline bool kho_is_enabled (void ) {
@@ -122,6 +123,11 @@ static inline bool kho_scratch_overlap(phys_addr_t phys, size_t size) {return false; } + +static inline void kho_init_scratch_migratetype (unsigned long start_pfn, +unsigned long end_pfn); +{ +}#endif /* CONFIG_KEXEC_HANDOVER */ #endif /* LINUX_KEXEC_HANDOVER_H */
include/linux/memblock.h +0 -19
@@ -613,28 +613,9 @@ static inline void memtest_report_meminfo(struct seq_file *m) { }#ifdef CONFIG_MEMBLOCK_KHO_SCRATCH void memblock_set_kho_scratch_only (void );void memblock_clear_kho_scratch_only (void ); -bool memblock_is_kho_scratch_memory (phys_addr_t addr); - -static inline enum migratetype kho_scratch_migratetype (unsigned long pfn, -enum migratetype mt) -{ -if (memblock_is_kho_scratch_memory (PFN_PHYS (pfn))) -return MIGRATE_CMA ; -return mt; -}#else static inline void memblock_set_kho_scratch_only (void ) { }static inline void memblock_clear_kho_scratch_only (void ) { } -static inline bool memblock_is_kho_scratch_memory (phys_addr_t addr) -{ -return false; -} - -static inline enum migratetype kho_scratch_migratetype (unsigned long pfn, -enum migratetype mt) -{ -return mt; -}#endif #endif /* _LINUX_MEMBLOCK_H */
kernel/liveupdate/kexec_handover.c +27 -0
@@ -781,6 +781,33 @@ bool kho_scratch_overlap(phys_addr_t phys, size_t size)return false; } +/* +* Mark all KHO scratch pageblocks between start_pfn and end_pfn as +* MIGRATE_CMA. +*/ +void kho_init_scratch_migratetype (unsigned long start_pfn,unsigned long end_pfn) +{ +for (unsigned int i =0 ; i < kho_scratch_cnt; i++) { +unsigned long scratch_start, scratch_end, pfn; + + scratch_start =PHYS_PFN (kho_scratch[i].addr); + scratch_end =PHYS_PFN (kho_scratch[i].addr + kho_scratch[i].size); + +/* Target range doesn't overlap with scratch. */ +if (start_pfn >= scratch_end || end_pfn <= scratch_start) +continue ; + + pfn = max_t(unsigned long, start_pfn, scratch_start); + pfn =pageblock_start_pfn (pfn); + +while (pfn < end_pfn && pfn < scratch_end) { +init_pageblock_migratetype (pfn_to_page (pfn), +MIGRATE_CMA , false); + pfn += pageblock_nr_pages; + } + } +} +/** * kho_reserve_scratch - Reserve a contiguous chunk of memory for kexec *
mm/memblock.c +0 -10
@@ -2511,16 +2511,6 @@ __init void memblock_clear_kho_scratch_only(void) { kho_scratch_only = false; } - -bool __init_memblock memblock_is_kho_scratch_memory(phys_addr_t addr) -{ -int i =memblock_search (&memblock.memory, addr); - -if (i ==-1 ) -return false; - -return memblock_is_kho_scratch (&memblock.memory.regions[i]); -}#endif #ifdef CONFIG_KEXEC_HANDOVER
mm/mm_init.c +30 -17
112 lines not on this page · raw message
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Mike Rapoport
maintainerRe: [PATCH v3 17/21] mm/mm_init: don't rely on memblock to get KHO scratch migratetype
Quoted 11 lines
Currently struct page init via
memmap_init()ordeferred_init_memmap()only queries the migrate type from KHO for each discrete memory range. That works currently since KHO scratch memory has a different memory type so it is always it its own region.An upcoming patch will add support for discovering blocks of memory with no preservations and it will mark it as
MEMBLOCK_KHO_SCRATCHto allow allocations from them. This can lead to the bootmem KHO scratch areas to be merged into larger free ranges. This merging breaks the selection of migrate type.
Quoted 6 lines
Get rid of
kho_scratch_migratetype(). Instead, let the struct page init logic set the migratetype toMIGRATE_MOVABLEfor all pages, and then give KHO a chance to update the migrate type of its bootmem scratch areas toMIGRATE_CMA. Since scratch areas should be a few order of magnitues smaller than the total system memory, doing it this way is more efficient than searching thekho_scratcharray for every pageblock.
It's true for non-deferred case, but with deferred initialization of struct pages the static part is small and deferred chunks are small, so we are going to update the migratetype nearly as much as we would check it with kho_scratch_migratetype().
For simplicity of this patchset I'd keep kho_scratch_migratetype(), just move it over to kexec_handover.h and make it rely on kho_scratch_overlap().
The bulk optimization can be a separate patch on top.
Quoted 86 lines
Since
kho_scratch_migratetype()no longer exists, drop the migratetype argument tomemmap_init_zone_range()anddeferred_init_pages().Signed-off-by Pratyush Yadav (Google) <[email protected]>include/linux/kexec_handover.h +6
-0@@ -39,6 +39,7 @@ void kho_populate(phys_addr_t fdt_phys, u64 fdt_len, phys_addr_t scratch_phys, u64 scratch_len);bool kho_scratch_overlap (phys_addr_t phys,size_t size); +void kho_init_scratch_migratetype (unsigned long start_pfn,unsigned long end_pfn);#else static inline bool kho_is_enabled (void ) {@@ -122,6 +123,11 @@ static inline bool kho_scratch_overlap(phys_addr_t phys, size_t size) {return false; } + +static inline void kho_init_scratch_migratetype (unsigned long start_pfn, +unsigned long end_pfn); +{ +}#endif /* CONFIG_KEXEC_HANDOVER */ #endif /* LINUX_KEXEC_HANDOVER_H */ include/linux/memblock.h +0
-19@@ -613,28 +613,9 @@ static inline void memtest_report_meminfo(struct seq_file *m) { }#ifdef CONFIG_MEMBLOCK_KHO_SCRATCH void memblock_set_kho_scratch_only (void );void memblock_clear_kho_scratch_only (void ); -bool memblock_is_kho_scratch_memory (phys_addr_t addr); - -static inline enum migratetype kho_scratch_migratetype (unsigned long pfn, -enum migratetype mt) -{ -if (memblock_is_kho_scratch_memory (PFN_PHYS (pfn))) -return MIGRATE_CMA ; -return mt; -}#else static inline void memblock_set_kho_scratch_only (void ) { }static inline void memblock_clear_kho_scratch_only (void ) { } -static inline bool memblock_is_kho_scratch_memory (phys_addr_t addr) -{ -return false; -} - -static inline enum migratetype kho_scratch_migratetype (unsigned long pfn, -enum migratetype mt) -{ -return mt; -}#endif #endif /* _LINUX_MEMBLOCK_H */ kernel/liveupdate/kexec_handover.c +16
-0@@ -781,6 +781,33 @@ bool kho_scratch_overlap(phys_addr_t phys, size_t size)return false; } +/* +* Mark all KHO scratch pageblocks between start_pfn and end_pfn as +* MIGRATE_CMA. +*/ +void kho_init_scratch_migratetype (unsigned long start_pfn,unsigned long end_pfn) +{ +for (unsigned int i =0 ; i < kho_scratch_cnt; i++) { +unsigned long scratch_start, scratch_end, pfn; + + scratch_start =PHYS_PFN (kho_scratch[i].addr); + scratch_end =PHYS_PFN (kho_scratch[i].addr + kho_scratch[i].size); + +/* Target range doesn't overlap with scratch. */ +if (start_pfn >= scratch_end || end_pfn <= scratch_start) +continue ; +
kernel/liveupdate/kexec_handover.c bool kho_scratch_overlap(phys_addr_t phys, size_t size) · line 809
Quoted 10 lines
kernel/liveupdate/kexec_handover.c +10
-0+ pfn = max_t(unsigned long, start_pfn, scratch_start); + pfn =pageblock_start_pfn (pfn); + +while (pfn < end_pfn && pfn < scratch_end) { +init_pageblock_migratetype (pfn_to_page (pfn), +MIGRATE_CMA , false); + pfn += pageblock_nr_pages; + } + } +}
static inline enum migratetype kho_scratch_migratetype(unsigned long pfn,
enum migratetype mt)
{
if (kho_scratch_overlap(PFN_PHYS(pfn)), PAGE_SIZE)
return MIGRATE_CMA;
return mt;
}Is way easier to parse and reason about ;-)
Quoted 30 lines
kernel/liveupdate/kexec_handover.c +1
-0+/** * kho_reserve_scratch - Reserve a contiguous chunk of memory for kexec * mm/memblock.c +0
-10@@ -2511,16 +2511,6 @@ __init void memblock_clear_kho_scratch_only(void) { kho_scratch_only = false; } - -bool __init_memblock memblock_is_kho_scratch_memory(phys_addr_t addr) -{ -int i =memblock_search (&memblock.memory, addr); - -if (i ==-1 ) -return false; - -return memblock_is_kho_scratch (&memblock.memory.regions[i]); -}#endif #ifdef CONFIG_KEXEC_HANDOVER mm/mm_init.c +0
-0
mm/mm_init.c void __meminit __init_page_from_nid(unsigned long pfn, int nid) · line 713
Quoted 10 lines
mm/mm_init.c +4
-3@@ -707,9 +707,10 @@ void __meminit __init_page_from_nid(unsigned long pfn, int nid)__init_single_page (pfn_to_page (pfn), pfn, zid, nid);if (pageblock_aligned (pfn)) { -enum migratetype mt = -kho_scratch_migratetype (pfn,MIGRATE_MOVABLE ); -init_pageblock_migratetype (pfn_to_page (pfn), mt, false); +init_pageblock_migratetype (pfn_to_page (pfn),MIGRATE_MOVABLE , + false); +/* Update the migrate type in case it is KHO scratch. */ +kho_init_scratch_migratetype (pfn, pfn +1 );
Calling kho_init_scratch_migratetype() instead of kho_scratch_migratetype() here won't have any benefit, right?
A better optimization would be to call kho_init_scratch_migratetype() from memmap_init_reserved_range() I think.
Quoted 7 lines
mm/mm_init.c +0
-0} }@@ -938,13 +939,22 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone } pfn++; }
mm/mm_init.c void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone · line 951
Quoted 10 lines
mm/mm_init.c +10
-0+ +/* +* Update the migrate type for any pageblocks that fall in a KHO scratch +* area. Do this here outside the loop because most pageblocks will not +* be KHO scratch, so only search once. +* +* Only pageblocks up to pfn are updated, deferred pageblocks will be +* updated later. +*/ +kho_init_scratch_migratetype (start_pfn, pfn);
It's no relevant for memory hotplug or ZONE_DEVICE so ideally this should have been in memmap_init_zone_range() and somehow take into account first_deferred_pfn. And then it becomes hairy :)
Quoted 61 lines
mm/mm_init.c +6
-9}static void __init memmap_init_zone_range (struct zone *zone,unsigned long start_pfn,unsigned long end_pfn, -unsigned long *hole_pfn, -enum migratetype mt) +unsigned long *hole_pfn) {unsigned long zone_start_pfn = zone->zone_start_pfn;unsigned long zone_end_pfn = zone_start_pfn + zone->spanned_pages;@@ -957,7 +967,8 @@ static void __init memmap_init_zone_range(struct zone *zone,return ;memmap_init_range (end_pfn - start_pfn, nid, zone_id, start_pfn, - zone_end_pfn,MEMINIT_EARLY ,NULL , mt, false); + zone_end_pfn,MEMINIT_EARLY ,NULL ,MIGRATE_MOVABLE , + false);if (*hole_pfn < start_pfn)init_unavailable_range (*hole_pfn, start_pfn, zone_id, nid);@@ -973,8 +984,6 @@ static void __init memmap_init(void)for_each_mem_pfn_range (i,MAX_NUMNODES , &start_pfn, &end_pfn, &nid) {struct pglist_data *node =NODE_DATA (nid); -enum migratetype mt = -kho_scratch_migratetype (start_pfn,MIGRATE_MOVABLE );for (j =0 ; j <MAX_NR_ZONES ; j++) {struct zone *zone = node->node_zones + j;@@ -983,7 +992,7 @@ static void __init memmap_init(void)continue ;memmap_init_zone_range (zone, start_pfn, end_pfn, - &hole_pfn, mt); + &hole_pfn); zone_id = j; } }@@ -1973,7 +1982,7 @@ unsigned long __init node_map_pfn_alignment(void)#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static void __init deferred_free_pages (unsigned long pfn, -unsigned long nr_pages,enum migratetype mt) +unsigned long nr_pages) {struct page *page;unsigned long i;@@ -2012,8 +2021,7 @@ static inline void __init pgdat_init_report_one_done(void)* Return number of pages initialized. */ static unsigned long __init deferred_init_pages (struct zone *zone, -unsigned long start_pfn,unsigned long end_pfn, -enum migratetype mt) +unsigned long start_pfn,unsigned long end_pfn) {int nid =zone_to_nid (zone);unsigned long nr_pages = end_pfn - start_pfn, pfn = start_pfn;@@ -2027,7 +2035,14 @@ static unsigned long __init deferred_init_pages(struct zone *zone, pfn =pageblock_align (start_pfn); page =pfn_to_page (pfn);
mm/mm_init.c static unsigned long __init deferred_init_pages(struct zone *zone, · line 2045
Quoted 10 lines
mm/mm_init.c +8
-1for (; pfn < end_pfn; pfn += pageblock_nr_pages, page += pageblock_nr_pages) -init_pageblock_migratetype (page, mt, false); +init_pageblock_migratetype (page,MIGRATE_MOVABLE , false); + +/* +* Update the migrate type for any pageblocks that fall in a KHO scratch +* area. Since most pageblocks will not be KHO scratch, do it outside +* the loop to only do the search once. +*/ +kho_init_scratch_migratetype (start_pfn, end_pfn);
And here the chunks are at most MAX_ORDER_NR_PAGES, so bulk init also wouldn't save much.
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerRe: [PATCH v3 17/21] mm/mm_init: don't rely on memblock to get KHO scratch migratetype
Quoted 23 lines · Mike Rapoport
Quoted 11 lines
Currently struct page init via
memmap_init()ordeferred_init_memmap()only queries the migrate type from KHO for each discrete memory range. That works currently since KHO scratch memory has a different memory type so it is always it its own region.An upcoming patch will add support for discovering blocks of memory with no preservations and it will mark it as
MEMBLOCK_KHO_SCRATCHto allow allocations from them. This can lead to the bootmem KHO scratch areas to be merged into larger free ranges. This merging breaks the selection of migrate type.Quoted 6 lines
Get rid of
kho_scratch_migratetype(). Instead, let the struct page init logic set the migratetype toMIGRATE_MOVABLEfor all pages, and then give KHO a chance to update the migrate type of its bootmem scratch areas toMIGRATE_CMA. Since scratch areas should be a few order of magnitues smaller than the total system memory, doing it this way is more efficient than searching thekho_scratcharray for every pageblock.It's true for non-deferred case, but with deferred initialization of struct pages the static part is small and deferred chunks are small, so we are going to update the migratetype nearly as much as we would check it with
kho_scratch_migratetype().
Quoted 4 lines
For simplicity of this patchset I'd keep
kho_scratch_migratetype(), just move it over tokexec_handover.hand make it rely onkho_scratch_overlap().The bulk optimization can be a separate patch on top.
Sure, will do.
Quoted 6 lines
Quoted 6 lines
Since
kho_scratch_migratetype()no longer exists, drop the migratetype argument tomemmap_init_zone_range()anddeferred_init_pages().Signed-off-by Pratyush Yadav (Google) <[email protected]>
[...]
Quoted 41 lines
Quoted 16 lines
kernel/liveupdate/kexec_handover.c +16
-0+/* +* Mark all KHO scratch pageblocks between start_pfn and end_pfn as +* MIGRATE_CMA. +*/ +void kho_init_scratch_migratetype (unsigned long start_pfn,unsigned long end_pfn) +{ +for (unsigned int i =0 ; i < kho_scratch_cnt; i++) { +unsigned long scratch_start, scratch_end, pfn; + + scratch_start =PHYS_PFN (kho_scratch[i].addr); + scratch_end =PHYS_PFN (kho_scratch[i].addr + kho_scratch[i].size); + +/* Target range doesn't overlap with scratch. */ +if (start_pfn >= scratch_end || end_pfn <= scratch_start) +continue ; +kernel/liveupdate/kexec_handover.c bool kho_scratch_overlap(phys_addr_t phys, size_t size) · line 809
Quoted 10 lines
kernel/liveupdate/kexec_handover.c +10
-0+ pfn = max_t(unsigned long, start_pfn, scratch_start); + pfn =pageblock_start_pfn (pfn); + +while (pfn < end_pfn && pfn < scratch_end) { +init_pageblock_migratetype (pfn_to_page (pfn), +MIGRATE_CMA , false); + pfn += pageblock_nr_pages; + } + } +}static inline enum migratetype kho_scratch_migratetype(unsigned long pfn,
enum migratetype mt){ if (kho_scratch_overlap(PFN_PHYS(pfn)), PAGE_SIZE) return MIGRATE_CMA; return mt; }Is way easier to parse and reason about ;-)
Quoted 4 lines
+
/**
*kho_reserve_scratch- Reserve a contiguous chunk of memory for kexec *
[...]
Quoted 18 lines
Quoted 18 lines
mm/mm_init.c +4
-3@@ -2012,8 +2021,7 @@ static inline void __init pgdat_init_report_one_done(void)* Return number of pages initialized. */ static unsigned long __init deferred_init_pages (struct zone *zone, -unsigned long start_pfn,unsigned long end_pfn, -enum migratetype mt) +unsigned long start_pfn,unsigned long end_pfn) {int nid =zone_to_nid (zone);unsigned long nr_pages = end_pfn - start_pfn, pfn = start_pfn;@@ -2027,7 +2035,14 @@ static unsigned long __init deferred_init_pages(struct zone *zone, pfn =pageblock_align (start_pfn); page =pfn_to_page (pfn);for (; pfn < end_pfn; pfn += pageblock_nr_pages, page += pageblock_nr_pages) -init_pageblock_migratetype (page, mt, false); +init_pageblock_migratetype (page,MIGRATE_MOVABLE , false); + +/*
Quoted 8 lines
mm/mm_init.c static unsigned long __init deferred_init_pages(struct zone *zone, · line 2045
Quoted 5 lines
mm/mm_init.c +5
-0+* Update the migrate type for any pageblocks that fall in a KHO scratch +* area. Since most pageblocks will not be KHO scratch, do it outside +* the loop to only do the search once. +*/ +kho_init_scratch_migratetype (start_pfn, end_pfn);And here the chunks are at most
MAX_ORDER_NR_PAGES, so bulk init also wouldn't save much.
Hmm, right. I didn't think of that. On my system pageblock order is 9 and max page order is 10, so it does halve the amount of work done, but still, I see your point. It took me a couple of tries to get kho_init_scratch_migratetype() right, and kho_scratch_overlap() is certainly easier to reason about.
Oh well, I suppose this gets added to the list of performance improvements we do at some point down the line
Signature · 2 lines
Regards, Pratyush Yadav
Pratyush Yadav
maintainerPATCH v3 18/21 kho: extend scratch
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Motivation
==========
The scratch space is allocated by the first kernel in the KHO chain, and is reused by all subsequent kernels. The size of the space is either set via the commandline by the system administrator or by calculating the amount of memory used by the kernel and adding a multiplier. In either case, the scratch space is a heuristic and is liable to fill up and fail allocation if a kernel uses more memory than expected.
In addition, gigantic huge pages (usually 1 GiB) are allocated via memblock, and in a KHO boot that memory comes from the scratch space. In hypervisors it is common to dedicate a major part of the system's memory to gigantic hugepages for VM memory.
If this memory needs to come from scratch space, then scratch needs to be greater than the memory needed for huge pages, which is impractical. In addition, hugepages can be preserved memory. Allocating them from scratch violates the assumption that scratch contains no preserved memory.
Methodology
===========
Discover areas that don't contain any preserved memory at boot by walking the preserved memory radix tree. Mark them as scratch to allow allocations from them. This makes KHO more resilient to memory pressure and allows supporting huge page preservation.
Since the preserved memory radix tree mixes both physical address and order into a single key, and does not track table pages, it is difficult to identify free areas from it directly. Walk the tree and digest it down into another radix tree. The latter tracks blocks of KHO_EXT_SHIFT (1 GiB as of now) granularity. Then walk the digested tree and mark the areas between the present keys as scratch.
Performance
===========
The discovery algorithm traverses the preserved memory radix tree exactly once. While it does use memory for the digested radix tree, since the blocks are split by 1 GiB, a single bitmap with 4k pages can track up to 32 TiB of memory. So there are likely to be very few radix tree pages used in this tracking. For systems with all physical memory below 32 TiB, this should result in a total of 6 pages being used (KHO_TREE_MAX_DEPTH == 6).
An alternate way of achieving this would be to call kho_mem_retrieve() earlier in boot and mark all the KHO preservations as reserved. But that can blow up memblock.reserved with a bunch of 4K pages scattered everywhere, which will reduce performance of subsequent allocations. Since the free blocks are tracked in chunks of 1 GiB, this won't blow up memblock.memory as much.
There is no inherent reason for using 1 GiB as the discovered block size. This can be changed later if needed. Currently, KHO is mainly targeted for server grade systems with hundreds of gigabytes to terabytes of memory. So 1 GiB is a reasonable granularity for those systems. For smaller systems this doesn't work as well, but we can arrive at a better heuristic when we have concrete use cases.
Practical evaluation
====================
The testing is done on a x86_64 qemu VM running under KVM with 64G memory and 12 CPUs. The machine pre-allocates 50 1G pages.
Since the performance scales with how busy the radix tree is, tests are done with 2 preservation patterns: first with two 1M memfds, second with two 1G memfds, both using 4k pages.
Test case 1 - 1M memfd
~~~~~~~~~~~~~~~~~~~~~~
This test case has two memfds with 1M memory each in 4k pages, plus other preservations from LUO core and other KHO users.
This is how the radix tree stats look like:
radix_nodes: 0x13
nr_preservations: 0x214
mem_preserved: 0x227000 per order preservations:
order 0: 0x20f
order 1: 0x4
order 4: 0x1and this is how long it takes to extend the scratch after KHO boot:
KHO: KHO extend time: 47 us
KHO: KHO extend total mem: 0xe6c17b000 (~57G)
Test case 2 - 1G memfd
~~~~~~~~~~~~~~~~~~~~~~
This test case has two memfds with 1G memory each in 4k pages, plus other preservations from LUO core and other KHO users.
This is how the radix tree stats look like:
radix_nodes: 0x28
nr_preservations: 0x80816
mem_preserved: 0x80829000 per order preservations:
order 0: 0x80811
order 1: 0x4
order 4: 0x1and this is how long it takes to extend the scratch after KHO boot:
KHO: KHO extend time: 22514 us
KHO: KHO extend total mem: 0xd3f200000 (~52G)
---
Notes:
As one might notice, the "scratch" terminology starts to break down here. There is the original "scratch", which is passed down by the previous kernel. It is marked MEMBLOCK_KHO_SCRATCH. There is also the discovered "scratch", which also gets marked MEMBLOCK_KHO_SCRATCH, but has nothing to do with the former.
For limiting the scope of this series, I haven't done the rename here. I can do it as a follow up series once this stabilizes and lands into -next.
I suggest the following scheme:
- Rename "KHO scratch" to "KHO bootmem". Update the documentation and all code to use this name. We have the kho_scratch kernel cmdline parameter, which is harder to change, but perhaps we can rename it to "kho_bootmem" and if someone complains we can add it back.
- Rename MEMBLOCK_KHO_SCRATCH to MEMBLOCK_KHO_NOPRSRV. This describes the property of the memory not its origin. Then KHO can mark its "bootmem" as KHO_NOPRSRV because bootmem never has any preserved memory. Later, kho_extend_scratch() (which is also due for a better name) can also mark its discovered areas as KHO_NOPRSRV.
kernel/liveupdate/kexec_handover.c | 158 +++++++++++++++++++++++++---- 1 file changed, 141 insertions(+), 17 deletions(-)
kernel/liveupdate/kexec_handover.c +141 -17
@@ -84,6 +84,23 @@ static struct kho_out kho_out = { }, }; +struct kho_in { +phys_addr_t fdt_phys; +phys_addr_t scratch_phys; +char previous_release[__NEW_UTS_LEN +1 ]; +u32 kexec_count; +struct kho_debugfs dbg; +struct kho_radix_tree radix_tree; +}; + +static struct kho_in kho_in = { +}; + +static const void *kho_get_fdt (void ) +{ +return kho_in.fdt_phys ?phys_to_virt (kho_in.fdt_phys) :NULL ; +} +/** * kho_encode_radix_key - Encodes a physical address and order into a radix key. * @phys: The physical address of the page.
@@ -895,6 +912,128 @@ static void __init kho_reserve_scratch(void) kho_enable = false; } +/* +* Look for free blocks of 1G. This is a heuristic chosen to work efficiently +* with large systems with hundreds of gigabytes of memory. It will work poorly +* on smaller systems. The algorithm itself doesn't depend on the actual value, +* so it can be changed to a different heuristic later if needed. +*/ +#define KHO_EXT_BLKSIZE SZ_1G +#define KHO_EXT_SHIFT const_ilog2(KHO_EXT_BLKSIZE) + +/* Called for the KHO preserved memory radix tree. */ +static int __init kho_ext_walk_leaf (unsigned long key,void *data) +{ +/* Radix tree tracking free blocks. */ +struct kho_radix_tree *tree = data; +phys_addr_t start, end; +unsigned int order; +int err; + + start =kho_decode_radix_key (key, &order); + end = start + (1UL << (order +PAGE_SHIFT )); + +while (start < end) { + err =kho_radix_add_key (tree, start >>KHO_EXT_SHIFT ); +if (err) +return err; + + start += (1UL <<KHO_EXT_SHIFT ); + } + +return 0 ; +} + +/* Called for the KHO preserved memory radix tree. */ +static int __init kho_ext_walk_node (phys_addr_t phys,void *data) +{ +/* Radix tree tracking free blocks. */ +struct kho_radix_tree *tree = data; + +return kho_radix_add_key (tree, phys >>KHO_EXT_SHIFT ); +} + +/* Called for the free block radix tree. */ +static int __init kho_ext_mark_scratch (unsigned long key,void *data) +{ +phys_addr_t *prev_end = data; +phys_addr_t start = key <<KHO_EXT_SHIFT ; +int err; + +if (start > *prev_end) { + err =memblock_mark_kho_scratch (*prev_end, start - *prev_end); +if (err) +return err; + } + + *prev_end = start + (1UL <<KHO_EXT_SHIFT ); +return 0 ; +} + +/* +* kho_extend_scratch - Extend the scratch regions +* +* The KHO preserved memory radix tree mixes both physical address and order +* into a single key. This makes it hard to look for free ranges directly. This +* function first walks the radix tree and digests it down into another radix +* tree, whose keys identify blocks of KHO_EXT_SHIFT which contain preserved +* memory. +* +* Then it walks the digested radix tree and marks everything that doesn't have +* preserved memory as scratch. +* +* NOTE: This function allocates memory so it should be called when scratch has +* available space. +* +* NOTE: The pages of the KHO preserved memory radix tree tables are not marked +* as preserved in the preserved memory tree. But they are expected to remain +* untouched until the tree is fully parsed. So this function also considers +* them to be "preserved memory" and marks their blocks as busy. +* +* NOTE: efi_init()::reserve_regions() removes all regions except +* MEMBLOCK_KHO_SCRATCH. This function adds such regions but they are not KHO +* scratch memory, so they should not be removed. This function should always be +* called after reserve_regions(). +*/ +static void __init kho_extend_scratch (void ) +{ +const struct kho_radix_walk_cb kho_cb = { + .leaf = kho_ext_walk_leaf, + .node = kho_ext_walk_node, + }; +const struct kho_radix_walk_cb ext_cb = { + .leaf = kho_ext_mark_scratch, + }; +struct kho_radix_tree radix; +phys_addr_t prev_end =0 ; +int err =0 ; + + err =kho_radix_init_tree (&radix,NULL ); +if (err) + goto print; + +/* Walk the KHO radix tree to find busy blocks. */ + err =kho_radix_walk_tree (&kho_in.radix_tree, &kho_cb, &radix); +if (err) + goto out; + +/* Walk the blocks and mark everything between keys as scratch. */ + err =kho_radix_walk_tree (&radix, &ext_cb, &prev_end); +if (err) + goto out; + +/* Mark everything from last busy block to end of DRAM. */ +if (prev_end <memblock_end_of_DRAM ()) + err =memblock_mark_kho_scratch (prev_end,memblock_end_of_DRAM () - prev_end); + +/* fallthrough */ +out: +kho_radix_destroy_tree (&radix); +print: +if (err) +pr_err ("Failed to extend scratch: %pe\n" ,ERR_PTR (err)); +} +/** * kho_add_subtree - record the physical address of a sub blob in KHO root tree. * @name: name of the sub tree.
@@ -1469,23 +1608,6 @@ void kho_restore_free(void *mem) }EXPORT_SYMBOL_GPL (kho_restore_free); -struct kho_in { -phys_addr_t fdt_phys; -phys_addr_t scratch_phys; -char previous_release[__NEW_UTS_LEN +1 ]; -u32 kexec_count; -struct kho_debugfs dbg; -struct kho_radix_tree radix_tree; -}; - -static struct kho_in kho_in = { -}; - -static const void *kho_get_fdt (void ) -{ -return kho_in.fdt_phys ?phys_to_virt (kho_in.fdt_phys) :NULL ; -} -/** * is_kho_boot - check if current kernel was booted via KHO-enabled * kexec
@@ -1796,6 +1918,8 @@ void __init kho_memory_init_early(void) kho_in.scratch_phys =0 ;return ; } + +kho_extend_scratch (); }void __init kho_memory_init (void )
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Mike Rapoport
maintainerRe: [PATCH v3 18/21] kho: extend scratch
Quoted 147 lines
Motivation
==========The scratch space is allocated by the first kernel in the KHO chain, and is reused by all subsequent kernels. The size of the space is either set via the commandline by the system administrator or by calculating the amount of memory used by the kernel and adding a multiplier. In either case, the scratch space is a heuristic and is liable to fill up and fail allocation if a kernel uses more memory than expected.
In addition, gigantic huge pages (usually 1 GiB) are allocated via memblock, and in a KHO boot that memory comes from the scratch space. In hypervisors it is common to dedicate a major part of the system's memory to gigantic hugepages for VM memory.
If this memory needs to come from scratch space, then scratch needs to be greater than the memory needed for huge pages, which is impractical. In addition, hugepages can be preserved memory. Allocating them from scratch violates the assumption that scratch contains no preserved memory.
Methodology
===========Discover areas that don't contain any preserved memory at boot by walking the preserved memory radix tree. Mark them as scratch to allow allocations from them. This makes KHO more resilient to memory pressure and allows supporting huge page preservation.
Since the preserved memory radix tree mixes both physical address and order into a single key, and does not track table pages, it is difficult to identify free areas from it directly. Walk the tree and digest it down into another radix tree. The latter tracks blocks of
KHO_EXT_SHIFT(1 GiB as of now) granularity. Then walk the digested tree and mark the areas between the present keys as scratch.Performance
===========The discovery algorithm traverses the preserved memory radix tree exactly once. While it does use memory for the digested radix tree, since the blocks are split by 1 GiB, a single bitmap with 4k pages can track up to 32 TiB of memory. So there are likely to be very few radix tree pages used in this tracking. For systems with all physical memory below 32 TiB, this should result in a total of 6 pages being used (
KHO_TREE_MAX_DEPTH== 6).An alternate way of achieving this would be to call
kho_mem_retrieve()earlier in boot and mark all the KHO preservations as reserved. But that can blow up memblock.reserved with a bunch of 4K pages scattered everywhere, which will reduce performance of subsequent allocations. Since the free blocks are tracked in chunks of 1 GiB, this won't blow up memblock.memory as much.There is no inherent reason for using 1 GiB as the discovered block size. This can be changed later if needed. Currently, KHO is mainly targeted for server grade systems with hundreds of gigabytes to terabytes of memory. So 1 GiB is a reasonable granularity for those systems. For smaller systems this doesn't work as well, but we can arrive at a better heuristic when we have concrete use cases.
Practical evaluation
====================The testing is done on a
x86_64qemu VM running under KVM with 64G memory and 12 CPUs. The machine pre-allocates 50 1G pages.Since the performance scales with how busy the radix tree is, tests are done with 2 preservation patterns: first with two 1M memfds, second with two 1G memfds, both using 4k pages.
Test case 1 - 1M memfd
~~~~~~~~~~~~~~~~~~~~~~This test case has two memfds with 1M memory each in 4k pages, plus other preservations from LUO core and other KHO users.
This is how the radix tree stats look like:
radix_nodes: 0x13 nr_preservations: 0x214 mem_preserved: 0x227000per order preservations: order 0: 0x20f order 1: 0x4 order 4: 0x1and this is how long it takes to extend the scratch after KHO boot:
KHO: KHO extend time: 47 us
KHO: KHO extend total mem:0xe6c17b000(~57G)Test case 2 - 1G memfd
~~~~~~~~~~~~~~~~~~~~~~This test case has two memfds with 1G memory each in 4k pages, plus other preservations from LUO core and other KHO users.
This is how the radix tree stats look like:
radix_nodes: 0x28 nr_preservations: 0x80816 mem_preserved: 0x80829000per order preservations: order 0: 0x80811 order 1: 0x4 order 4: 0x1and this is how long it takes to extend the scratch after KHO boot:
KHO: KHO extend time: 22514 us
KHO: KHO extend total mem:0xd3f200000(~52G)Signed-off-by Pratyush Yadav (Google) <[email protected]>kernel/liveupdate/kexec_handover.c +17
-0@@ -84,6 +84,23 @@ static struct kho_out kho_out = { }, }; +struct kho_in { +phys_addr_t fdt_phys; +phys_addr_t scratch_phys; +char previous_release[__NEW_UTS_LEN +1 ]; +u32 kexec_count; +struct kho_debugfs dbg; +struct kho_radix_tree radix_tree; +}; + +static struct kho_in kho_in = { +}; + +static const void *kho_get_fdt (void ) +{ +return kho_in.fdt_phys ?phys_to_virt (kho_in.fdt_phys) :NULL ; +} +/** * kho_encode_radix_key - Encodes a physical address and order into a radix key. * @phys: The physical address of the page. @@ -895,6 +912,128 @@ static void __init kho_reserve_scratch(void) kho_enable = false;
kernel/liveupdate/kexec_handover.c static void __init kho_reserve_scratch(void) · line 922
Quoted 10 lines
kernel/liveupdate/kexec_handover.c +8
-0} +/* +* Look for free blocks of 1G. This is a heuristic chosen to work efficiently +* with large systems with hundreds of gigabytes of memory. It will work poorly +* on smaller systems. The algorithm itself doesn't depend on the actual value, +* so it can be changed to a different heuristic later if needed. +*/ +#define KHO_EXT_BLKSIZE SZ_1G +#define KHO_EXT_SHIFT const_ilog2(KHO_EXT_BLKSIZE)
Nit: EXT_ what? ;-)
kernel/liveupdate/kexec_handover.c static void __init kho_reserve_scratch(void) · line 925
Quoted 3 lines
kernel/liveupdate/kexec_handover.c +3
-0+ +/* Called for the KHO preserved memory radix tree. */ +static int __init kho_ext_walk_leaf (unsigned long key,void *data)
I have to say this function still confuses me :)
kernel/liveupdate/kexec_handover.c static void __init kho_reserve_scratch(void) · line 933
Quoted 8 lines
kernel/liveupdate/kexec_handover.c +8
-0+{ +/* Radix tree tracking free blocks. */ +struct kho_radix_tree *tree = data; +phys_addr_t start, end; +unsigned int order; +int err; + + start =kho_decode_radix_key (key, &order);
I would add a comment here, that the key is from the memory tracker and it is decoded to an address.
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerRe: [PATCH v3 18/21] kho: extend scratch
Quoted 160 lines · Mike Rapoport
Quoted 147 lines
Motivation
==========The scratch space is allocated by the first kernel in the KHO chain, and is reused by all subsequent kernels. The size of the space is either set via the commandline by the system administrator or by calculating the amount of memory used by the kernel and adding a multiplier. In either case, the scratch space is a heuristic and is liable to fill up and fail allocation if a kernel uses more memory than expected.
In addition, gigantic huge pages (usually 1 GiB) are allocated via memblock, and in a KHO boot that memory comes from the scratch space. In hypervisors it is common to dedicate a major part of the system's memory to gigantic hugepages for VM memory.
If this memory needs to come from scratch space, then scratch needs to be greater than the memory needed for huge pages, which is impractical. In addition, hugepages can be preserved memory. Allocating them from scratch violates the assumption that scratch contains no preserved memory.
Methodology
===========Discover areas that don't contain any preserved memory at boot by walking the preserved memory radix tree. Mark them as scratch to allow allocations from them. This makes KHO more resilient to memory pressure and allows supporting huge page preservation.
Since the preserved memory radix tree mixes both physical address and order into a single key, and does not track table pages, it is difficult to identify free areas from it directly. Walk the tree and digest it down into another radix tree. The latter tracks blocks of
KHO_EXT_SHIFT(1 GiB as of now) granularity. Then walk the digested tree and mark the areas between the present keys as scratch.Performance
===========The discovery algorithm traverses the preserved memory radix tree exactly once. While it does use memory for the digested radix tree, since the blocks are split by 1 GiB, a single bitmap with 4k pages can track up to 32 TiB of memory. So there are likely to be very few radix tree pages used in this tracking. For systems with all physical memory below 32 TiB, this should result in a total of 6 pages being used (
KHO_TREE_MAX_DEPTH== 6).An alternate way of achieving this would be to call
kho_mem_retrieve()earlier in boot and mark all the KHO preservations as reserved. But that can blow up memblock.reserved with a bunch of 4K pages scattered everywhere, which will reduce performance of subsequent allocations. Since the free blocks are tracked in chunks of 1 GiB, this won't blow up memblock.memory as much.There is no inherent reason for using 1 GiB as the discovered block size. This can be changed later if needed. Currently, KHO is mainly targeted for server grade systems with hundreds of gigabytes to terabytes of memory. So 1 GiB is a reasonable granularity for those systems. For smaller systems this doesn't work as well, but we can arrive at a better heuristic when we have concrete use cases.
Practical evaluation
====================The testing is done on a
x86_64qemu VM running under KVM with 64G memory and 12 CPUs. The machine pre-allocates 50 1G pages.Since the performance scales with how busy the radix tree is, tests are done with 2 preservation patterns: first with two 1M memfds, second with two 1G memfds, both using 4k pages.
Test case 1 - 1M memfd
~~~~~~~~~~~~~~~~~~~~~~This test case has two memfds with 1M memory each in 4k pages, plus other preservations from LUO core and other KHO users.
This is how the radix tree stats look like:
radix_nodes: 0x13 nr_preservations: 0x214 mem_preserved: 0x227000per order preservations: order 0: 0x20f order 1: 0x4 order 4: 0x1and this is how long it takes to extend the scratch after KHO boot:
KHO: KHO extend time: 47 us
KHO: KHO extend total mem:0xe6c17b000(~57G)Test case 2 - 1G memfd
~~~~~~~~~~~~~~~~~~~~~~This test case has two memfds with 1G memory each in 4k pages, plus other preservations from LUO core and other KHO users.
This is how the radix tree stats look like:
radix_nodes: 0x28 nr_preservations: 0x80816 mem_preserved: 0x80829000per order preservations: order 0: 0x80811 order 1: 0x4 order 4: 0x1and this is how long it takes to extend the scratch after KHO boot:
KHO: KHO extend time: 22514 us
KHO: KHO extend total mem:0xd3f200000(~52G)Signed-off-by Pratyush Yadav (Google) <[email protected]>kernel/liveupdate/kexec_handover.c +17
-0@@ -84,6 +84,23 @@ static struct kho_out kho_out = { }, }; +struct kho_in { +phys_addr_t fdt_phys; +phys_addr_t scratch_phys; +char previous_release[__NEW_UTS_LEN +1 ]; +u32 kexec_count; +struct kho_debugfs dbg; +struct kho_radix_tree radix_tree; +}; + +static struct kho_in kho_in = { +}; + +static const void *kho_get_fdt (void ) +{ +return kho_in.fdt_phys ?phys_to_virt (kho_in.fdt_phys) :NULL ; +} +/** * kho_encode_radix_key - Encodes a physical address and order into a radix key. * @phys: The physical address of the page. @@ -895,6 +912,128 @@ static void __init kho_reserve_scratch(void) kho_enable = false;kernel/liveupdate/kexec_handover.c static void __init kho_reserve_scratch(void) · line 922
Quoted 10 lines
kernel/liveupdate/kexec_handover.c +8
-0} +/* +* Look for free blocks of 1G. This is a heuristic chosen to work efficiently +* with large systems with hundreds of gigabytes of memory. It will work poorly +* on smaller systems. The algorithm itself doesn't depend on the actual value, +* so it can be changed to a different heuristic later if needed. +*/ +#define KHO_EXT_BLKSIZE SZ_1G +#define KHO_EXT_SHIFT const_ilog2(KHO_EXT_BLKSIZE)Nit:
EXT_what? ;-)
Quoted 5 lines
kernel/liveupdate/kexec_handover.c static void __init kho_reserve_scratch(void) · line 925
Quoted 3 lines
kernel/liveupdate/kexec_handover.c +3
-0+ +/* Called for the KHO preserved memory radix tree. */ +static int __init kho_ext_walk_leaf (unsigned long key,void *data)I have to say this function still confuses me :)
Honestly, I think this whole tree walking API makes things really hard to reason about. I think we should replace it with something like kho_radix_for_each(). It would improve code readability a lot.
I plan to do it at some point, but not as a part of this series.
Quoted 12 lines
kernel/liveupdate/kexec_handover.c static void __init kho_reserve_scratch(void) · line 933
Quoted 9 lines
kernel/liveupdate/kexec_handover.c +8
-0+{ +/* Radix tree tracking free blocks. */ +struct kho_radix_tree *tree = data; +phys_addr_t start, end; +unsigned int order; +int err; + + start =kho_decode_radix_key (key, &order);I would add a comment here, that the key is from the memory tracker and it is decoded to an address.
ACK, will add.
Signature · 2 lines
Regards, Pratyush Yadav
Pratyush Yadav
maintainerPATCH v3 19/21 memblock: make HugeTLB bootmem allocation work with KHO
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Gigantic huge page allocation is somewhat broken currently when KHO is used.
Firstly, they break KHO scratch size accounting. RSRV_KERN is used to track how much memory is reserved for use by the kernel. Since hugetlb::alloc_bootmem() calls the memblock_alloc*() APIs, the hugepages allocated also get marked as RSRV_KERN.
Allocations marked RSRV_KERN are used by KHO to calculate how much scratch space it should reserve to make sure the next kernel has enough memory to boot when it is in scratch-only phase. Counting hugepages in that blows up scratch size, and can lead to the scratch allocation failing, making KHO unusable. This will show up when huge pages make up more than 50% of the system, which is a fairly common use case.
Secondly, while not supported right now, huge pages are user memory and can be preserved via KHO. The scratch spaces should not have any preserved memory. Allocating hugepages from scratch (on a KHO boot) can lead to them being un-preservable.
Introduce memblock_alloc_hugetlb(). This lets memblock tailor to the needs of hugetb without exposing those details to the general allocation routines.
First, it does not use mirrored memory for hugetlb. Mirrored memory is a limited resource that is best saved for kernel data structures, not user memory.
Second, if the free memory area found by memblock_find_in_range_node() is a part of a KHO scratch area, the free area is not used. Allocation is retried starting after the free area to ensure no hugepages come from KHO scratch.
Third, it simplifies the argument list by baking in some hugetlb assumptions like alignment and exact_nid. This also simplifies allocation logic in alloc_bootmem().
Also introduce MEMBLOCK_RSRV_HUGETLB to mark reservations made for HugeTLB. This will be used by KHO in future patches to correctly calculate scratch sizes.
Refactor some of the preparation logic like kmemleak tracking and accepting memory into a separate helper memblock_prep_allocation(), and use it from both memblock_alloc_hugetlb() and the usual memblock_alloc_range_nid().
Add a stub for kho_scratch_overlap to memblock tests to make sure it compiles.
--- include/linux/memblock.h | 3 + mm/hugetlb.c | 22 ++---- mm/memblock.c | 112 ++++++++++++++++++++++++------ tools/testing/memblock/internal.h | 5 ++ 4 files changed, 105 insertions(+), 37 deletions(-)
include/linux/memblock.h +3 -0
@@ -51,6 +51,7 @@ extern unsigned long long max_possible_pfn;* memory reservations yet, so we get scratch memory from the previous * kernel that we know is good to use. It is the only memory that * allocations may happen from in this phase. +* @MEMBLOCK_RSRV_HUGETLB: memory is reserved for hugetlb pages */ enum memblock_flags {MEMBLOCK_NONE =0x0 ,/* No special request */
@@ -61,6 +62,7 @@ enum memblock_flags {MEMBLOCK_RSRV_NOINIT =0x10 ,/* don't initialize struct pages */ MEMBLOCK_RSRV_KERN =0x20 ,/* memory reserved for kernel use */ MEMBLOCK_KHO_SCRATCH =0x40 ,/* scratch memory for kexec handover */ + MEMBLOCK_RSRV_HUGETLB = 0x80, /* memory reserved for hugetlb pages */ };/**
@@ -420,6 +422,7 @@ void *memblock_alloc_try_nid_raw(phys_addr_t size, phys_addr_t align,void *memblock_alloc_try_nid (phys_addr_t size,phys_addr_t align,phys_addr_t min_addr,phys_addr_t max_addr,int nid); +void *memblock_alloc_hugetlb (phys_addr_t size,int nid,bool exact_nid); static __always_inline void *memblock_alloc(phys_addr_t size, phys_addr_t align) {
mm/hugetlb.c +7 -15
@@ -3033,29 +3033,21 @@ static __init void *alloc_bootmem(struct hstate *h, int nid, bool node_exact)if (hugetlb_early_cma (h)) m =hugetlb_cma_alloc_bootmem (h, &listnode, node_exact);else { -if (node_exact) - m =memblock_alloc_exact_nid_raw (huge_page_size (h), -huge_page_size (h),0 , -MEMBLOCK_ALLOC_ACCESSIBLE , nid); -else { - m =memblock_alloc_try_nid_raw (huge_page_size (h), -huge_page_size (h),0 , -MEMBLOCK_ALLOC_ACCESSIBLE , nid); + m =memblock_alloc_hugetlb (huge_page_size (h), nid, node_exact); +if (m) { + m->flags =0 ; + m->cma =NULL ; +/* * For pre-HVO to work correctly, pages need to be on * the list for the node they were actually allocated * from. That node may be different in the case of -* fallback by memblock_alloc_try_nid_raw. So, +* fallback by memblock_alloc_hugetlb_bootmem. So, * extract the actual node first. */ -if (m) +if (!node_exact) listnode =early_pfn_to_nid (PHYS_PFN (__pa (m))); } - -if (m) { - m->flags =0 ; - m->cma =NULL ; - } }if (m) {
mm/memblock.c +90 -22
130 lines not on this page · raw message
tools/testing/memblock/internal.h +5 -0
@@ -66,4 +66,9 @@ static inline void init_deferred_page(unsigned long pfn, int nid)#define __SetPageReserved (p) ((void)(p)) +static inline bool kho_scratch_overlap (phys_addr_t phys,size_t size) +{ +return false; +} +#endif
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Mike Rapoport
maintainerRe: [PATCH v3 19/21] memblock: make HugeTLB bootmem allocation work with KHO
Quoted 125 lines
Gigantic huge page allocation is somewhat broken currently when KHO is used.
Firstly, they break KHO scratch size accounting.
RSRV_KERNis used to track how much memory is reserved for use by the kernel. Sincehugetlb::alloc_bootmem()calls the memblock_alloc*() APIs, the hugepages allocated also get marked asRSRV_KERN.Allocations marked
RSRV_KERNare used by KHO to calculate how much scratch space it should reserve to make sure the next kernel has enough memory to boot when it is in scratch-only phase. Counting hugepages in that blows up scratch size, and can lead to the scratch allocation failing, making KHO unusable. This will show up when huge pages make up more than 50% of the system, which is a fairly common use case.Secondly, while not supported right now, huge pages are user memory and can be preserved via KHO. The scratch spaces should not have any preserved memory. Allocating hugepages from scratch (on a KHO boot) can lead to them being un-preservable.
Introduce
memblock_alloc_hugetlb(). This lets memblock tailor to the needs of hugetb without exposing those details to the general allocation routines.First, it does not use mirrored memory for hugetlb. Mirrored memory is a limited resource that is best saved for kernel data structures, not user memory.
Second, if the free memory area found by
memblock_find_in_range_node()is a part of a KHO scratch area, the free area is not used. Allocation is retried starting after the free area to ensure no hugepages come from KHO scratch.Third, it simplifies the argument list by baking in some hugetlb assumptions like alignment and
exact_nid. This also simplifies allocation logic inalloc_bootmem().Also introduce
MEMBLOCK_RSRV_HUGETLBto mark reservations made for HugeTLB. This will be used by KHO in future patches to correctly calculate scratch sizes.Refactor some of the preparation logic like kmemleak tracking and accepting memory into a separate helper
memblock_prep_allocation(), and use it from bothmemblock_alloc_hugetlb()and the usualmemblock_alloc_range_nid().Add a stub for
kho_scratch_overlapto memblock tests to make sure it compiles.Signed-off-by Pratyush Yadav (Google) <[email protected]>include/linux/memblock.h +3
-0@@ -51,6 +51,7 @@ extern unsigned long long max_possible_pfn;* memory reservations yet, so we get scratch memory from the previous * kernel that we know is good to use. It is the only memory that * allocations may happen from in this phase. +* @MEMBLOCK_RSRV_HUGETLB: memory is reserved for hugetlb pages */ enum memblock_flags {MEMBLOCK_NONE =0x0 ,/* No special request */ @@ -61,6 +62,7 @@ enum memblock_flags {MEMBLOCK_RSRV_NOINIT =0x10 ,/* don't initialize struct pages */ MEMBLOCK_RSRV_KERN =0x20 ,/* memory reserved for kernel use */ MEMBLOCK_KHO_SCRATCH =0x40 ,/* scratch memory for kexec handover */ + MEMBLOCK_RSRV_HUGETLB = 0x80, /* memory reserved for hugetlb pages */ };/** @@ -420,6 +422,7 @@ void *memblock_alloc_try_nid_raw(phys_addr_t size, phys_addr_t align,void *memblock_alloc_try_nid (phys_addr_t size,phys_addr_t align,phys_addr_t min_addr,phys_addr_t max_addr,int nid); +void *memblock_alloc_hugetlb (phys_addr_t size,int nid,bool exact_nid); static __always_inline void *memblock_alloc(phys_addr_t size, phys_addr_t align) {mm/hugetlb.c +7
-15@@ -3033,29 +3033,21 @@ static __init void *alloc_bootmem(struct hstate *h, int nid, bool node_exact)if (hugetlb_early_cma (h)) m =hugetlb_cma_alloc_bootmem (h, &listnode, node_exact);else { -if (node_exact) - m =memblock_alloc_exact_nid_raw (huge_page_size (h), -huge_page_size (h),0 , -MEMBLOCK_ALLOC_ACCESSIBLE , nid); -else { - m =memblock_alloc_try_nid_raw (huge_page_size (h), -huge_page_size (h),0 , -MEMBLOCK_ALLOC_ACCESSIBLE , nid); + m =memblock_alloc_hugetlb (huge_page_size (h), nid, node_exact); +if (m) { + m->flags =0 ; + m->cma =NULL ; +/* * For pre-HVO to work correctly, pages need to be on * the list for the node they were actually allocated * from. That node may be different in the case of -* fallback by memblock_alloc_try_nid_raw. So, +* fallback by memblock_alloc_hugetlb_bootmem. So, * extract the actual node first. */ -if (m) +if (!node_exact) listnode =early_pfn_to_nid (PHYS_PFN (__pa (m))); } - -if (m) { - m->flags =0 ; - m->cma =NULL ; - } }if (m) {mm/memblock.c +0
-0
mm/memblock.c int __init_memblock memblock_set_node(phys_addr_t base, phys_addr_t size, · line 1510
Quoted 5 lines
mm/memblock.c +2
-0@@ -1506,6 +1506,32 @@ int __init_memblock memblock_set_node(phys_addr_t base, phys_addr_t size,return 0 ; } +static void memblock_prep_allocation (phys_addr_t start,phys_addr_t size, +bool leaktrace)
Let's call this kmemleak_trace
Quoted 132 lines
mm/memblock.c +88
-22+{ +/* +* Skip kmemleak for those places like kasan_init() and +* early_pgtable_alloc() due to high volume. +*/ +if (leaktrace) +/* +* Memblock allocated blocks are never reported as +* leaks. This is because many of these blocks are +* only referred via the physical address which is +* not looked up by kmemleak. +*/ +kmemleak_alloc_phys (start, size,0 ); + +/* +* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, +* require memory to be accepted before it can be used by the +* guest. +* +* Accept the memory of the allocated buffer. +*/ +accept_memory (start, size); +} +/** * memblock_alloc_range_nid - allocate boot memory block * @size: size of memory block to be allocated in bytes @@ -1580,28 +1606,7 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,return 0 ; done: -/* -* Skip kmemleak for those places like kasan_init() and -* early_pgtable_alloc() due to high volume. -*/ -if (end !=MEMBLOCK_ALLOC_NOLEAKTRACE ) -/* -* Memblock allocated blocks are never reported as -* leaks. This is because many of these blocks are -* only referred via the physical address which is -* not looked up by kmemleak. -*/ -kmemleak_alloc_phys (found, size,0 ); - -/* -* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, -* require memory to be accepted before it can be used by the -* guest. -* -* Accept the memory of the allocated buffer. -*/ -accept_memory (found, size); - +memblock_prep_allocation (found, size, end !=MEMBLOCK_ALLOC_NOLEAKTRACE );return found; }@@ -1756,6 +1761,69 @@ void * __init memblock_alloc_try_nid_raw( false); } +/** +* memblock_alloc_hugetlb - allocate boot memory for HugeTLB pages +* @size: size of the memory to be allocated in bytes +* @nid: nid of the free memory to find, %NUMA_NO_NODE for any node +* @exact_nid: only allocate from the specified nid. If %false, the specified +* nid is tried first, and then all nodes are tried as fallback. +* +* HugeTLB pages are always aligned by their size, so the alignment matches +* @size. Since the memory is for userspace, mirrored memory is not used. The +* memory is not zeroed. Does not panic if request cannot be satisfied. +* +* Return: +* Virtual address of allocated memory block on success, %NULL on failure. +*/ +void *__init memblock_alloc_hugetlb (phys_addr_t size,int nid,bool exact_nid) +{ +enum memblock_flags flags =choose_memblock_flags (); +phys_addr_t addr, start =0 , end =MEMBLOCK_ALLOC_ACCESSIBLE ; + +memblock_dbg ("%s: %llu bytes, nid=%d, exact_nid=%d %pS\n" , __func__, + (u64 )size, nid, exact_nid, (void *)_RET_IP_); + +/* Don't waste mirrored memory on HugeTLB pages. */ + flags &= ~MEMBLOCK_MIRROR ; +retry: +/* HugeTLB pages are always aligned by their size. */ + addr =memblock_find_in_range_node (size, size, start, end, nid, flags); +if (addr) + goto found; + +/* Try all nodes if allowed. */ +if (numa_valid_node (nid) && !exact_nid) { + nid =NUMA_NO_NODE ; + goto retry; + } + +/* Found nothing... :-( */ +return NULL ; + +found: +/* +* HugeTLB pages can be preserved with KHO and no preserved memory can +* be in scratch. So retry if found address overlaps with scratch. +* +* Scratch areas are normally not very large, so this shouldn't take too +* many retries. +*/ +if (kho_scratch_overlap (addr, size)) { +if (memblock_bottom_up ()) + start = addr + size; +else + start = addr - size; + + goto retry; + } + +if (__memblock_reserve (addr, size, nid,MEMBLOCK_RSRV_KERN |MEMBLOCK_RSRV_HUGETLB )) +return NULL ; + +memblock_prep_allocation (addr, size, true); +return phys_to_virt (addr); +} +/** * memblock_alloc_try_nid - allocate boot memory block * @size: size of memory block to be allocated in bytes tools/testing/memblock/internal.h +0
-0
tools/testing/memblock/internal.h static inline void init_deferred_page(unsigned long pfn, int nid) · line 73
Quoted 8 lines
tools/testing/memblock/internal.h +5
-0@@ -66,4 +66,9 @@ static inline void init_deferred_page(unsigned long pfn, int nid)#define __SetPageReserved (p) ((void)(p)) +static inline bool kho_scratch_overlap (phys_addr_t phys,size_t size) +{ +return false; +} +
This should be in the patch that exposes kho_scratch_overlap() to kexec_handover.h
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerRe: [PATCH v3 19/21] memblock: make HugeTLB bootmem allocation work with KHO
Quoted 50 lines · Mike Rapoport
Quoted 50 lines
Gigantic huge page allocation is somewhat broken currently when KHO is used.
Firstly, they break KHO scratch size accounting.
RSRV_KERNis used to track how much memory is reserved for use by the kernel. Sincehugetlb::alloc_bootmem()calls the memblock_alloc*() APIs, the hugepages allocated also get marked asRSRV_KERN.Allocations marked
RSRV_KERNare used by KHO to calculate how much scratch space it should reserve to make sure the next kernel has enough memory to boot when it is in scratch-only phase. Counting hugepages in that blows up scratch size, and can lead to the scratch allocation failing, making KHO unusable. This will show up when huge pages make up more than 50% of the system, which is a fairly common use case.Secondly, while not supported right now, huge pages are user memory and can be preserved via KHO. The scratch spaces should not have any preserved memory. Allocating hugepages from scratch (on a KHO boot) can lead to them being un-preservable.
Introduce
memblock_alloc_hugetlb(). This lets memblock tailor to the needs of hugetb without exposing those details to the general allocation routines.First, it does not use mirrored memory for hugetlb. Mirrored memory is a limited resource that is best saved for kernel data structures, not user memory.
Second, if the free memory area found by
memblock_find_in_range_node()is a part of a KHO scratch area, the free area is not used. Allocation is retried starting after the free area to ensure no hugepages come from KHO scratch.Third, it simplifies the argument list by baking in some hugetlb assumptions like alignment and
exact_nid. This also simplifies allocation logic inalloc_bootmem().Also introduce
MEMBLOCK_RSRV_HUGETLBto mark reservations made for HugeTLB. This will be used by KHO in future patches to correctly calculate scratch sizes.Refactor some of the preparation logic like kmemleak tracking and accepting memory into a separate helper
memblock_prep_allocation(), and use it from bothmemblock_alloc_hugetlb()and the usualmemblock_alloc_range_nid().Add a stub for
kho_scratch_overlapto memblock tests to make sure it compiles.Signed-off-by Pratyush Yadav (Google) <[email protected]>
[...]
Quoted 8 lines
Quoted 8 lines
mm/memblock.c +0
-0@@ -1506,6 +1506,32 @@ int __init_memblock memblock_set_node(phys_addr_t base, phys_addr_t size,return 0 ; }
Quoted 4 lines
mm/memblock.c int __init_memblock memblock_set_node(phys_addr_t base, phys_addr_t size, · line 1510
Quoted 2 lines
mm/memblock.c +2
-0+static void memblock_prep_allocation (phys_addr_t start,phys_addr_t size, +bool leaktrace)Let's call this
kmemleak_trace
ACK.
Quoted 14 lines
Quoted 14 lines
mm/memblock.c +13
-0+{ +/* +* Skip kmemleak for those places like kasan_init() and +* early_pgtable_alloc() due to high volume. +*/ +if (leaktrace) +/* +* Memblock allocated blocks are never reported as +* leaks. This is because many of these blocks are +* only referred via the physical address which is +* not looked up by kmemleak. +*/ +kmemleak_alloc_phys (start, size,0 );
[...]
Quoted 8 lines
Quoted 8 lines
tools/testing/memblock/internal.h +0
-0@@ -66,4 +66,9 @@ static inline void init_deferred_page(unsigned long pfn, int nid)#define __SetPageReserved (p) ((void)(p))
Quoted 7 lines
tools/testing/memblock/internal.h static inline void init_deferred_page(unsigned long pfn, int nid) · line 73
Quoted 5 lines
tools/testing/memblock/internal.h +5
-0+static inline bool kho_scratch_overlap (phys_addr_t phys,size_t size) +{ +return false; +} +This should be in the patch that exposes
kho_scratch_overlap()tokexec_handover.h
Hm, sure, why not. I added it here because _this patch_ uses kho_scratch_overlap() for the first time on memblock.c, and with that breaks build of the memblock tests. So I recokned it makes sense to have it in the place that broke it.
But I don't mind, I can move it to the other patch as well.
Signature · 2 lines
Regards, Pratyush Yadav
Pratyush Yadav
maintainerPATCH v3 20/21 memblock: add memblock_reserved_hugetlb_size()
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
Similar to memblock_reserved_kern_size(), but calculates only the memory reserved for hugetlb pages. This is needed in an upcoming commit that subtracts hugetlb reservation size when computing the size of KHO scratch areas.
--- include/linux/memblock.h | 1 + mm/memblock.c | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+)
include/linux/memblock.h +1 -0
@@ -487,6 +487,7 @@ static inline __init_memblock bool memblock_bottom_up(void)phys_addr_t memblock_phys_mem_size (void );phys_addr_t memblock_reserved_size (void );phys_addr_t memblock_reserved_kern_size (phys_addr_t limit,int nid); +phys_addr_t memblock_reserved_hugetlb_size (phys_addr_t limit,int nid);unsigned long memblock_estimated_nr_free_pages (void );phys_addr_t memblock_start_of_DRAM (void );phys_addr_t memblock_end_of_DRAM (void );
mm/memblock.c +22 -0
@@ -1893,6 +1893,28 @@ phys_addr_t __init_memblock memblock_reserved_size(void)return memblock.reserved.total_size; } +phys_addr_t __init_memblock memblock_reserved_hugetlb_size(phys_addr_t limit, int nid) +{ +struct memblock_region *r; +phys_addr_t total =0 ; + +for_each_reserved_mem_region (r) { +phys_addr_t size = r->size; + +if (r->base > limit) +break ; + +if (r->base + r->size > limit) + size = limit - r->base; + +if (nid ==memblock_get_region_node (r) || !numa_valid_node (nid)) +if (r->flags &MEMBLOCK_RSRV_HUGETLB ) + total += size; + } + +return total; +} + phys_addr_t __init_memblock memblock_reserved_kern_size(phys_addr_t limit, int nid) {struct memblock_region *r;
Signature · 1 line
2.55.0.141.g00534a21ce-goog
Pratyush Yadav
maintainerPATCH v3 21/21 kho: exclude hugetlb memory from scratch size calculation
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
HugeTLB pages can be preserved memory. So they are never allocated from scratch. Instead, they are allocated from the memory blocks with no preserved memory. These areas are detected at runtime on each boot.
But since they are allocated via memblock, they show up as RSRV_KERN, and blow up the scratch size when scratch scale is in use.
All hugetlb pages are marked RSRV_HUGETLB. Subtract their size from RSRV_KERN when calculating scratch sizes.
--- kernel/liveupdate/kexec_handover.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
kernel/liveupdate/kexec_handover.c +10 -2
@@ -742,19 +742,24 @@ early_param("kho_scratch", kho_parse_scratch_size);static void __init scratch_size_update (void ) {/* -* If fixed sizes are not provided via command line, calculate them -* now. +* If fixed sizes are not provided via command line, calculate them now. +* Remove HugeTLB allocations from it because they never get allocated +* from scratch. */ if (scratch_scale) {phys_addr_t size; size =memblock_reserved_kern_size (ARCH_LOW_ADDRESS_LIMIT ,NUMA_NO_NODE ); + size -=memblock_reserved_hugetlb_size (ARCH_LOW_ADDRESS_LIMIT , +NUMA_NO_NODE ); size = size * scratch_scale /100 ; scratch_size_lowmem = size; size =memblock_reserved_kern_size (MEMBLOCK_ALLOC_ANYWHERE ,NUMA_NO_NODE ); + size -=memblock_reserved_hugetlb_size (MEMBLOCK_ALLOC_ANYWHERE , +NUMA_NO_NODE ); size = size * scratch_scale /100 - scratch_size_lowmem; scratch_size_global = size; }
@@ -774,6 +779,9 @@ static phys_addr_t __init scratch_size_node(int nid)if (scratch_scale) { size =memblock_reserved_kern_size (MEMBLOCK_ALLOC_ANYWHERE , nid); +/* Do not count HugeTLB pages. */ + size -=memblock_reserved_hugetlb_size (MEMBLOCK_ALLOC_ANYWHERE , + nid); size = size * scratch_scale /100 ; }else { size = scratch_size_pernode;
Signature · 1 line
2.55.0.141.g00534a21ce-goog