kho: add support for preserving vmalloc allocations

From: "Mike Rapoport (Microsoft)" <[email protected]>

Hi,

Following the discussion about preservation of memfd with LUO [1] these patches add support for preserving vmalloc allocations.

Any KHO uses case presumes that there's a data structure that lists physical addresses of preserved folios (and potentially some additional metadata). Allowing vmalloc preservations with KHO allows scalable preservation of such data structures.

For instance, instead of allocating array describing preserved folios in the fdt, memfd preservation can use vmalloc:

        preserved_folios = vmalloc_array(nr_folios, sizeof(*preserved_folios));
        memfd_luo_preserve_folios(preserved_folios, folios, nr_folios);
        kho_preserve_vmalloc(preserved_folios, &folios_info);

[1] https://lore.kernel.org/all/[email protected]

Mike Rapoport (Microsoft) (2):
kho: add support for preserving vmalloc allocations lib/test_kho: use kho_preserve_vmalloc instead of storing addresses in fdt

 include/linux/kexec_handover.h |  12 +++
 kernel/kexec_handover.c        | 140 +++++++++++++++++++++++++++++++++
 lib/test_kho.c                 |  30 +++++--
 3 files changed, 175 insertions(+), 7 deletions(-)

base-commit: b320789d6883cc00ac78ce83bccbfe7ed58afcf0 --
2.50.1

From: "Mike Rapoport (Microsoft)" <[email protected]>

A vmalloc allocation is preserved using binary structure similar to global KHO memory tracker. It's a linked list of pages where each page is an array of physical address of pages in vmalloc area.

kho_preserve_vmalloc() hands out the physical address of the head page to the caller. This address is used as the argument to kho_vmalloc_restore() to restore the mapping in the vmalloc address space and populate it with the preserved pages.

---
 include/linux/kexec_handover.h |  12 +++
 kernel/kexec_handover.c        | 140 +++++++++++++++++++++++++++++++++
 2 files changed, 152 insertions(+)

index 348844cffb13..b7bf3bf11019 100644
@@ -42,8 +42,10 @@ struct kho_serialization;
 bool kho_is_enabled(void);
 
 int kho_preserve_folio(struct folio *folio);
+int kho_preserve_vmalloc(void *ptr, phys_addr_t *preservation);
 int kho_preserve_phys(phys_addr_t phys, size_t size);
 struct folio *kho_restore_folio(phys_addr_t phys);
+void *kho_restore_vmalloc(phys_addr_t preservation);
 int kho_add_subtree(struct kho_serialization *ser, const char *name, void *fdt);
 int kho_retrieve_subtree(const char *name, phys_addr_t *phys);
 

@@ -70,11 +72,21 @@ static inline int kho_preserve_phys(phys_addr_t phys, size_t size)
 	return -EOPNOTSUPP;
 }
 
+static inline int kho_preserve_vmalloc(void *ptr, phys_addr_t *preservation)
+{
+	return -EOPNOTSUPP;
+}
+
 static inline struct folio *kho_restore_folio(phys_addr_t phys)
 {
 	return NULL;
 }
 
+static inline void *kho_restore_vmalloc(phys_addr_t preservation)
+{
+	return NULL;
+}
+
 static inline int kho_add_subtree(struct kho_serialization *ser,
 				  const char *name, void *fdt)
 {

2.50.1

We can add version or flags to kho_vmalloc_chunk, e.g. make it

struct kho_vmalloc_hdr {
	DECLARE_KHOSER_PTR(next, struct kho_vmalloc_chunk *);
	unsigned int total_pages;	/* only valid in the first chunk */
	unsigned short version;		/* only valid in the first chunk */
	unsigned short num_elms;
};

I'm thinking about actually adding support for HUGE_VMAP for the next resping, but version/flags seems useful anyway.

Ok, I'll add kho_preserve_page() ;-P.

Now seriously, by no means this is a folio, so it's either kho_preserve_phys() or __kho_preserve_order(). I don't mind switching to latter, but I really see no point doing it.

How do you suggest to preserve memblock?

Sincerely yours,
Mike.

Cast it to a folio :P

It really is. The entire bitmap thing is about preserving folios/page which are basically the same thing ATM. folio is the prefered type for what used to be compound pages.

As Matthew moves ahead it will effectively become preserving memdescs. This may even start to happen this year..

Every memdesc has a type, so when ever the physical pages are restored KHO will need to recreate the struct page and page->memdesc with the correct values, including the memdesc type code and any memdesc allocation that Matthew plans.

Meaning everything should be struct page or folio based at this API level, and restore functions should be logically paired with the allocation functions that created the memory in the first place.

vmalloc() is calling alloc_pages_bulk_node_noprof() to allocate the memory, so the restore of that memory should also have a 'kho restore page' type of name that clearly refers back to the allocator it pairs with.

In the more general case this should be setting the cgroup and charging it as well.

Does the memory have a struct page? Then it should be a preserved folio list so you get back struct pages in the right state for what memblock is doing. Someday that will turn into some specific memdesc type and so on.

If it doesn't have a struct page then it shouldn't be in the bitmaps at all.

Jason

I'm actually all for having a single entry point
kho_{preserve,restore}_page() that will do

	if (folio)
		do_folio()
	else if (vmalloc)
		do_vmalloc()

etc.

It seems that our major disagreement is about using 'folio' vs 'page' in the naming.

In my view calling everything 'folio' is a bad idea as we are moving fast from Ottawa interpretation to New York interpretation of folio.

I'd rather stick to the good old 'page' and when the time comes we can 's/page/memdesc/g' supposing Matthew actually plans for it.

This way we won't need to handle the fallback from divorce of folio from page. This indeed is less relevant to KHO, but there are a lot of folio_alloc() in LUO and PCI patches that will have to be changed to a different allocation apparently this year.

Yes, eventually :)

page list you mean ;-)

There is a struct page for everything that's memblock_alloc()ed. And we can do page list, but for large physically contiguous allocation it does not make sense.

I'd rather replace kho_preserve_phys() with kho_preserve_memblock() and add a restore counterpart to properly set the struct pages for it which we lack now.

Sincerely yours,
Mike.

It is a folio because folio is the name for something that is a high order page and it signals that the pointer is the head page. Which is excatly what KHO preservation works on.

I don't know what the next step is when folio is split - presumably we will get a new type to represent an abstract memdesc head of a high order allocation that the lowest KHO primitives will change over to.

I think you should just convert from the vmap page to folio for now and most likely vmap will stop using page someday..

I'm not sure what to do about this, really we should make slab work :\

But yes whatever it gets changed into needs to have matching restoration. Most likely we will get some kind of API to allocate PAGE_SIZE units of frozen pages with a special memdesc, so that will need to be paired through the preserver/restore as well.

Arguably you could make them into high order pages and preserve those..

Sure, but my point is if memblock physical memory has struct pages then you should be preserving the struct pages as struct pages not as phys. Upon restoring the 64 bytes of struct page memory should be restored back to whatever memblock expects, which will eventually include the proper memdesc value.

Jason

kmalloc_large() and vmalloc(VMAP_HUGE) are not folios and won't be.

This is wrong. vmalloc is not a folio and according to memdesc plan [1] it will be be page until it becomes memdesc.

They are not aligned by order and they may be partially freed starting at arbitrary page. Making them high order pages will be a mess.

[1] https://kernelnewbies.org/MatthewWilcox/Memdescs

Sincerely yours,
Mike.

From: "Mike Rapoport (Microsoft)" <[email protected]>

KHO test stores physical addresses of the preserved folios directly in fdt.
Use kho_preserve_vmalloc() instead of it and kho_restore_vmalloc() to retrieve the addresses after kexec.

This makes the test more scalable from one side and adds tests coverage for kho_preserve_vmalloc() from the other.

---
 lib/test_kho.c | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)

index c2eb899c3b45..10045f5979a0 100644
@@ -32,6 +32,7 @@ module_param(max_mem, long, 0644);
 struct kho_test_state {
 	unsigned int nr_folios;
 	struct folio **folios;
+	phys_addr_t *folios_info;
 	struct folio *fdt;
 	__wsum csum;
 };

@@ -68,13 +69,17 @@ static struct notifier_block kho_test_nb = {
 static int kho_test_save_data(struct kho_test_state *state, void *fdt)
 {
 	phys_addr_t *folios_info __free(kvfree) = NULL;
+	phys_addr_t folios_info_phys;
 	int err = 0;
 
-	folios_info = kvmalloc_array(state->nr_folios, sizeof(*folios_info),
-				     GFP_KERNEL);
+	folios_info = vmalloc_array(state->nr_folios, sizeof(*folios_info));
 	if (!folios_info)
 		return -ENOMEM;
 
+	err = kho_preserve_vmalloc(folios_info, &folios_info_phys);
+	if (err)
+		return err;
+
 	for (int i = 0; i < state->nr_folios; i++) {
 		struct folio *folio = state->folios[i];
 		unsigned int order = folio_order(folio);

@@ -89,11 +94,14 @@ static int kho_test_save_data(struct kho_test_state *state, void *fdt)
 	err |= fdt_begin_node(fdt, "data");
 	err |= fdt_property(fdt, "nr_folios", &state->nr_folios,
 			    sizeof(state->nr_folios));
-	err |= fdt_property(fdt, "folios_info", folios_info,
-			    state->nr_folios * sizeof(*folios_info));
+	err |= fdt_property(fdt, "folios_info", &folios_info_phys,
+			    sizeof(folios_info_phys));
 	err |= fdt_property(fdt, "csum", &state->csum, sizeof(state->csum));
 	err |= fdt_end_node(fdt);
 
+	if (!err)
+		state->folios_info = no_free_ptr(folios_info);
+
 	return err;
 }
 

@@ -197,7 +205,8 @@ static int kho_test_save(void)
 static int kho_test_restore_data(const void *fdt, int node)
 {
 	const unsigned int *nr_folios;
-	const phys_addr_t *folios_info;
+	const phys_addr_t *folios_info_phys;
+	phys_addr_t *folios_info;
 	const __wsum *old_csum;
 	__wsum csum = 0;
 	int len;

@@ -212,8 +221,12 @@ static int kho_test_restore_data(const void *fdt, int node)
 	if (!old_csum || len != sizeof(*old_csum))
 		return -EINVAL;
 
-	folios_info = fdt_getprop(fdt, node, "folios_info", &len);
-	if (!folios_info || len != sizeof(*folios_info) * *nr_folios)
+	folios_info_phys = fdt_getprop(fdt, node, "folios_info", &len);
+	if (!folios_info_phys || len != sizeof(*folios_info_phys))
+		return -EINVAL;
+
+	folios_info = kho_restore_vmalloc(*folios_info_phys);
+	if (!folios_info)
 		return -EINVAL;
 
 	for (int i = 0; i < *nr_folios; i++) {

@@ -233,6 +246,8 @@ static int kho_test_restore_data(const void *fdt, int node)
 		folio_put(folio);
 	}
 
+	vfree(folios_info);
+
 	if (csum != *old_csum)
 		return -EINVAL;
 

@@ -291,6 +306,7 @@ static void kho_test_cleanup(void)
 		folio_put(kho_test_state.folios[i]);
 
 	kvfree(kho_test_state.folios);
+	vfree(kho_test_state.folios_info);
 }
 
 static void __exit kho_test_exit(void)

2.50.1