Group pages on the direct map for permissioned vmallocs
Subsystems (6)
cover Group pages on the direct map for permissioned vmallocs
Hi,
This is a followup to the previous attempt to overhaul how vmalloc permissions are done:
https://lore.kernel.org/lkml/[email protected]/
In working on a next version it dawned on me that we can significantly reduce direct map breakage on x86 with a much less invasive change, so I thought maybe I would start there in the meantime.
In a test of booting fedora and running the BPF unit tests, this reduced 4k direct map pages by 98%.
It simply uses pages for x86 module_alloc() mappings from a cache created out of 2MB pages. This results in all the later breakage clustering in 2MB blocks on the direct map. The trade-off is colder pages are used for these allocations. All module_alloc() users (modules, ebpf jit, ftrace, kprobes) get this behavior.
Potentially this behavior should be enabled for eBPF byte code allocations as well in the case of !CONFIG_BPF_JIT_ALWAYS_ON.
The new APIs and invasive changes in the callers can happen after vmalloc huge pages bring more benefits. Although, I can post shootdown reduction changes with previous comments integrated if anyone disagrees.
Based on v5.11.
Thanks,
Rick
Rick Edgecombe (3):
list: Support getting most recent element in list_lru
vmalloc: Support grouped page allocations x86/module: Use VM_GROUP_PAGES flag
arch/x86/Kconfig | 1 + arch/x86/kernel/module.c | 2 +- include/linux/list_lru.h | 13 +++ include/linux/vmalloc.h | 1 + mm/Kconfig | 9 ++ mm/list_lru.c | 28 +++++ mm/vmalloc.c | 215 +++++++++++++++++++++++++++++++++++++-- 7 files changed, 257 insertions(+), 12 deletions(-)
Signature · 1 line
2.29.2
RFC 1/3 list: Support getting most recent element in list_lru
In future patches, some functionality will use list_lru that also needs to keep track of the most recently used element on a node. Since this information is already contained within list_lru, add a function to get it so that an additional list is not needed in the caller.
Do not support memcg aware list_lru's since it is not needed by the intended caller.
--- include/linux/list_lru.h | 13 +++++++++++++ mm/list_lru.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+)
include/linux/list_lru.h +13 -0
@@ -103,6 +103,19 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item);*/ bool list_lru_del (struct list_lru *lru,struct list_head *item); +/** +* list_lru_get_mru: gets and removes the tail from one of the node lists +* @list_lru: the lru pointer +* @nid: the node id +* +* This function removes the most recently added item from one of the node +* id specified. This function should not be used if the list_lru is memcg +* aware. +* +* Return value: The element removed +*/ +struct list_head *list_lru_get_mru (struct list_lru *lru,int nid); +/** * list_lru_count_one: return the number of objects currently held by @lru * @lru: the lru pointer.
mm/list_lru.c +28 -0
@@ -156,6 +156,34 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item) }EXPORT_SYMBOL_GPL (list_lru_del); +struct list_head *list_lru_get_mru (struct list_lru *lru,int nid) +{ +struct list_lru_node *nlru = &lru->node[nid]; +struct list_lru_one *l = &nlru->lru; +struct list_head *ret; + +/* This function does not attempt to search through the memcg lists */ +if (list_lru_memcg_aware (lru)) { +WARN_ONCE (1 ,"list_lru: %s not supported on memcg aware list_lrus" , __func__); +return NULL ; + } + +spin_lock (&nlru->lock); +if (list_empty (&l->list)) { + ret =NULL ; + }else { +/* Get tail */ + ret = l->list.prev; +list_del_init (ret); + + l->nr_items--; + nlru->nr_items--; + } +spin_unlock (&nlru->lock); + +return ret; +} +void list_lru_isolate (struct list_lru_one *list,struct list_head *item) {list_del_init (item);
Signature · 1 line
2.29.2
RFC 2/3 vmalloc: Support grouped page allocations
In reply to Rick EdgecombeFor x86, setting memory permissions on vmalloc allocations results in fracturing large pages on the direct map. Direct map fracturing can be reduced by locating pages that will have their permissions set close together.
Create a simple page cache in vmalloc that allocates pages from huge page size blocks. Define a new flag, VM_GROUP_PAGES, that vmalloc callers can use to ask vmalloc to try to use pages from this new cache. Don't guarantee that a page will come from a huge page grouping, instead fallback to non-grouped pages to fulfill the allocation if needed. Also, register a shrinker such that the system can ask for the pages back if needed.
Since this is only needed for cases where there are direct map permissions and there might be large pages on the direct map, compile it out for non-x86 and highmem using a new arch CONFIG.
Free pages in the cache are kept track of in per-node list inside a list_lru. NUMA_NO_NODE requests are serviced by checking each per-node list in a round robin fashion. If pages are requested for a certain node but the cache is empty for that node, a whole additional huge page size page is allocated.
When replenishing the cache, use a specific GFP flag that matches the intended user of this vm_flag (module_alloc()). In the case of the vm and GFP flags mismatching, fail the page allocation. In the case of a huge page size page not being available, fallback to the normal page allocator logic and use non-grouped pages.
--- arch/x86/Kconfig | 1 + include/linux/vmalloc.h | 1 + mm/Kconfig | 9 ++ mm/vmalloc.c | 215 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 215 insertions(+), 11 deletions(-)
arch/x86/Kconfig +1 -0
@@ -80,6 +80,7 @@ config X86 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY select ARCH_HAS_SET_DIRECT_MAP + select ARCH_HAS_HPAGE_DIRECT_MAP_PERMS if !HIGHMEM select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE
include/linux/vmalloc.h +1 -0
@@ -26,6 +26,7 @@ struct notifier_block; /* in notifier.h */#define VM_KASAN 0x00000080/* has allocated kasan shadow memory */ #define VM_FLUSH_RESET_PERMS 0x00000100/* reset direct map and flush TLB on unmap, can't be freed in atomic context */ #define VM_MAP_PUT_PAGES 0x00000200/* put pages and free array in vfree */ +#define VM_GROUP_PAGES 0x00000400/* try to group the pages on the direct map */ /* * VM_KASAN is used slighly differently depending on CONFIG_KASAN_VMALLOC.
mm/Kconfig +9 -0
@@ -869,6 +869,15 @@ config ARCH_HAS_PTE_SPECIAL config ARCH_HAS_HUGEPD bool +# +# Select if arch tries to preserve large pages on the direct map while also +# setting permissions on the direct map. When set, generic mm code will make +# an effort to centralize the breakage of the direct map that results from +# setting kernel memory permissions. +# +config ARCH_HAS_HPAGE_DIRECT_MAP_PERMS + bool + config MAPPING_DIRTY_HELPERS bool
mm/vmalloc.c +204 -11
254 lines not on this page · raw message
Signature · 1 line
2.29.2
Re: [RFC 2/3] vmalloc: Support grouped page allocations
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 2248
Quoted 11 lines · Rick Edgecombe
mm/vmalloc.c +11
-0+static void __dispose_pages (struct list_head *head) +{ +struct list_head *cur, *next; + +list_for_each_safe (cur, next, head) { +list_del (cur); + +/* The list head is stored at the start of the page */ +free_page ((unsigned long )cur); + } +}
This is interesting.
While the page is in the allocator, you're using the page contents themselves to store the list_head. It took me a minute to figure out what you were doing here because: "start of the page" is a bit ambiguous. It could mean:
* the first 16 bytes in 'struct page'
or
* the first 16 bytes in the page itself, aka *page_address(page)
The fact that this doesn't work on higmem systems makes this an OK thing to do, but it is a bit weird. It's also doubly susceptible to bugs where there's a page_to_virt() or virt_to_page() screwup.
I was *hoping* there was still sufficient space in 'struct page' for this second list_head in addition to page->lru. I think there *should* be. That would at least make this allocator a bit more "normal" in not caring about page contents while the page is free in the allocator. If you were able to do that you could do things like kmemcheck or page alloc debugging while the page is in the allocator.
Anyway, I think I'd prefer that you *try* to use 'struct page' alone. But, if that doesn't work out, please comment the snot out of this thing because it _is_ weird.
Re: [RFC 2/3] vmalloc: Support grouped page allocations
Quoted 35 lines · Dave Hansen
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 2248
Quoted 11 lines · Rick Edgecombe
mm/vmalloc.c +11
-0+static void __dispose_pages (struct list_head *head) +{ +struct list_head *cur, *next; + +list_for_each_safe (cur, next, head) { +list_del (cur); + +/* The list head is stored at the start of the page */ +free_page ((unsigned long )cur); + } +}This is interesting.
While the page is in the allocator, you're using the page contents themselves to store the
list_head. It took me a minute to figure out what you were doing here because: "start of the page" is a bit ambiguous. It could mean:* the first 16 bytes in 'struct page'
or
* the first 16 bytes in the page itself, aka*page_address(page)The fact that this doesn't work on higmem systems makes this an OK thing to do, but it is a bit weird. It's also doubly susceptible to bugs where there's a
page_to_virt()orvirt_to_page()screwup.I was *hoping* there was still sufficient space in 'struct page' for this second
list_headin addition topage->lru. I think there *should* be. That would at least make this allocator a bit more "normal" in not caring about page contents while the page is free in the allocator. If you were able to do that you could do things like kmemcheck or page alloc debugging while the page is in the allocator.
Quoted 3 lines
Anyway, I think I'd prefer that you *try* to use 'struct page' alone. But, if that doesn't work out, please comment the snot out of this thing because it _is_ weird.
Hi! Current closest-thing-we-have-to-an-expert-on-struct-page here!
I haven't read over these patches yet. If these pages are in use by vmalloc, they can't use mapping+index because get_user_pages() will call page_mapping() and the list_head will confuse it. I think it could use index+private for a list_head.
If the pages are in the buddy, I _think_ mapping+index are free. private is in use for buddy order. But I haven't read through the buddy code in a while.
Does it need to be a doubly linked list? Can it be an hlist?
Re: [RFC 2/3] vmalloc: Support grouped page allocations
Quoted 58 lines · Matthew Wilcox
Quoted 42 lines · Dave Hansen
Quoted 8 lines · Rick Edgecombe
mm/vmalloc.c +8
-0+static void __dispose_pages (struct list_head *head) +{ + struct list_head *cur, *next; + + list_for_each_safe(cur, next, head) { + list_del(cur); + + /* The list head is stored at the start of theQuoted 4 lines
page */
+ free_page((unsigned long)cur);
+ }
+}This is interesting.
While the page is in the allocator, you're using the page contents themselves to store the list_head. It took me a minute to figure out
what you were doing here because: "start of the page" is a bit ambiguous. It could mean:* the first 16 bytes in 'struct page'
or
* the first 16 bytes in the page itself, aka*page_address(page)The fact that this doesn't work on higmem systems makes this an OK thing
to do, but it is a bit weird. It's also doubly susceptible to bugs where there's apage_to_virt()orvirt_to_page()screwup.I was *hoping* there was still sufficient space in 'struct page' for
this secondlist_headin addition to page->lru. I think there *should*
be. That would at least make this allocator a bit more "normal" in not
caring about page contents while the page is free in the allocator. If
you were able to do that you could do things like kmemcheck or page alloc debugging while the page is in the allocator.Quoted 5 lines
Anyway, I think I'd prefer that you *try* to use 'struct page' alone.
But, if that doesn't work out, please comment the snot out of this thing
because it _is_ weird.Hi! Current closest-thing-we-have-to-an-expert-on-struct-page here!
I haven't read over these patches yet. If these pages are in use by vmalloc, they can't use mapping+index because
get_user_pages()will callpage_mapping()and thelist_headwill confuse it. I think it could use
index+private for alist_head.
Quoted 6 lines
If the pages are in the buddy, I _think_ mapping+index are free. private
is in use for buddy order. But I haven't read through the buddy code in a while.Does it need to be a doubly linked list? Can it be an hlist?
It does need to be a doubly linked list. I think they should never be mapped to userspace. As far as the page allocator is concerned these pages are not free. And they are not compound.
Originally I was just using the lru member. Would it be ok in that case?
Re: [RFC 2/3] vmalloc: Support grouped page allocations
In reply to Dave HansenQuoted 40 lines · Dave Hansen
Quoted 8 lines · Rick Edgecombe
mm/vmalloc.c +8
-0+static void __dispose_pages (struct list_head *head) +{ + struct list_head *cur, *next; + + list_for_each_safe(cur, next, head) { + list_del(cur); + + /* The list head is stored at the start of the pageQuoted 4 lines
*/
+ free_page((unsigned long)cur);
+ }
+}This is interesting.
While the page is in the allocator, you're using the page contents themselves to store the list_head. It took me a minute to figure out what you were doing here because: "start of the page" is a bit ambiguous. It could mean:
* the first 16 bytes in 'struct page'
or
* the first 16 bytes in the page itself, aka*page_address(page)The fact that this doesn't work on higmem systems makes this an OK thing
to do, but it is a bit weird. It's also doubly susceptible to bugs where there's apage_to_virt()orvirt_to_page()screwup.I was *hoping* there was still sufficient space in 'struct page' for this second
list_headin addition to page->lru. I think there *should*
be. That would at least make this allocator a bit more "normal" in not
caring about page contents while the page is free in the allocator. If
you were able to do that you could do things like kmemcheck or page alloc debugging while the page is in the allocator.
Quoted 4 lines
Anyway, I think I'd prefer that you *try* to use 'struct page' alone. But, if that doesn't work out, please comment the snot out of this thing
because it _is_ weird.
Yes sorry, that deserved more explanation. I tried putting it in struct page actually. The problem was list_lru automatically determines the node id from the list_head provided to it via page_to_nid(virt_to_page(head)). I guess it assumes the list_head is on the actual item. I started adding another list_lru function that let the node id be passed in separately, but I remembered this trick from the deferred free list in vmalloc.
If this ever expands to handle direct map unmapped pages (which would probably be the next step), the list_head will have to be moved out of the actual page anyway. But in the meantime it resulted in the smallest change.
I can try the other way if it's still too weird.
RFC 3/3 x86/module: Use VM_GROUP_PAGES flag
In reply to Rick EdgecombeCallers of module_alloc() will set permissions on the allocation. Use the VM_GROUP_PAGES to reduce direct map breakage.
--- arch/x86/kernel/module.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/module.c +1 -1
@@ -75,7 +75,7 @@ void *module_alloc(unsigned long size) p =__vmalloc_node_range (size,MODULE_ALIGN ,MODULES_VADDR +get_module_load_offset (),MODULES_END ,GFP_KERNEL , -PAGE_KERNEL ,0 ,NUMA_NO_NODE , +PAGE_KERNEL ,VM_GROUP_PAGES ,NUMA_NO_NODE ,__builtin_return_address (0 ));if (p && (kasan_module_alloc (p, size) <0 )) {vfree (p);
Signature · 1 line
2.29.2