mm/large system hash: use vmalloc for size > MAX_ORDER when !hashdist
Subsystems (5)
PATCH 1/4 mm/large system hash: use vmalloc for size > MAX_ORDER when !hashdist
The kernel currently clamps large system hashes to MAX_ORDER when hashdist is not set, which is rather arbitrary.
vmalloc space is limited on 32-bit machines, but this shouldn't result in much more used because of small physical memory limiting system hash sizes.
--- mm/page_alloc.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
mm/page_alloc.c +3 -5
@@ -8029,7 +8029,7 @@ void *__init alloc_large_system_hash(const char *tablename,else table =memblock_alloc_raw (size,SMP_CACHE_BYTES ); - }else if (hashdist) { + }else if (get_order (size) >=MAX_ORDER || hashdist) { table =__vmalloc (size, gfp_flags,PAGE_KERNEL ); }else {/*
@@ -8037,10 +8037,8 @@ void *__init alloc_large_system_hash(const char *tablename,* some pages at the end of hash table which * alloc_pages_exact() automatically does */ -if (get_order (size) <MAX_ORDER ) { - table =alloc_pages_exact (size, gfp_flags); -kmemleak_alloc (table, size,1 , gfp_flags); - } + table =alloc_pages_exact (size, gfp_flags); +kmemleak_alloc (table, size,1 , gfp_flags); } }while (!table && size >PAGE_SIZE && --log2qty);
Signature · 1 line
2.20.1
PATCH 2/4 mm/large system hash: avoid vmap for non-NUMA machines when hashdist
hashdist currently always uses vmalloc when hashdist is true. When there is only 1 online node and size <= MAX_ORDER, vmalloc can be avoided.
--- mm/page_alloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/page_alloc.c +2 -1
@@ -8029,7 +8029,8 @@ void *__init alloc_large_system_hash(const char *tablename,else table =memblock_alloc_raw (size,SMP_CACHE_BYTES ); - }else if (get_order (size) >=MAX_ORDER || hashdist) { + }else if (get_order (size) >=MAX_ORDER || + (hashdist &&num_online_nodes () >1 )) { table =__vmalloc (size, gfp_flags,PAGE_KERNEL ); }else {/*
Signature · 1 line
2.20.1
PATCH 3/4 mm: Move ioremap page table mapping function to mm/
In reply to Nicholas Pigginioremap_page_range is a useful generic function to create a kernel virtual mapping, move it to mm/vmalloc.c and rename it vmap_range to be used by a subsequent change.
For clarity with this move, also:
- Rename vunmap_page_range (vmap_range's inverse) to vunmap_range.
- Rename vmap_page_range (which takes a page array) to vmap_pages.
--- include/linux/vmalloc.h | 3 + lib/ioremap.c | 173 +++--------------------------- mm/vmalloc.c | 227 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 228 insertions(+), 175 deletions(-)
include/linux/vmalloc.h +3 -0
@@ -147,6 +147,9 @@ extern struct vm_struct *find_vm_area(const void *addr);extern int map_vm_area (struct vm_struct *area,pgprot_t prot,struct page **pages);#ifdef CONFIG_MMU +extern int vmap_range (unsigned long addr, +unsigned long end,phys_addr_t phys_addr,pgprot_t prot, +unsigned int max_page_shift);extern int map_kernel_range_noflush (unsigned long start,unsigned long size,pgprot_t prot,struct page **pages);extern void unmap_kernel_range_noflush (unsigned long addr,unsigned long size);
lib/ioremap.c +16 -157
181 lines not on this page · raw message
mm/vmalloc.c +209 -18
337 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 4/4 mm/vmalloc: Hugepage vmalloc mappings
In reply to Nicholas PigginFor platforms that define HAVE_ARCH_HUGE_VMAP, have vmalloc try to allocate and map PMD-level huge pages when size is appropriate, and fallback if unsuccessful.
Using 2MB page mappings in an 8MB dentry cache hash brings dTLB misses for linux kernel tree git diff from 45,000 to 8,000 on a Kaby Lake CPU wth mitigations=off (performance is in the noise, under 1% difference, page tables are likely to be well cached for this workload).
--- include/linux/vmalloc.h | 1 + mm/vmalloc.c | 132 +++++++++++++++++++++++++++++++--------- 2 files changed, 105 insertions(+), 28 deletions(-)
include/linux/vmalloc.h +1 -0
@@ -42,6 +42,7 @@ struct vm_struct {unsigned long size;unsigned long flags;struct page **pages; +unsigned int page_shift;unsigned int nr_pages;phys_addr_t phys_addr;const void *caller;
mm/vmalloc.c +104 -28
233 lines not on this page · raw message
Signature · 1 line
2.20.1
Quoted 3 lines · Nicholas Piggin
The kernel currently clamps large system hashes to
MAX_ORDERwhen hashdist is not set, which is rather arbitrary.
I think the *really* arbitrary part here is "hashdist".
If you enable NUMA support, hashdist is just set to 1 by default on 64-bit, whether the machine actually has any numa characteristics or not. So you take that vmalloc() TLB overhead whether you need it or not.
So I think your series looks sane, and should help the vmalloc case for big hash allocations, but I also think that this whole alloc_large_system_hash() function should be smarter in general.
Yes, it's called "alloc_large_system_hash()", but it's used on small and perfectly normal-sized systems too, and often for not all that big hashes.
Yes, we tend to try to make some of those hashes large (dentry one in particular), but we also use this for small stuff.
For example, on my machine I have several network hashes that have order 6-8 sizes, none of which really make any sense to use vmalloc space for (and which are smaller than a large page, so your patch series wouldn't help).
So on the whole I have no issues with this series, but I do think we should maybe fix that crazy "if (hashdist)" case. Hmm?
Linus
Linus Torvalds's on June 1, 2019 4:30 am:
Quoted 11 lines · Nicholas Piggin
Quoted 3 lines
The kernel currently clamps large system hashes to
MAX_ORDERwhen hashdist is not set, which is rather arbitrary.I think the *really* arbitrary part here is "hashdist".
If you enable NUMA support, hashdist is just set to 1 by default on 64-bit, whether the machine actually has any numa characteristics or not. So you take that
vmalloc()TLB overhead whether you need it or not.
Yeah, that's strange it seems to just be an oversight nobody ever picked up. Patch 2/4 actually fixed that exactly the way you said.
Quoted 12 lines
So I think your series looks sane, and should help the vmalloc case for big hash allocations, but I also think that this whole
alloc_large_system_hash()function should be smarter in general.Yes, it's called "
alloc_large_system_hash()", but it's used on small and perfectly normal-sized systems too, and often for not all that big hashes.Yes, we tend to try to make some of those hashes large (dentry one in particular), but we also use this for small stuff.
Quoted 7 lines
For example, on my machine I have several network hashes that have order 6-8 sizes, none of which really make any sense to use vmalloc space for (and which are smaller than a large page, so your patch series wouldn't help).
So on the whole I have no issues with this series, but I do think we should maybe fix that crazy "if (hashdist)" case. Hmm?
Yes agreed. Even after this series with 2MB mappings it's actually a bit sad that we can't use the linear map for the non-NUMA case. My laptop has a 32MB dentry cache and 16MB inode cache so doing a bunch of name lookups is quite a waste of TLB entries (although at least with 2MB pages it doesn't blow the TLB completely).
We might be able to go a step further and use memblock allocator for those as well, or reserve some boot CMA for that common case ot just use the linear map for these hashes. I'll look into that.
Thanks,
Nick