Skip to contentSkip to conversation

mm/large system hash: use vmalloc for size > MAX_ORDER when !hashdist

PATCH In review Nicholas Piggin 2 participants 6 messages May 28, 2019 – Jun 3, 2019

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/4 mm/large system hash: use vmalloc for size > MAX_ORDER when !hashdist

The kernel currently clamps large system hashes to MAX_ORDER when hashdist is not set, which is rather arbitrary.

vmalloc space is limited on 32-bit machines, but this shouldn't result in much more used because of small physical memory limiting system hash sizes.

---
 mm/page_alloc.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
mm/page_alloc.c +3 -5
index d66bc8abe0af..dd419a074141 100644
@@ -8029,7 +8029,7 @@ void *__init alloc_large_system_hash(const char *tablename,
 			else
 				table = memblock_alloc_raw(size,
 							   SMP_CACHE_BYTES);
-		} else if (hashdist) {
+		} else if (get_order(size) >= MAX_ORDER || hashdist) {
 			table = __vmalloc(size, gfp_flags, PAGE_KERNEL);
 		} else {
 			/*
@@ -8037,10 +8037,8 @@ void *__init alloc_large_system_hash(const char *tablename,
 			 * some pages at the end of hash table which
 			 * alloc_pages_exact() automatically does
 			 */
-			if (get_order(size) < MAX_ORDER) {
-				table = alloc_pages_exact(size, gfp_flags);
-				kmemleak_alloc(table, size, 1, gfp_flags);
-			}
+			table = alloc_pages_exact(size, gfp_flags);
+			kmemleak_alloc(table, size, 1, gfp_flags);
 		}
 	} while (!table && size > PAGE_SIZE && --log2qty);
 
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/4 mm/large system hash: avoid vmap for non-NUMA machines when hashdist

hashdist currently always uses vmalloc when hashdist is true. When there is only 1 online node and size <= MAX_ORDER, vmalloc can be avoided.

---
 mm/page_alloc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/page_alloc.c +2 -1
index dd419a074141..15478dba1144 100644
@@ -8029,7 +8029,8 @@ void *__init alloc_large_system_hash(const char *tablename,
 			else
 				table = memblock_alloc_raw(size,
 							   SMP_CACHE_BYTES);
-		} else if (get_order(size) >= MAX_ORDER || hashdist) {
+		} else if (get_order(size) >= MAX_ORDER ||
+				(hashdist && num_online_nodes() > 1)) {
 			table = __vmalloc(size, gfp_flags, PAGE_KERNEL);
 		} else {
 			/*
Signature · 1 line
2.20.1
#2 lore raw

PATCH 3/4 mm: Move ioremap page table mapping function to mm/

In reply to Nicholas Piggin

ioremap_page_range is a useful generic function to create a kernel virtual mapping, move it to mm/vmalloc.c and rename it vmap_range to be used by a subsequent change.

For clarity with this move, also:
- Rename vunmap_page_range (vmap_range's inverse) to vunmap_range.
- Rename vmap_page_range (which takes a page array) to vmap_pages.

---
 include/linux/vmalloc.h |   3 +
 lib/ioremap.c           | 173 +++---------------------------
 mm/vmalloc.c            | 227 ++++++++++++++++++++++++++++++++++++----
 3 files changed, 228 insertions(+), 175 deletions(-)
include/linux/vmalloc.h +3 -0
index 51e131245379..812bea5866d6 100644
@@ -147,6 +147,9 @@ extern struct vm_struct *find_vm_area(const void *addr);
 extern int map_vm_area(struct vm_struct *area, pgprot_t prot,
 			struct page **pages);
 #ifdef CONFIG_MMU
+extern int vmap_range(unsigned long addr,
+		       unsigned long end, phys_addr_t phys_addr, pgprot_t prot,
+		       unsigned int max_page_shift);
 extern int map_kernel_range_noflush(unsigned long start, unsigned long size,
 				    pgprot_t prot, struct page **pages);
 extern void unmap_kernel_range_noflush(unsigned long addr, unsigned long size);
lib/ioremap.c +16 -157

181 lines not on this page · raw message

mm/vmalloc.c +209 -18

337 lines not on this page · raw message

Signature · 1 line
2.20.1
#3 lore raw

PATCH 4/4 mm/vmalloc: Hugepage vmalloc mappings

In reply to Nicholas Piggin

For platforms that define HAVE_ARCH_HUGE_VMAP, have vmalloc try to allocate and map PMD-level huge pages when size is appropriate, and fallback if unsuccessful.

Using 2MB page mappings in an 8MB dentry cache hash brings dTLB misses for linux kernel tree git diff from 45,000 to 8,000 on a Kaby Lake CPU wth mitigations=off (performance is in the noise, under 1% difference, page tables are likely to be well cached for this workload).

---
 include/linux/vmalloc.h |   1 +
 mm/vmalloc.c            | 132 +++++++++++++++++++++++++++++++---------
 2 files changed, 105 insertions(+), 28 deletions(-)
include/linux/vmalloc.h +1 -0
index 812bea5866d6..4c92dc608928 100644
@@ -42,6 +42,7 @@ struct vm_struct {
 	unsigned long		size;
 	unsigned long		flags;
 	struct page		**pages;
+	unsigned int		page_shift;
 	unsigned int		nr_pages;
 	phys_addr_t		phys_addr;
 	const void		*caller;
mm/vmalloc.c +104 -28

233 lines not on this page · raw message

Signature · 1 line
2.20.1
#4 lore raw
In reply to Nicholas Piggin
Quoted 3 lines · Nicholas Piggin

The kernel currently clamps large system hashes to MAX_ORDER when hashdist is not set, which is rather arbitrary.

I think the *really* arbitrary part here is "hashdist".

If you enable NUMA support, hashdist is just set to 1 by default on 64-bit, whether the machine actually has any numa characteristics or not. So you take that vmalloc() TLB overhead whether you need it or not.

So I think your series looks sane, and should help the vmalloc case for big hash allocations, but I also think that this whole alloc_large_system_hash() function should be smarter in general.

Yes, it's called "alloc_large_system_hash()", but it's used on small and perfectly normal-sized systems too, and often for not all that big hashes.

Yes, we tend to try to make some of those hashes large (dentry one in particular), but we also use this for small stuff.

For example, on my machine I have several network hashes that have order 6-8 sizes, none of which really make any sense to use vmalloc space for (and which are smaller than a large page, so your patch series wouldn't help).

So on the whole I have no issues with this series, but I do think we should maybe fix that crazy "if (hashdist)" case. Hmm?

Linus

#5 lore raw

Linus Torvalds's on June 1, 2019 4:30 am:

Quoted 11 lines · Nicholas Piggin
Quoted 3 lines

The kernel currently clamps large system hashes to MAX_ORDER when hashdist is not set, which is rather arbitrary.

I think the *really* arbitrary part here is "hashdist".

If you enable NUMA support, hashdist is just set to 1 by default on 64-bit, whether the machine actually has any numa characteristics or not. So you take that vmalloc() TLB overhead whether you need it or not.

Yeah, that's strange it seems to just be an oversight nobody ever picked up. Patch 2/4 actually fixed that exactly the way you said.

Quoted 12 lines

So I think your series looks sane, and should help the vmalloc case for big hash allocations, but I also think that this whole alloc_large_system_hash() function should be smarter in general.

Yes, it's called "alloc_large_system_hash()", but it's used on small and perfectly normal-sized systems too, and often for not all that big hashes.

Yes, we tend to try to make some of those hashes large (dentry one in particular), but we also use this for small stuff.

Quoted 7 lines

For example, on my machine I have several network hashes that have order 6-8 sizes, none of which really make any sense to use vmalloc space for (and which are smaller than a large page, so your patch series wouldn't help).

So on the whole I have no issues with this series, but I do think we should maybe fix that crazy "if (hashdist)" case. Hmm?

Yes agreed. Even after this series with 2MB mappings it's actually a bit sad that we can't use the linear map for the non-NUMA case. My laptop has a 32MB dentry cache and 16MB inode cache so doing a bunch of name lookups is quite a waste of TLB entries (although at least with 2MB pages it doesn't blow the TLB completely).

We might be able to go a step further and use memblock allocator for those as well, or reserve some boot CMA for that common case ot just use the linear map for these hashes. I'll look into that.

Thanks,
Nick