decrease unnecessary gap due to pmem kmem alignment
Subsystems (7)
cover decrease unnecessary gap due to pmem kmem alignment
When enabling dax pmem as RAM device on arm64, I noticed that kmem_start addr in dev_dax_kmem_probe() should be aligned w/ SECTION_SIZE_BITS(30),i.e. 1G memblock size. Even Dan Williams' sub-section patch series [1] had been upstream merged, it was not helpful due to hard limitation of kmem_start:
$ndctl create-namespace -e namespace0.0 --mode=devdax --map=dev -s 2g -f -a 2M $echo dax0.0 > /sys/bus/dax/drivers/device_dax/unbind
$echo dax0.0 > /sys/bus/dax/drivers/kmem/new_id
$cat /proc/iomem
...
23c000000-23fffffff : System RAM
23dd40000-23fecffff : reserved
23fed0000-23fffffff : reserved
240000000-33fdfffff : Persistent Memory
240000000-2403fffff : namespace0.0
280000000-2bfffffff : dax0.0 <- aligned with 1G boundary
280000000-2bfffffff : System RAM
Hence there is a big gap between 0x2403fffff and 0x280000000 due to the 1G alignment.
Without this series, if qemu creates a 4G bytes nvdimm device, we can only use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000, 2ffffffff] due to the hard limitation. It wastes too much memory space.
Decreasing the SECTION_SIZE_BITS on arm64 might be an alternative, but there are too many concerns from other constraints, e.g. PAGE_SIZE, hugetlb, SPARSEMEM_VMEMMAP, page bits in struct page ...
Beside decreasing the SECTION_SIZE_BITS, we can also relax the kmem alignment with memory_block_size_bytes().
Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device. dax pmem can be used as ram with smaller gap. Also the kmem hotplug add/remove are both tested on arm64/x86 guest.
This patch series (mainly patch6/6) is based on the fixing patch, ~v5.8-rc5 [2].
[1] https://lkml.org/lkml/2019/6/19/67
[2] https://lkml.org/lkml/2020/7/8/1546
Jia He (6):
mm/memory_hotplug: remove redundant memory block size alignment check
resource: export find_next_iomem_res() helper mm/memory_hotplug: allow pmem kmem not to align with memory_block_size mm/page_alloc: adjust the start,end in dax pmem kmem case
device-dax: relax the memblock size alignment for kmem_start
arm64: fall back to vmemmap_populate_basepages if not aligned with
PMD_SIZE
arch/arm64/mm/mmu.c | 4 ++++ drivers/base/memory.c | 24 ++++++++++++++++-------- drivers/dax/kmem.c | 22 +++++++++++++--------- include/linux/ioport.h | 3 +++ kernel/resource.c | 3 ++- mm/memory_hotplug.c | 39 ++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 14 ++++++++++++++ 7 files changed, 90 insertions(+), 19 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH 1/6 mm/memory_hotplug: remove redundant memory block size alignment check
The alignment check has been done by check_hotplug_memory_range(). Hence the redundant one in create_memory_block_devices() can be removed.
The similar redundant check is removed in remove_memory_block_devices().
--- drivers/base/memory.c | 8 -------- 1 file changed, 8 deletions(-)
drivers/base/memory.c +0 -8
@@ -642,10 +642,6 @@ int create_memory_block_devices(unsigned long start, unsigned long size)unsigned long block_id;int ret =0 ; -if (WARN_ON_ONCE (!IS_ALIGNED (start,memory_block_size_bytes ()) || - !IS_ALIGNED (size,memory_block_size_bytes ()))) -return -EINVAL ; -for (block_id = start_block_id; block_id != end_block_id; block_id++) { ret =init_memory_block (&mem, block_id,MEM_OFFLINE );if (ret)
@@ -678,10 +674,6 @@ void remove_memory_block_devices(unsigned long start, unsigned long size)struct memory_block *mem;unsigned long block_id; -if (WARN_ON_ONCE (!IS_ALIGNED (start,memory_block_size_bytes ()) || - !IS_ALIGNED (size,memory_block_size_bytes ()))) -return ; -for (block_id = start_block_id; block_id != end_block_id; block_id++) { mem =find_memory_block_by_id (block_id);if (WARN_ON_ONCE (!mem))
Signature · 1 line
2.17.1
RFC PATCH 2/6 resource: export find_next_iomem_res() helper
In reply to Jia HeThe helper is to find the lowest iomem resource that covers part of [@start..@end]
It is useful when relaxing the alignment check for dax pmem kmem.
--- include/linux/ioport.h | 3 +++ kernel/resource.c | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-)
include/linux/ioport.h +3 -0
@@ -247,6 +247,9 @@ extern struct resource * __request_region(struct resource *,extern void __release_region (struct resource *,resource_size_t ,resource_size_t ); +extern int find_next_iomem_res (resource_size_t start,resource_size_t end, +unsigned long flags,unsigned long desc, +bool first_lvl,struct resource *res);#ifdef CONFIG_MEMORY_HOTREMOVE extern int release_mem_region_adjustable (struct resource *,resource_size_t ,resource_size_t );
kernel/resource.c +2 -1
@@ -338,7 +338,7 @@ EXPORT_SYMBOL(release_resource);* @first_lvl: walk only the first level children, if set * @res: return ptr, if resource found */ -static int find_next_iomem_res (resource_size_t start,resource_size_t end, +int find_next_iomem_res (resource_size_t start,resource_size_t end,unsigned long flags,unsigned long desc,bool first_lvl,struct resource *res) {
@@ -391,6 +391,7 @@ static int find_next_iomem_res(resource_size_t start, resource_size_t end,read_unlock (&resource_lock);return p ?0 : -ENODEV ; } +EXPORT_SYMBOL (find_next_iomem_res); static int __walk_iomem_res_desc(resource_size_t start, resource_size_t end, unsigned long flags, unsigned long desc,
Signature · 1 line
2.17.1
RFC PATCH 3/6 mm/memory_hotplug: allow pmem kmem not to align with memory_block_size
In reply to Jia HeWhen dax pmem is probed as RAM device on arm64, previously, kmem_start in dev_dax_kmem_probe() should be aligned with 1G memblock size on arm64 due to SECTION_SIZE_BITS(30).
There will be some meta data at the beginning/end of the iomem space, e.g. namespace info and nvdimm label:
240000000-33fdfffff : Persistent Memory
240000000-2403fffff : namespace0.0
280000000-2bfffffff : dax0.0
280000000-2bfffffff : System RAM
Hence it makes the whole kmem space not aligned with memory_block_size for both start addr and end addr. Hence there is a big gap when kmem is added into memory block which causes big memory space wasting.
This changes it by relaxing the alignment check for dax pmem kmem in the path of online/offline memory blocks.
--- drivers/base/memory.c | 16 ++++++++++++++++ mm/memory_hotplug.c | 39 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 1 deletion(-)
drivers/base/memory.c +16 -0
@@ -334,6 +334,22 @@ static ssize_t valid_zones_show(struct device *dev,* online nodes otherwise the page_zone is not reliable */ if (mem->state ==MEM_ONLINE ) { +#ifdef CONFIG_ZONE_DEVICE +struct resource res; +int ret; + +/* adjust start_pfn for dax pmem kmem */ + ret =find_next_iomem_res (start_pfn <<PAGE_SHIFT , + ((start_pfn + nr_pages) <<PAGE_SHIFT ) -1 , +IORESOURCE_SYSTEM_RAM |IORESOURCE_BUSY , +IORES_DESC_PERSISTENT_MEMORY , + false, &res); +if (!ret &&PFN_UP (res.start) > start_pfn) { + nr_pages -=PFN_UP (res.start) - start_pfn; + start_pfn =PFN_UP (res.start); + } +#endif +/* * The block contains more than one zone can not be offlined. * This can happen e.g. for ZONE_DMA and ZONE_DMA32
mm/memory_hotplug.c +38 -1
@@ -999,6 +999,20 @@ int try_online_node(int nid)static int check_hotplug_memory_range (u64 start,u64 size) { +#ifdef CONFIG_ZONE_DEVICE +struct resource res; +int ret; + +/* Allow pmem kmem not to align with block size */ + ret =find_next_iomem_res (start, start + size -1 , +IORESOURCE_SYSTEM_RAM |IORESOURCE_BUSY , +IORES_DESC_PERSISTENT_MEMORY , + false, &res); +if (!ret) { +return 0 ; + } +#endif +/* memory range must be block size aligned */ if (!size || !IS_ALIGNED (start,memory_block_size_bytes ()) || !IS_ALIGNED (size,memory_block_size_bytes ())) {
@@ -1481,19 +1495,42 @@ static int __ref __offline_pages(unsigned long start_pfn,mem_hotplug_begin ();/* -* Don't allow to offline memory blocks that contain holes. +* Don't allow to offline memory blocks that contain holes except +* for pmem. * Consequently, memory blocks with holes can never get onlined * via the hotplug path - online_pages() - as hotplugged memory has * no holes. This way, we e.g., don't have to worry about marking * memory holes PG_reserved, don't need pfn_valid() checks, and can * avoid using walk_system_ram_range() later. +* When dax pmem is used as RAM (kmem), holes at the beginning is +* allowed. */ walk_system_ram_range (start_pfn, end_pfn - start_pfn, &nr_pages, count_system_ram_pages_cb);if (nr_pages != end_pfn - start_pfn) { +#ifdef CONFIG_ZONE_DEVICE +struct resource res; + +/* Allow pmem kmem not to align with block size */ + ret =find_next_iomem_res (start_pfn <<PAGE_SHIFT , + (end_pfn <<PAGE_SHIFT ) -1 , +IORESOURCE_SYSTEM_RAM |IORESOURCE_BUSY , +IORES_DESC_PERSISTENT_MEMORY , + false, &res); +if (ret) { + ret = -EINVAL ; + reason ="memory holes" ; + goto failed_removal; + } + +/* adjust start_pfn for dax pmem kmem */ + start_pfn =PFN_UP (res.start); + end_pfn =PFN_DOWN (res.end +1 ); +#else ret = -EINVAL ; reason ="memory holes" ; goto failed_removal; +#endif }/* This makes hotplug much easier...and readable.
Signature · 1 line
2.17.1
RFC PATCH 4/6 mm/page_alloc: adjust the start,end in dax pmem kmem case
In reply to Jia HeThere are 3 cases when doing online pages:
- normal RAM, should be aligned with memory block size
- persistent memory with ZONE_DEVICE
- persistent memory used as normal RAM (kmem) with ZONE_NORMAL, this patch tries to adjust the start_pfn/end_pfn after finding the corresponding resource range.
Without this patch, the check of __init_single_page when doing online memory will be failed because those pages haven't been mapped in mmu(not present from mmu's point of view).
--- mm/page_alloc.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
mm/page_alloc.c +14 -0
@@ -5971,6 +5971,20 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,if (start_pfn == altmap->base_pfn) start_pfn += altmap->reserve; end_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + }else { +struct resource res; +int ret; + +/* adjust the start,end in dax pmem kmem case */ + ret =find_next_iomem_res (start_pfn <<PAGE_SHIFT , + (end_pfn <<PAGE_SHIFT ) -1 , +IORESOURCE_SYSTEM_RAM |IORESOURCE_BUSY , +IORES_DESC_PERSISTENT_MEMORY , + false, &res); +if (!ret) { + start_pfn =PFN_UP (res.start); + end_pfn =PFN_DOWN (res.end +1 ); + } }#endif
Signature · 1 line
2.17.1
RFC PATCH 5/6 device-dax: relax the memblock size alignment for kmem_start
In reply to Jia HePreviously, kmem_start in dev_dax_kmem_probe should be aligned with SECTION_SIZE_BITS(30), i.e. 1G memblock size on arm64. Even with Dan Williams' sub-section patch series, it was not helpful when adding the dax pmem kmem to memblock:
$ndctl create-namespace -e namespace0.0 --mode=devdax --map=dev -s 2g -f -a 2M $echo dax0.0 > /sys/bus/dax/drivers/device_dax/unbind
$echo dax0.0 > /sys/bus/dax/drivers/kmem/new_id
$cat /proc/iomem
...
23c000000-23fffffff : System RAM
23dd40000-23fecffff : reserved
23fed0000-23fffffff : reserved
240000000-33fdfffff : Persistent Memory
240000000-2403fffff : namespace0.0
280000000-2bfffffff : dax0.0 <- boundary are aligned with 1G
280000000-2bfffffff : System RAM (kmem) $ lsmem RANGE SIZE STATE REMOVABLE BLOCK 0x0000000040000000-0x000000023fffffff 8G online yes 1-8 0x0000000280000000-0x00000002bfffffff 1G online yes 10
Memory block size: 1G Total online memory: 9G Total offline memory: 0B
...
Hence there is a big gap between 0x2403fffff and 0x280000000 due to the 1G alignment on arm64. More than that, only 1G memory is returned while 2G is requested.
On x86, the gap is relatively small due to SECTION_SIZE_BITS(27).
Besides descreasing SECTION_SIZE_BITS on arm64, we can relax the alignment when adding the kmem.
After this patch:
240000000-33fdfffff : Persistent Memory
240000000-2421fffff : namespace0.0
242400000-2bfffffff : dax0.0
242400000-2bfffffff : System RAM (kmem) $ lsmem RANGE SIZE STATE REMOVABLE BLOCK 0x0000000040000000-0x00000002bfffffff 10G online yes 1-10
Memory block size: 1G Total online memory: 10G Total offline memory: 0B
Notes, block 9-10 are the newly hotplug added.
This patches remove the tight alignment constraint of memory_block_size_bytes(), but still keep the constraint from online_pages_range().
--- drivers/dax/kmem.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
drivers/dax/kmem.c +13 -9
@@ -30,9 +30,20 @@ int dev_dax_kmem_probe(struct device *dev)const char *new_res_name;int numa_node;int rc; +int order; -/* Hotplug starting at the beginning of the next block: */ - kmem_start =ALIGN (res->start,memory_block_size_bytes ()); +/* kmem_start needn't be aligned with memory_block_size_bytes(). +* But given the constraint in online_pages_range(), adjust the +* alignment of kmem_start and kmem_size +*/ + kmem_size =resource_size (res); + order =min_t (int,MAX_ORDER -1 ,get_order (kmem_size)); + kmem_start =ALIGN (res->start,1ul << (order +PAGE_SHIFT )); +/* Adjust the size down to compensate for moving up kmem_start: */ + kmem_size -= kmem_start - res->start; +/* Align the size down to cover only complete blocks: */ + kmem_size &= ~((1ul << (order +PAGE_SHIFT )) -1 ); + kmem_end = kmem_start + kmem_size;/* * Ensure good NUMA information for the persistent memory.
@@ -48,13 +59,6 @@ int dev_dax_kmem_probe(struct device *dev) numa_node, res); } - kmem_size =resource_size (res); -/* Adjust the size down to compensate for moving up kmem_start: */ - kmem_size -= kmem_start - res->start; -/* Align the size down to cover only complete blocks: */ - kmem_size &= ~(memory_block_size_bytes () -1 ); - kmem_end = kmem_start + kmem_size; - new_res_name =kstrdup (dev_name (dev),GFP_KERNEL );if (!new_res_name)return -ENOMEM ;
Signature · 1 line
2.17.1
RFC PATCH 6/6 arm64: fall back to vmemmap_populate_basepages if not aligned with PMD_SIZE
In reply to Jia HeIn dax pmem kmem (dax pmem used as RAM device) case, the start address might not be aligned with PMD_SIZE
e.g.
240000000-33fdfffff : Persistent Memory
240000000-2421fffff : namespace0.0
242400000-2bfffffff : dax0.0
242400000-2bfffffff : System RAM (kmem)
pfn_to_page(0x242400000) is fffffe0007e90000.
Without this patch, vmemmap_populate(fffffe0007e90000, ...) will incorrectly create a pmd mapping [fffffe0007e00000, fffffe0008000000] which contains fffffe0007e90000.
This adds the check and then falls back to vmemmap_populate_basepages()
--- arch/arm64/mm/mmu.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/mm/mmu.c +4 -0
@@ -1102,6 +1102,10 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, do { next = pmd_addr_end(addr, end); + if (next - addr < PMD_SIZE) { + vmemmap_populate_basepages(start, next, node, altmap); + continue; + } pgdp = vmemmap_pgd_populate(addr, node); if (!pgdp)
Signature · 1 line
2.17.1
Quoted 34 lines
Am 29.07.2020 um 05:35 schrieb Jia He <[email protected]>:
When enabling dax pmem as RAM device on arm64, I noticed that
kmem_startaddr indev_dax_kmem_probe()should be aligned w/ SECTION_SIZE_BITS(30),i.e. 1G memblock size. Even Dan Williams' sub-section patch series [1] had been upstream merged, it was not helpful due to hard limitation ofkmem_start:
$ndctl create-namespace -e namespace0.0 --mode=devdax --map=dev -s 2g -f -a 2M $echo dax0.0 >/sys/bus/dax/drivers/device_dax/unbind
$echo dax0.0 >/sys/bus/dax/drivers/kmem/new_id
$cat/proc/iomem
...
23c000000-23fffffff : System RAM
23dd40000-23fecffff : reserved
23fed0000-23fffffff : reserved
240000000-33fdfffff : Persistent Memory
240000000-2403fffff : namespace0.0
280000000-2bfffffff : dax0.0 <- aligned with 1G boundary
280000000-2bfffffff : System RAM
Hence there is a big gap between0x2403fffffand0x280000000due to the 1G alignment.Without this series, if qemu creates a 4G bytes nvdimm device, we can only use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000,2ffffffff] due to the hard limitation. It wastes too much memory space.Decreasing the
SECTION_SIZE_BITSon arm64 might be an alternative, but there are too many concerns from other constraints, e.g.PAGE_SIZE, hugetlb,SPARSEMEM_VMEMMAP, page bits in struct page ...
Quoted 7 lines
Beside decreasing the
SECTION_SIZE_BITS, we can also relax the kmem alignment withmemory_block_size_bytes().Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device. dax pmem can be used as ram with smaller gap. Also the kmem hotplug add/remove are both tested on arm64/x86 guest.
Hi,
I am not convinced this use case is worth such hacks (that’s what it is) for now. On real machines pmem is big - your example (losing 50% is extreme).
I would much rather want to see the section size on arm64 reduced. I remember there were patches and that at least with a base page size of 4k it can be reduced drastically (64k base pages are more problematic due to the ridiculous THP size of 512M). But could be a section size of 512 is possible on all configs right now.
In the long term we might want to rework the memory block device model (eventually supporting old/new as discussed with Michal some time ago using a kernel parameter), dropping the fixed sizes
- allowing sizes / addresses aligned with subsection size
- drastically reducing the number of devices for boot memory to only a hand full (e.g., one per resource / DIMM we can actually unplug again.
Long story short, I don’t like this hack.
Quoted 25 lines
This patch series (mainly patch6/6) is based on the fixing patch, ~v5.8-rc5 [2].
[1] https://lkml.org/lkml/2019/6/19/67
[2] https://lkml.org/lkml/2020/7/8/1546
Jia He (6):
mm/memory_hotplug: remove redundant memory block size alignment check
resource: exportfind_next_iomem_res()helpermm/memory_hotplug: allow pmem kmem not to align withmemory_block_sizemm/page_alloc: adjust the start,end in dax pmem kmem case
device-dax: relax the memblock size alignment forkmem_start
arm64: fall back tovmemmap_populate_basepagesif not aligned with
PMD_SIZEarch/arm64/mm/mmu.c | 4 ++++ drivers/base/memory.c | 24 ++++++++++++++++-------- drivers/dax/kmem.c | 22 +++++++++++++--------- include/linux/ioport.h | 3 +++ kernel/resource.c | 3 ++- mm/memory_hotplug.c | 39 ++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 14 ++++++++++++++ 7 files changed, 90 insertions(+), 19 deletions(-)Signature · 1 line
2.17.1
Hi David
Quoted 80 lines
-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Wednesday, July 29, 2020 2:37 PM
To: Justin He <[email protected]>
Cc: Dan Williams <[email protected]>; Vishal Verma <[email protected]>; Mike Rapoport <[email protected]>; David Hildenbrand <[email protected]>; Catalin Marinas <[email protected]>; Will Deacon <[email protected]>; Greg Kroah-Hartman <[email protected]>; Rafael J. Wysocki <[email protected]>; Dave Jiang <[email protected]>; Andrew Morton <[email protected]>; Steve Capper <[email protected]>; Mark Rutland <[email protected]>; Logan Gunthorpe <[email protected]>; Anshuman Khandual <[email protected]>; Hsin-Yi Wang <[email protected]>; Jason Gunthorpe <[email protected]>; Dave Hansen <[email protected]>; Kees Cook <[email protected]>; [email protected]; linux- [email protected]; [email protected]; [email protected]; Wei Yang <[email protected]>; Pankaj Gupta <[email protected]>; Ira Weiny <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [RFC PATCH 0/6] decrease unnecessary gap due to pmem kmem alignmentQuoted 4 lines
Am 29.07.2020 um 05:35 schrieb Jia He <[email protected]>:
When enabling dax pmem as RAM device on arm64, I noticed that
kmem_startaddr indev_dax_kmem_probe()should be aligned w/SECTION_SIZE_BITS(30),i.e.
Quoted 1 line
1G memblock size. Even Dan Williams' sub-section patch series [1] had
been
Quoted 2 lines
upstream merged, it was not helpful due to hard limitation of
kmem_start:
$ndctl create-namespace -e namespace0.0 --mode=devdax --map=dev -s 2g -f-a 2M
Quoted 4 lines
$echo dax0.0 >
/sys/bus/dax/drivers/device_dax/unbind$echo dax0.0 >/sys/bus/dax/drivers/kmem/new_id$cat/proc/iomem
...Quoted 8 lines
23c000000-23fffffff : System RAM
23dd40000-23fecffff : reserved
23fed0000-23fffffff : reserved
240000000-33fdfffff : Persistent Memory
240000000-2403fffff : namespace0.0
280000000-2bfffffff : dax0.0 <- aligned with 1G boundary
280000000-2bfffffff : System RAM
Hence there is a big gap between0x2403fffffand0x280000000due to the1G
Quoted 3 lines
alignment.
Without this series, if qemu creates a 4G bytes nvdimm device, we can
only
Quoted 4 lines
use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000,2ffffffff] due to thehard
Quoted 3 lines
limitation. It wastes too much memory space.
Decreasing the
SECTION_SIZE_BITSon arm64 might be an alternative, butthere
Quoted 4 lines
are too many concerns from other constraints, e.g.
PAGE_SIZE, hugetlb,SPARSEMEM_VMEMMAP, page bits in struct page ...Beside decreasing the
SECTION_SIZE_BITS, we can also relax the kmemalignment
Quoted 3 lines
with
memory_block_size_bytes().Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device. dax
pmem
Quoted 1 line
can be used as ram with smaller gap. Also the kmem hotplug add/remove
are both
Quoted 2 lines
tested on arm64/x86 guest.
Hi,
I am not convinced this use case is worth such hacks (that’s what it is) for now. On real machines pmem is big - your example (losing 50% is extreme).
Quoted 5 lines
I would much rather want to see the section size on arm64 reduced. I remember there were patches and that at least with a base page size of 4k it can be reduced drastically (64k base pages are more problematic due to the ridiculous THP size of 512M). But could be a section size of 512 is possible on all configs right now.
Yes, I once investigated how to reduce section size on arm64 thoughtfully:
There are many constraints for reducing SECTION_SIZE_BITS
1. Given page->flags bits is limited, SECTION_SIZE_BITS can't be reduced too much.
2. Once CONFIG_SPARSEMEM_VMEMMAP is enabled, section id will not be counted into page->flags.
3. MAX_ORDER depends on SECTION_SIZE_BITS
- 3.1 mmzone.h
#if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS
#error Allocator MAX_ORDER exceeds SECTION_SIZE
#endif
- 3.2 hugepage_init()MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER >= MAX_ORDER);
Hence when ARM64_4K_PAGES && CONFIG_SPARSEMEM_VMEMMAP are enabled, SECTION_SIZE_BITS can be reduced to 27.
But when ARM64_64K_PAGES, given 3.2, MAX_ORDER > 29-16 = 13. Given 3.1 SECTION_SIZE_BITS >= MAX_ORDER+15 > 28. So SECTION_SIZE_BITS can not be reduced to 27.
In one word, if we considered to reduce SECTION_SIZE_BITS on arm64, the Kconfig might be very complicated,e.g. we still need to consider the case for ARM64_16K_PAGES.
Quoted 4 lines
In the long term we might want to rework the memory block device model (eventually supporting old/new as discussed with Michal some time ago using a kernel parameter), dropping the fixed sizes
Has this been posted to Linux mm maillist? Sorry, searched and didn't find it.
--
Cheers,
Justin (Jia He)
Quoted 4 lines
- allowing sizes / addresses aligned with subsection size
- drastically reducing the number of devices for boot memory to only a hand full (e.g., one per resource / DIMM we can actually unplug again.
Quoted 29 lines
Long story short, I don’t like this hack.
Quoted 1 line
This patch series (mainly patch6/6) is based on the fixing patch, ~v5.8-
rc5 [2].
Quoted 24 lines
[1] https://lkml.org/lkml/2019/6/19/67
[2] https://lkml.org/lkml/2020/7/8/1546
Jia He (6):
mm/memory_hotplug: remove redundant memory block size alignment check
resource: exportfind_next_iomem_res()helpermm/memory_hotplug: allow pmem kmem not to align withmemory_block_sizemm/page_alloc: adjust the start,end in dax pmem kmem case
device-dax: relax the memblock size alignment forkmem_start
arm64: fall back tovmemmap_populate_basepagesif not aligned with
PMD_SIZEarch/arm64/mm/mmu.c | 4 ++++ drivers/base/memory.c | 24 ++++++++++++++++-------- drivers/dax/kmem.c | 22 +++++++++++++--------- include/linux/ioport.h | 3 +++ kernel/resource.c | 3 ++- mm/memory_hotplug.c | 39 ++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 14 ++++++++++++++ 7 files changed, 90 insertions(+), 19 deletions(-)--
2.17.1
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]
Quoted 88 lines · Justin He
Hi David
Quoted 80 lines
-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Wednesday, July 29, 2020 2:37 PM
To: Justin He <[email protected]>
Cc: Dan Williams <[email protected]>; Vishal Verma <[email protected]>; Mike Rapoport <[email protected]>; David Hildenbrand <[email protected]>; Catalin Marinas <[email protected]>; Will Deacon <[email protected]>; Greg Kroah-Hartman <[email protected]>; Rafael J. Wysocki <[email protected]>; Dave Jiang <[email protected]>; Andrew Morton <[email protected]>; Steve Capper <[email protected]>; Mark Rutland <[email protected]>; Logan Gunthorpe <[email protected]>; Anshuman Khandual <[email protected]>; Hsin-Yi Wang <[email protected]>; Jason Gunthorpe <[email protected]>; Dave Hansen <[email protected]>; Kees Cook <[email protected]>; [email protected]; linux- [email protected]; [email protected]; [email protected]; Wei Yang <[email protected]>; Pankaj Gupta <[email protected]>; Ira Weiny <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [RFC PATCH 0/6] decrease unnecessary gap due to pmem kmem alignmentQuoted 4 lines
Am 29.07.2020 um 05:35 schrieb Jia He <[email protected]>:
When enabling dax pmem as RAM device on arm64, I noticed that
kmem_startaddr indev_dax_kmem_probe()should be aligned w/SECTION_SIZE_BITS(30),i.e.
Quoted 1 line
1G memblock size. Even Dan Williams' sub-section patch series [1] had
been
Quoted 2 lines
upstream merged, it was not helpful due to hard limitation of
kmem_start:
$ndctl create-namespace -e namespace0.0 --mode=devdax --map=dev -s 2g -f-a 2M
Quoted 4 lines
$echo dax0.0 >
/sys/bus/dax/drivers/device_dax/unbind$echo dax0.0 >/sys/bus/dax/drivers/kmem/new_id$cat/proc/iomem
...Quoted 8 lines
23c000000-23fffffff : System RAM
23dd40000-23fecffff : reserved
23fed0000-23fffffff : reserved
240000000-33fdfffff : Persistent Memory
240000000-2403fffff : namespace0.0
280000000-2bfffffff : dax0.0 <- aligned with 1G boundary
280000000-2bfffffff : System RAM
Hence there is a big gap between0x2403fffffand0x280000000due to the1G
Quoted 3 lines
alignment.
Without this series, if qemu creates a 4G bytes nvdimm device, we can
only
Quoted 4 lines
use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000,2ffffffff] due to thehard
Quoted 3 lines
limitation. It wastes too much memory space.
Decreasing the
SECTION_SIZE_BITSon arm64 might be an alternative, butthere
Quoted 4 lines
are too many concerns from other constraints, e.g.
PAGE_SIZE, hugetlb,SPARSEMEM_VMEMMAP, page bits in struct page ...Beside decreasing the
SECTION_SIZE_BITS, we can also relax the kmemalignment
Quoted 3 lines
with
memory_block_size_bytes().Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device. dax
pmem
Quoted 1 line
can be used as ram with smaller gap. Also the kmem hotplug add/remove
are both
Quoted 2 lines
tested on arm64/x86 guest.
Hi,
I am not convinced this use case is worth such hacks (that’s what it is) for now. On real machines pmem is big - your example (losing 50% is extreme).
Quoted 5 lines
I would much rather want to see the section size on arm64 reduced. I remember there were patches and that at least with a base page size of 4k it can be reduced drastically (64k base pages are more problematic due to the ridiculous THP size of 512M). But could be a section size of 512 is possible on all configs right now.
Quoted 6 lines
Yes, I once investigated how to reduce section size on arm64 thoughtfully:
There are many constraints for reducingSECTION_SIZE_BITS
1. Givenpage->flagsbits is limited,SECTION_SIZE_BITScan't be reduced too much.
2. OnceCONFIG_SPARSEMEM_VMEMMAPis enabled, section id will not be counted intopage->flags.
Yep.
Quoted 5 lines
3.
MAX_ORDERdepends onSECTION_SIZE_BITS
- 3.1mmzone.h
#if (MAX_ORDER- 1 +PAGE_SHIFT) >SECTION_SIZE_BITS#error AllocatorMAX_ORDERexceedsSECTION_SIZE#endif
Yep, with 4k base pages it's 4 MB. However, with 64k base pages its 512MB ( :( ).
Quoted 8 lines
- 3.2
hugepage_init()MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER >= MAX_ORDER);Hence when
ARM64_4K_PAGES&&CONFIG_SPARSEMEM_VMEMMAPare enabled,SECTION_SIZE_BITScan be reduced to 27.
But whenARM64_64K_PAGES, given 3.2,MAX_ORDER> 29-16 = 13. Given 3.1SECTION_SIZE_BITS>= MAX_ORDER+15 > 28. SoSECTION_SIZE_BITScan not be reduced to 27.
I think there were plans to eventually switch to 2MB THP with 64k base pages as well (which can be emulated using some sort of consecutive PTE entries under arm64, don't ask me how this feature is called), theoretically also allowing smaller section sizes (when also reducing MAX_ORDER properly) I would highly appreciate that switch. Having max allocation/THP in the size of gigantic pages sounds very weird to me (and creates issues e.g., to support hot(un)plug of small memory blocks for virtio-mem). But I guess this is not under our control :)
Quoted 4 lines
In one word, if we considered to reduce
SECTION_SIZE_BITSon arm64, theKconfigmight be very complicated,e.g. we still need to consider the case forARM64_16K_PAGES.
Haven't looked into 16k base pages yet. But I remember it's in general more similar to 4k than to 64k (speaking about sane THP sizes and similar ...).
Quoted 7 lines
Quoted 5 lines
In the long term we might want to rework the memory block device model (eventually supporting old/new as discussed with Michal some time ago using a kernel parameter), dropping the fixed sizes
Has this been posted to Linux mm maillist? Sorry, searched and didn't find it.
Yeah, but I might not be able to dig it out anymore ...
Anyhow, the idea would be to have some magic switch that converts between old and new world, to not break userspace that relies on that.
With old, everything would continue to work as it is. With *new* we would have the reduced number of memory blocks for boot memory and decoupled it from a strict, static memory block size.
There would be another option in corner cases right now. If you would *know* that the metadata memory has no memmap/idendity mapping and have 1G alignment for your pmem device (including the metadata part)
1. add_memory_device_managed() the whole memory, including the metadata part
2. use generic_online_pages() to not expose metadata pages to the buddy
3. Mark metdata pages in a special way, such that you can e.g., allow to offline memory again, including the metdata pages (e.g., PG_offline + memory notifier like virtio-mem does)
3. would only be relevant to support offlining of memory again.
If the metadata part is, however, already ZONE_DEVICE with a memmap, then that's not an option. (I have no idea how that metadata part is used, sorry)
Signature · 3 lines
Thanks, David / dhildenb
Hi Justin,
Quoted 59 lines · Justin He
Hi David
Quoted 32 lines
Quoted 2 lines
Without this series, if qemu creates a 4G bytes nvdimm device, we can
only
Quoted 4 lines
use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000,2ffffffff] due to thehard
Quoted 3 lines
limitation. It wastes too much memory space.
Decreasing the
SECTION_SIZE_BITSon arm64 might be an alternative, butthere
Quoted 4 lines
are too many concerns from other constraints, e.g.
PAGE_SIZE, hugetlb,SPARSEMEM_VMEMMAP, page bits in struct page ...Beside decreasing the
SECTION_SIZE_BITS, we can also relax the kmemalignment
Quoted 3 lines
with
memory_block_size_bytes().Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device. dax
pmem
Quoted 1 line
can be used as ram with smaller gap. Also the kmem hotplug add/remove
are both
Quoted 2 lines
tested on arm64/x86 guest.
Hi,
I am not convinced this use case is worth such hacks (that’s what it is) for now. On real machines pmem is big - your example (losing 50% is extreme).
Quoted 5 lines
I would much rather want to see the section size on arm64 reduced. I remember there were patches and that at least with a base page size of 4k it can be reduced drastically (64k base pages are more problematic due to the ridiculous THP size of 512M). But could be a section size of 512 is possible on all configs right now.
Yes, I once investigated how to reduce section size on arm64 thoughtfully:
There are many constraints for reducingSECTION_SIZE_BITS
1. Givenpage->flagsbits is limited,SECTION_SIZE_BITScan't be reduced too much.
2. OnceCONFIG_SPARSEMEM_VMEMMAPis enabled, section id will not be counted intopage->flags.
3.MAX_ORDERdepends onSECTION_SIZE_BITS
- 3.1mmzone.h
#if (MAX_ORDER- 1 +PAGE_SHIFT) >SECTION_SIZE_BITS
#error AllocatorMAX_ORDERexceedsSECTION_SIZE
#endif
- 3.2hugepage_init()MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER >= MAX_ORDER);Hence when
ARM64_4K_PAGES&&CONFIG_SPARSEMEM_VMEMMAPare enabled,SECTION_SIZE_BITScan be reduced to 27.
But whenARM64_64K_PAGES, given 3.2,MAX_ORDER> 29-16 = 13. Given 3.1SECTION_SIZE_BITS>= MAX_ORDER+15 > 28. SoSECTION_SIZE_BITScan not be reduced to 27.
Quoted 3 lines
In one word, if we considered to reduce
SECTION_SIZE_BITSon arm64, theKconfigmight be very complicated,e.g. we still need to consider the case forARM64_16K_PAGES.
It is not necessary to pollute Kconfig with that. arch/arm64/include/asm/sparesemem.h can have something like
#ifdef CONFIG_ARM64_64K_PAGES
#define SPARSE_SECTION_SIZE 29
#elif defined(CONFIG_ARM16K_PAGES)
#define SPARSE_SECTION_SIZE 28
#elif defined(CONFIG_ARM4K_PAGES)
#define SPARSE_SECTION_SIZE 27
#else
#error
#endif
There is still large gap with ARM64_64K_PAGES, though.
As for SPARSEMEM without VMEMMAP, are there actual benefits to use it?
Quoted 48 lines
Quoted 4 lines
In the long term we might want to rework the memory block device model (eventually supporting old/new as discussed with Michal some time ago using a kernel parameter), dropping the fixed sizes
Has this been posted to Linux mm maillist? Sorry, searched and didn't find it.
--
Cheers,
Justin (Jia He)Quoted 33 lines
- allowing sizes / addresses aligned with subsection size
- drastically reducing the number of devices for boot memory to only a hand full (e.g., one per resource / DIMM we can actually unplug again.Long story short, I don’t like this hack.
Quoted 1 line
This patch series (mainly patch6/6) is based on the fixing patch, ~v5.8-
rc5 [2].
Quoted 24 lines
[1] https://lkml.org/lkml/2019/6/19/67
[2] https://lkml.org/lkml/2020/7/8/1546
Jia He (6):
mm/memory_hotplug: remove redundant memory block size alignment check
resource: exportfind_next_iomem_res()helpermm/memory_hotplug: allow pmem kmem not to align withmemory_block_sizemm/page_alloc: adjust the start,end in dax pmem kmem case
device-dax: relax the memblock size alignment forkmem_start
arm64: fall back tovmemmap_populate_basepagesif not aligned with
PMD_SIZEarch/arm64/mm/mmu.c | 4 ++++ drivers/base/memory.c | 24 ++++++++++++++++-------- drivers/dax/kmem.c | 22 +++++++++++++--------- include/linux/ioport.h | 3 +++ kernel/resource.c | 3 ++- mm/memory_hotplug.c | 39 ++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 14 ++++++++++++++ 7 files changed, 90 insertions(+), 19 deletions(-)--
2.17.1
Signature · 2 lines
Sincerely yours, Mike.
Quoted 69 lines · Mike Rapoport
Hi Justin,
Quoted 59 lines · Justin He
Hi David
Quoted 32 lines
Quoted 2 lines
Without this series, if qemu creates a 4G bytes nvdimm device, we can
only
Quoted 4 lines
use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000,2ffffffff] due to thehard
Quoted 3 lines
limitation. It wastes too much memory space.
Decreasing the
SECTION_SIZE_BITSon arm64 might be an alternative, butthere
Quoted 4 lines
are too many concerns from other constraints, e.g.
PAGE_SIZE, hugetlb,SPARSEMEM_VMEMMAP, page bits in struct page ...Beside decreasing the
SECTION_SIZE_BITS, we can also relax the kmemalignment
Quoted 3 lines
with
memory_block_size_bytes().Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device. dax
pmem
Quoted 1 line
can be used as ram with smaller gap. Also the kmem hotplug add/remove
are both
Quoted 2 lines
tested on arm64/x86 guest.
Hi,
I am not convinced this use case is worth such hacks (that’s what it is) for now. On real machines pmem is big - your example (losing 50% is extreme).
Quoted 5 lines
I would much rather want to see the section size on arm64 reduced. I remember there were patches and that at least with a base page size of 4k it can be reduced drastically (64k base pages are more problematic due to the ridiculous THP size of 512M). But could be a section size of 512 is possible on all configs right now.
Yes, I once investigated how to reduce section size on arm64 thoughtfully:
There are many constraints for reducingSECTION_SIZE_BITS
1. Givenpage->flagsbits is limited,SECTION_SIZE_BITScan't be reduced too much.
2. OnceCONFIG_SPARSEMEM_VMEMMAPis enabled, section id will not be counted intopage->flags.
3.MAX_ORDERdepends onSECTION_SIZE_BITS
- 3.1mmzone.h
#if (MAX_ORDER- 1 +PAGE_SHIFT) >SECTION_SIZE_BITS
#error AllocatorMAX_ORDERexceedsSECTION_SIZE
#endif
- 3.2hugepage_init()MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER >= MAX_ORDER);Hence when
ARM64_4K_PAGES&&CONFIG_SPARSEMEM_VMEMMAPare enabled,SECTION_SIZE_BITScan be reduced to 27.
But whenARM64_64K_PAGES, given 3.2,MAX_ORDER> 29-16 = 13. Given 3.1SECTION_SIZE_BITS>= MAX_ORDER+15 > 28. SoSECTION_SIZE_BITScan not be reduced to 27.Quoted 3 lines
In one word, if we considered to reduce
SECTION_SIZE_BITSon arm64, theKconfigmight be very complicated,e.g. we still need to consider the case forARM64_16K_PAGES.It is not necessary to pollute
Kconfigwith that.arch/arm64/include/asm/sparesemem.hcan have something like
Quoted 9 lines
#ifdef
CONFIG_ARM64_64K_PAGES
#defineSPARSE_SECTION_SIZE29
#elifdefined(CONFIG_ARM16K_PAGES)
#defineSPARSE_SECTION_SIZE28
#elifdefined(CONFIG_ARM4K_PAGES)
#defineSPARSE_SECTION_SIZE27
#else
#error
#endif
ack
Quoted 4 lines
There is still large gap with
ARM64_64K_PAGES, though.As for SPARSEMEM without VMEMMAP, are there actual benefits to use it?
I was asking myself the same question a while ago and didn't really find a compelling one.
I think it's always enabled as default (SPARSEMEM_VMEMMAP_ENABLE) and would require config tweaks to even disable it.
Signature · 3 lines
Thanks, David / dhildenb
Quoted 83 lines · David Hildenbrand
Quoted 69 lines · Mike Rapoport
Hi Justin,
Quoted 59 lines · Justin He
Hi David
Quoted 32 lines
Quoted 2 lines
Without this series, if qemu creates a 4G bytes nvdimm device, we can
only
Quoted 4 lines
use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000,2ffffffff] due to thehard
Quoted 3 lines
limitation. It wastes too much memory space.
Decreasing the
SECTION_SIZE_BITSon arm64 might be an alternative, butthere
Quoted 4 lines
are too many concerns from other constraints, e.g.
PAGE_SIZE, hugetlb,SPARSEMEM_VMEMMAP, page bits in struct page ...Beside decreasing the
SECTION_SIZE_BITS, we can also relax the kmemalignment
Quoted 3 lines
with
memory_block_size_bytes().Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device. dax
pmem
Quoted 1 line
can be used as ram with smaller gap. Also the kmem hotplug add/remove
are both
Quoted 2 lines
tested on arm64/x86 guest.
Hi,
I am not convinced this use case is worth such hacks (that’s what it is) for now. On real machines pmem is big - your example (losing 50% is extreme).
Quoted 5 lines
I would much rather want to see the section size on arm64 reduced. I remember there were patches and that at least with a base page size of 4k it can be reduced drastically (64k base pages are more problematic due to the ridiculous THP size of 512M). But could be a section size of 512 is possible on all configs right now.
Yes, I once investigated how to reduce section size on arm64 thoughtfully:
There are many constraints for reducingSECTION_SIZE_BITS
1. Givenpage->flagsbits is limited,SECTION_SIZE_BITScan't be reduced too much.
2. OnceCONFIG_SPARSEMEM_VMEMMAPis enabled, section id will not be counted intopage->flags.
3.MAX_ORDERdepends onSECTION_SIZE_BITS
- 3.1mmzone.h
#if (MAX_ORDER- 1 +PAGE_SHIFT) >SECTION_SIZE_BITS
#error AllocatorMAX_ORDERexceedsSECTION_SIZE
#endif
- 3.2hugepage_init()MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER >= MAX_ORDER);Hence when
ARM64_4K_PAGES&&CONFIG_SPARSEMEM_VMEMMAPare enabled,SECTION_SIZE_BITScan be reduced to 27.
But whenARM64_64K_PAGES, given 3.2,MAX_ORDER> 29-16 = 13. Given 3.1SECTION_SIZE_BITS>= MAX_ORDER+15 > 28. SoSECTION_SIZE_BITScan not be reduced to 27.Quoted 3 lines
In one word, if we considered to reduce
SECTION_SIZE_BITSon arm64, theKconfigmight be very complicated,e.g. we still need to consider the case forARM64_16K_PAGES.It is not necessary to pollute
Kconfigwith that.arch/arm64/include/asm/sparesemem.hcan have something likeQuoted 9 lines
#ifdef
CONFIG_ARM64_64K_PAGES
#defineSPARSE_SECTION_SIZE29
#elifdefined(CONFIG_ARM16K_PAGES)
#defineSPARSE_SECTION_SIZE28
#elifdefined(CONFIG_ARM4K_PAGES)
#defineSPARSE_SECTION_SIZE27
#else
#error
#endifack
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
There is still large gap with
ARM64_64K_PAGES, though.As for SPARSEMEM without VMEMMAP, are there actual benefits to use it?
I was asking myself the same question a while ago and didn't really find a compelling one.
Memory overhead for VMEMMAP is larger, especially for arm64 that knows how to free empty parts of the memory map with "classic" SPARSEMEM.
Quoted 2 lines
I think it's always enabled as default (
SPARSEMEM_VMEMMAP_ENABLE) and would require config tweaks to even disable it.
Nope, it's right there in menuconfig,
"Memory Management options" -> "Sparse Memory virtual memmap"
Quoted 5 lines
Signature · 3 lines
Thanks, David / dhildenb
Signature · 2 lines
Sincerely yours, Mike.
Quoted 86 lines · Mike Rapoport
Quoted 85 lines · David Hildenbrand
Quoted 69 lines · Mike Rapoport
Hi Justin,
Quoted 59 lines · Justin He
Hi David
Quoted 32 lines
Quoted 2 lines
Without this series, if qemu creates a 4G bytes nvdimm device, we can
only
Quoted 4 lines
use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000,2ffffffff] due to thehard
Quoted 3 lines
limitation. It wastes too much memory space.
Decreasing the
SECTION_SIZE_BITSon arm64 might be an alternative, butthere
Quoted 4 lines
are too many concerns from other constraints, e.g.
PAGE_SIZE, hugetlb,SPARSEMEM_VMEMMAP, page bits in struct page ...Beside decreasing the
SECTION_SIZE_BITS, we can also relax the kmemalignment
Quoted 3 lines
with
memory_block_size_bytes().Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device. dax
pmem
Quoted 1 line
can be used as ram with smaller gap. Also the kmem hotplug add/remove
are both
Quoted 2 lines
tested on arm64/x86 guest.
Hi,
I am not convinced this use case is worth such hacks (that’s what it is) for now. On real machines pmem is big - your example (losing 50% is extreme).
Quoted 5 lines
I would much rather want to see the section size on arm64 reduced. I remember there were patches and that at least with a base page size of 4k it can be reduced drastically (64k base pages are more problematic due to the ridiculous THP size of 512M). But could be a section size of 512 is possible on all configs right now.
Yes, I once investigated how to reduce section size on arm64 thoughtfully:
There are many constraints for reducingSECTION_SIZE_BITS
1. Givenpage->flagsbits is limited,SECTION_SIZE_BITScan't be reduced too much.
2. OnceCONFIG_SPARSEMEM_VMEMMAPis enabled, section id will not be counted intopage->flags.
3.MAX_ORDERdepends onSECTION_SIZE_BITS
- 3.1mmzone.h
#if (MAX_ORDER- 1 +PAGE_SHIFT) >SECTION_SIZE_BITS
#error AllocatorMAX_ORDERexceedsSECTION_SIZE
#endif
- 3.2hugepage_init()MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER >= MAX_ORDER);Hence when
ARM64_4K_PAGES&&CONFIG_SPARSEMEM_VMEMMAPare enabled,SECTION_SIZE_BITScan be reduced to 27.
But whenARM64_64K_PAGES, given 3.2,MAX_ORDER> 29-16 = 13. Given 3.1SECTION_SIZE_BITS>= MAX_ORDER+15 > 28. SoSECTION_SIZE_BITScan not be reduced to 27.Quoted 3 lines
In one word, if we considered to reduce
SECTION_SIZE_BITSon arm64, theKconfigmight be very complicated,e.g. we still need to consider the case forARM64_16K_PAGES.It is not necessary to pollute
Kconfigwith that.arch/arm64/include/asm/sparesemem.hcan have something likeQuoted 9 lines
#ifdef
CONFIG_ARM64_64K_PAGES
#defineSPARSE_SECTION_SIZE29
#elifdefined(CONFIG_ARM16K_PAGES)
#defineSPARSE_SECTION_SIZE28
#elifdefined(CONFIG_ARM4K_PAGES)
#defineSPARSE_SECTION_SIZE27
#else
#error
#endifack
Quoted 3 lines
There is still large gap with
ARM64_64K_PAGES, though.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
As for SPARSEMEM without VMEMMAP, are there actual benefits to use it?
I was asking myself the same question a while ago and didn't really find a compelling one.
Memory overhead for VMEMMAP is larger, especially for arm64 that knows how to free empty parts of the memory map with "classic" SPARSEMEM.
You mean the hole punching within section memmap? (which is why their pfn_valid() implementation is special)
(I do wonder why that shouldn't work with VMEMMAP, or is it simply not implemented?)
Quoted 7 lines
Quoted 3 lines
I think it's always enabled as default (
SPARSEMEM_VMEMMAP_ENABLE) and would require config tweaks to even disable it.Nope, it's right there in menuconfig,
"Memory Management options" -> "Sparse Memory virtual memmap"
Ah, good to know.
Signature · 6 lines
Thanks, David / dhildenb _______________________________________________ Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]
Quoted 7 lines · David Hildenbrand
Quoted 6 lines · Mike Rapoport
Quoted 5 lines · David Hildenbrand
Quoted 4 lines
There is still large gap with
ARM64_64K_PAGES, though.As for SPARSEMEM without VMEMMAP, are there actual benefits to use it?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I was asking myself the same question a while ago and didn't really find a compelling one.
Memory overhead for VMEMMAP is larger, especially for arm64 that knows how to free empty parts of the memory map with "classic" SPARSEMEM.
You mean the hole punching within section memmap? (which is why their
pfn_valid()implementation is special)
Yes, arm (both 32 and 64) do this. And for smaller systems with a few memory banks this is very reasonable to trade slight (if any) slowdown in pfn_valid() for several megs of memory.
Quoted 2 lines
(I do wonder why that shouldn't work with VMEMMAP, or is it simply not implemented?)
It's not implemented. There was a patch [1] recently to implement this.
[1] https://lore.kernel.org/lkml/[email protected]/
Quoted 5 lines
Signature · 3 lines
Thanks, David / dhildenb
Signature · 5 lines
Sincerely yours, Mike. _______________________________________________ Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]
Quoted 114 lines
-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Wednesday, July 29, 2020 5:35 PM
To: Mike Rapoport <[email protected]>; Justin He <[email protected]>
Cc: Dan Williams <[email protected]>; Vishal Verma <[email protected]>; Catalin Marinas <[email protected]>; Will Deacon <[email protected]>; Greg Kroah-Hartman <[email protected]>; Rafael J. Wysocki <[email protected]>; Dave Jiang <[email protected]>; Andrew Morton <[email protected]>; Steve Capper <[email protected]>; Mark Rutland <[email protected]>; Logan Gunthorpe <[email protected]>; Anshuman Khandual <[email protected]>; Hsin-Yi Wang <[email protected]>; Jason Gunthorpe <[email protected]>; Dave Hansen <[email protected]>; Kees Cook <[email protected]>; [email protected]; linux- [email protected]; [email protected]; [email protected]; Wei Yang <[email protected]>; Pankaj Gupta <[email protected]>; Ira Weiny <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [RFC PATCH 0/6] decrease unnecessary gap due to pmem kmem alignmentQuoted 11 lines · Mike Rapoport
Hi Justin,
Quoted 8 lines · Justin He
Hi David
Quoted 7 lines
Quoted 2 lines
Without this series, if qemu creates a 4G bytes nvdimm device, we can
only
Quoted 4 lines
use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000,2ffffffff] due tothe
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
hard
Quoted 3 lines
limitation. It wastes too much memory space.
Decreasing the
SECTION_SIZE_BITSon arm64 might be an alternative,but
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
there
Quoted 4 lines
are too many concerns from other constraints, e.g.
PAGE_SIZE, hugetlb,SPARSEMEM_VMEMMAP, page bits in struct page ...Beside decreasing the
SECTION_SIZE_BITS, we can also relax the kmemalignment
Quoted 3 lines
with
memory_block_size_bytes().Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device.
dax
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
pmem
Quoted 1 line
can be used as ram with smaller gap. Also the kmem hotplug add/remove
are both
Quoted 2 lines
tested on arm64/x86 guest.
Hi,
I am not convinced this use case is worth such hacks (that’s what it
is)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
for now. On real machines pmem is big - your example (losing 50% is extreme).
I would much rather want to see the section size on arm64 reduced. I remember there were patches and that at least with a base page size of
4k
Quoted 1 line
Quoted 1 line
Quoted 1 line
it can be reduced drastically (64k base pages are more problematic due
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
the ridiculous THP size of 512M). But could be a section size of 512
is
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
possible on all configs right now.
Yes, I once investigated how to reduce section size on arm64
thoughtfully:
Quoted 2 lines
Quoted 2 lines
There are many constraints for reducing
SECTION_SIZE_BITS
1. Givenpage->flagsbits is limited,SECTION_SIZE_BITScan't bereduced too
Quoted 2 lines
Quoted 2 lines
much.
2. OnceCONFIG_SPARSEMEM_VMEMMAPis enabled, section id will not becounted
Quoted 9 lines
Quoted 9 lines
into page->flags.3.
MAX_ORDERdepends onSECTION_SIZE_BITS
- 3.1mmzone.h
#if (MAX_ORDER- 1 +PAGE_SHIFT) >SECTION_SIZE_BITS#error AllocatorMAX_ORDERexceedsSECTION_SIZE#endif
- 3.2hugepage_init()MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER >= MAX_ORDER);Quoted 4 lines
Quoted 4 lines
Hence when
ARM64_4K_PAGES&&CONFIG_SPARSEMEM_VMEMMAPare enabled,SECTION_SIZE_BITScan be reduced to 27.
But whenARM64_64K_PAGES, given 3.2,MAX_ORDER> 29-16 = 13. Given 3.1SECTION_SIZE_BITS>= MAX_ORDER+15 > 28. SoSECTION_SIZE_BITScan not
Quoted 3 lines
Quoted 3 lines
be reduced to 27.
In one word, if we considered to reduce
SECTION_SIZE_BITSon arm64, the
KconfigQuoted 6 lines
Quoted 2 lines
might be very complicated,e.g. we still need to consider the case for
ARM64_16K_PAGES.It is not necessary to pollute
Kconfigwith that.arch/arm64/include/asm/sparesemem.hcan have something likeQuoted 9 lines
#ifdef
CONFIG_ARM64_64K_PAGES
#defineSPARSE_SECTION_SIZE29
#elifdefined(CONFIG_ARM16K_PAGES)
#defineSPARSE_SECTION_SIZE28
#elifdefined(CONFIG_ARM4K_PAGES)
#defineSPARSE_SECTION_SIZE27
#else
#error
#endif
Quoted 1 line
ack
Thanks, David and Mike. Will discuss it further more with arm internally about the thoughtful section_size change
--
Cheers,
Justin (Jia He)
_______________________________________________
Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]