aarch64/mm: speedup memory initialisation
PATCH 1/1 aarch64/mm: speedup memory initialisation
On ARM64 memmap_init_zone is used during bootmem_init, which iterates over all pages in the memory starting at the lowest address until the highest address is reached. On arm64 this ends up in searching a memmory region containing for each single page between lowest and highest available physicall address.
Having a sparse memory system there may be some big holes in the memory map. For each page in this holes a lookup is done, which is implemented as a binary search on the available memory blocks.
Adding a memmap_init for aarch64 to do the init only for the available memory areas reduces the time needed for initialising memory on startup. On a Renesas R-CAR M3 based system with a total hole of 20GB bootmem_init execution time is reduced from 378ms to 84ms.
--- arch/arm64/include/asm/pgtable.h | 7 +++++++ arch/arm64/mm/init.c | 24 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+)
arch/arm64/include/asm/pgtable.h +7 -0
@@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)return (lhs == rhs); } +#ifdef CONFIG_SPARSEMEM +/* arch mem_map init routine is needed due to holes in a memmap */ +# define __HAVE_ARCH_MEMMAP_INIT +void memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn); +#endif /* CONFIG_SPARSEMEM */ +/* * Huge pte definitions. */
arch/arm64/mm/init.c +24 -0
@@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn) }EXPORT_SYMBOL (pfn_valid); +#ifdef CONFIG_SPARSEMEM +void __meminit +memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn) +{ +struct memblock_region *reg; + +for_each_memblock (memory, reg) { +unsigned long start =memblock_region_memory_base_pfn (reg); +unsigned long end =memblock_region_memory_end_pfn (reg); + +if (start < start_pfn) + start = start_pfn; +if (end > start_pfn + size) + end = start_pfn + size; + +if (start < end) { +memmap_init_zone (end - start, nid, zone, start, +MEMMAP_EARLY ,NULL ); + } + } +} +#endif /* CONFIG_SPARSEMEM */ +static phys_addr_t memory_limit =PHYS_ADDR_MAX ;/*
Signature · 6 lines
2.23.0 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 9 lines · Hubert Ralf
On ARM64
memmap_init_zoneis used duringbootmem_init, which iterates over all pages in the memory starting at the lowest address until the highest address is reached. On arm64 this ends up in searching a memmory region containing for each single page between lowest and highest available physicall address.Having a sparse memory system there may be some big holes in the memory map. For each page in this holes a lookup is done, which is implemented as a binary search on the available memory blocks.
It sounds like that's one of the many related issues for which the real solution is "handle EFI no-map regions in a way which makes pfn_valid() not terrible" - if we are prepared to have workarounds for individual hot-spots, they should probably carry a big reminder to put things back once pfn_valid() is sorted out.
That said, even with the current implementation, the memblock search is short-circuited by the valid_section() check, so the impact should already be limited to just boundary regions where RAM is weirdly aligned and a hole starts or ends mid-section. Of course, regardless of any check overhead there might well still be an argument for not walking through large avoidable holes one PFN at a time, but the rationale as given doesn't seem to quite add up.
Robin.
Quoted 55 lines
Adding a
memmap_initfor aarch64 to do the init only for the available memory areas reduces the time needed for initialising memory on startup. On a Renesas R-CAR M3 based system with a total hole of 20GBbootmem_initexecution time is reduced from 378ms to 84ms.Signed-off-by Ralf Hubert <[email protected]>--- arch/arm64/include/asm/pgtable.h | 7 +++++++ arch/arm64/mm/init.c | 24 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+)arch/arm64/include/asm/pgtable.h +7
-0@@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)return (lhs == rhs); } +#ifdef CONFIG_SPARSEMEM +/* arch mem_map init routine is needed due to holes in a memmap */ +# define __HAVE_ARCH_MEMMAP_INIT +void memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn); +#endif /* CONFIG_SPARSEMEM */ +/* * Huge pte definitions. */ arch/arm64/mm/init.c +17
-0@@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn) }EXPORT_SYMBOL (pfn_valid); +#ifdef CONFIG_SPARSEMEM +void __meminit +memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn) +{ +struct memblock_region *reg; + +for_each_memblock (memory, reg) { +unsigned long start =memblock_region_memory_base_pfn (reg); +unsigned long end =memblock_region_memory_end_pfn (reg); + +if (start < start_pfn) + start = start_pfn; +if (end > start_pfn + size) + end = start_pfn + size; + +if (start < end) {
arch/arm64/mm/init.c int pfn_valid(unsigned long pfn) · line 279
Quoted 10 lines
arch/arm64/mm/init.c +7
-0+memmap_init_zone (end - start, nid, zone, start, + MEMMAP_EARLY, NULL); + } + } +} +#endif /* CONFIG_SPARSEMEM */ +static phys_addr_t memory_limit =PHYS_ADDR_MAX ;/*
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Hubert,
(Subject-Nit: The prefix for this part of the kernel is 'arm64: mm:'. git log --oneline $file will usually give you enough examples you can spot the pattern.)
Quoted 6 lines · Hubert Ralf
On ARM64
memmap_init_zoneis used duringbootmem_init, which iterates over all pages in the memory starting at the lowest address until the highest address is reached. On arm64 this ends up in searching a memmory region containing for each single page between lowest and highest available physicall address.
Quoted 8 lines
Having a sparse memory system there may be some big holes in the memory map. For each page in this holes a lookup is done, which is implemented as a binary search on the available memory blocks.
Adding a
memmap_initfor aarch64 to do the init only for the available memory areas reduces the time needed for initialising memory on startup. On a Renesas R-CAR M3 based system with a total hole of 20GBbootmem_initexecution time is reduced from 378ms to 84ms.
Hmm, there is nothing arm64 specific about this SPARSEMEM behaviour.
Is there any reason this can't be done in core code, where it would benefit other architectures too?
(You'd need it to depend on !ARCH_DISCARD_MEMBLOCK as it looks like memory-hotplug uses this late).
Thanks,
James
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi James,
Quoted 5 lines · On 09/11/2019, 17:22 +0100 James Morse wrote:
Hi Hubert,
(Subject-Nit: The prefix for this part of the kernel is 'arm64: mm:'. git log --oneline $file will usually give you enough examples you can spot the pattern.)
Thanks for the hint.
Quoted 11 lines · Hubert Ralf
Quoted 9 lines
On ARM64
memmap_init_zoneis used duringbootmem_init, which iterates over all pages in the memory starting at the lowest address until the highest address is reached. On arm64 this ends up in searching a memmory region containing for each single page between lowest and highest available physicall address.
Having a sparse memory system there may be some big holes in the memory map. For each page in this holes a lookup is done, which is implemented as a binary search on the available memory blocks.
Quoted 8 lines
Quoted 4 lines
Adding a
memmap_initfor aarch64 to do the init only for the available memory areas reduces the time needed for initialising memory on startup. On a Renesas R-CAR M3 based system with a total hole of 20GBbootmem_initexecution time is reduced from 378ms to 84ms.Hmm, there is nothing arm64 specific about this SPARSEMEM behaviour. Is there any reason this can't be done in core code, where it would benefit other architectures too?
I'll try to move this to the core code.
Quoted 8 lines
(You'd need it to depend on !ARCH_DISCARD_MEMBLOCK as it looks like memory-hotplug uses this late).
Thanks,
James
Thanks,
Ralf
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 26 lines · Hubert Ralf
On ARM64
memmap_init_zoneis used duringbootmem_init, which iterates over all pages in the memory starting at the lowest address until the highest address is reached. On arm64 this ends up in searching a memmory region containing for each single page between lowest and highest available physicall address.Having a sparse memory system there may be some big holes in the memory map. For each page in this holes a lookup is done, which is implemented as a binary search on the available memory blocks.
Adding a
memmap_initfor aarch64 to do the init only for the available memory areas reduces the time needed for initialising memory on startup. On a Renesas R-CAR M3 based system with a total hole of 20GBbootmem_initexecution time is reduced from 378ms to 84ms.Signed-off-by Ralf Hubert <[email protected]>--- arch/arm64/include/asm/pgtable.h | 7 +++++++ arch/arm64/mm/init.c | 24 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+)arch/arm64/include/asm/pgtable.h +0
-0
arch/arm64/include/asm/pgtable.h static inline int pte_same(pte_t pte_a, pte_t pte_b) · line 303
Quoted 6 lines
arch/arm64/include/asm/pgtable.h +3
-0@@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)return (lhs == rhs); } +#ifdef CONFIG_SPARSEMEM +/* arch mem_map init routine is needed due to holes in a memmap */ +# define __HAVE_ARCH_MEMMAP_INIT
This is not required any more. Its gone with the following commit which also made generic memmap_init() an weak function currently overridden only on ia64.
dfb3ccd00a0 ("mm: make memmap_init a proper function")
Quoted 28 lines
arch/arm64/include/asm/pgtable.h +4
-0+void memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn); +#endif /* CONFIG_SPARSEMEM */ +/* * Huge pte definitions. */ arch/arm64/mm/init.c +13
-0@@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn) }EXPORT_SYMBOL (pfn_valid); +#ifdef CONFIG_SPARSEMEM +void __meminit +memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn) +{ +struct memblock_region *reg; + +for_each_memblock (memory, reg) { +unsigned long start =memblock_region_memory_base_pfn (reg); +unsigned long end =memblock_region_memory_end_pfn (reg); + +if (start < start_pfn) + start = start_pfn;
arch/arm64/mm/init.c int pfn_valid(unsigned long pfn) · line 275
Quoted 10 lines
arch/arm64/mm/init.c +10
-0+if (end > start_pfn + size) + end = start_pfn + size; + +if (start < end) { +memmap_init_zone (end - start, nid, zone, start, +MEMMAP_EARLY ,NULL ); + } + } +} +#endif /* CONFIG_SPARSEMEM */
In generic mmap_init(), the current high cost comes from early_pfn_valid() check for each pfn in memmap_init_zone() given that early_pfn_valid() is pfn_valid() when CONFIG_SPARSEMEM which is known to be expensive on arm64.
Though we cannot do anything about pfns which are really present but the high cost for non present pfns should be eliminated. The following check in the above for_each_memblock() loop can achieve that.
if (reg->flags & MEMBLOCK_NOMAP)
continue;
MEMBLOCK_NOMAP universally should not be initialized into a zone and holes if any should also be universally skipped across platforms. So these changes can be moved into generic memmap_init() which will benefit other platforms.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Am Donnerstag, den 12.09.2019, 13:42 +0530 schrieb Anshuman Khandual:
Quoted 88 lines · Hubert Ralf
Quoted 26 lines
On ARM64
memmap_init_zoneis used duringbootmem_init, which iterates over all pages in the memory starting at the lowest address until the highest address is reached. On arm64 this ends up in searching a memmory region containing for each single page between lowest and highest available physicall address.Having a sparse memory system there may be some big holes in the memory map. For each page in this holes a lookup is done, which is implemented as a binary search on the available memory blocks.
Adding a
memmap_initfor aarch64 to do the init only for the available memory areas reduces the time needed for initialising memory on startup. On a Renesas R-CAR M3 based system with a total hole of 20GBbootmem_initexecution time is reduced from 378ms to 84ms.Signed-off-by Ralf Hubert <[email protected]>--- arch/arm64/include/asm/pgtable.h | 7 +++++++ arch/arm64/mm/init.c | 24 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+)arch/arm64/include/asm/pgtable.h +0
-0arch/arm64/include/asm/pgtable.h static inline int pte_same(pte_t pte_a, pte_t pte_b) · line 303
Quoted 6 lines
arch/arm64/include/asm/pgtable.h +3
-0@@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)return (lhs == rhs); } +#ifdef CONFIG_SPARSEMEM +/* arch mem_map init routine is needed due to holes in a memmap */ +# define __HAVE_ARCH_MEMMAP_INITThis is not required any more. Its gone with the following commit which also made generic
memmap_init()an weak function currently overridden only on ia64.
dfb3ccd00a0("mm: makememmap_inita proper function")Quoted 28 lines
arch/arm64/include/asm/pgtable.h +4
-0+void memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn); +#endif /* CONFIG_SPARSEMEM */ +/* * Huge pte definitions. */ arch/arm64/mm/init.c +13
-0@@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn) }EXPORT_SYMBOL (pfn_valid); +#ifdef CONFIG_SPARSEMEM +void __meminit +memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn) +{ +struct memblock_region *reg; + +for_each_memblock (memory, reg) { +unsigned long start =memblock_region_memory_base_pfn (reg); +unsigned long end =memblock_region_memory_end_pfn (reg); + +if (start < start_pfn) + start = start_pfn;arch/arm64/mm/init.c int pfn_valid(unsigned long pfn) · line 275
Quoted 10 lines
arch/arm64/mm/init.c +10
-0+if (end > start_pfn + size) + end = start_pfn + size; + +if (start < end) { +memmap_init_zone (end - start, nid, zone, start, +MEMMAP_EARLY ,NULL ); + } + } +} +#endif /* CONFIG_SPARSEMEM */ In generic
mmap_init(), the current high cost comes fromearly_pfn_valid()check for each pfn inmemmap_init_zone()given thatearly_pfn_valid()ispfn_valid()whenCONFIG_SPARSEMEMwhich is known to be expensive on arm64.Though we cannot do anything about pfns which are really present but the high cost for non present pfns should be eliminated. The following check in the above
for_each_memblock()loop can achieve that.
Quoted 6 lines
if (
reg->flags&MEMBLOCK_NOMAP)continue;
MEMBLOCK_NOMAPuniversally should not be initialized into a zone and holes if any should also be universally skipped across platforms. So these changes can be moved into genericmemmap_init()which will benefit other platforms.
Not sure if I got this. This is a additional short path for memblocks with the MEMBLOCK_NOMAP flag set, right? For memblocks without these flag the remaining code still needs to be executed?
In my case I have 4 memblocks with 1.5GB RAM at 0x4000 0000, 0x4 8000 0000, 0x6 0000 0000 and 0x6 8000 0000. None of them has the MEMBLOCK_NOMAP flag set.
_______________________________________________
linux-arm-kernel mailing list [email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 92 lines · Hubert Ralf
Am Donnerstag, den 12.09.2019, 13:42 +0530 schrieb Anshuman Khandual:
Quoted 91 lines · Hubert Ralf
Quoted 26 lines
On ARM64
memmap_init_zoneis used duringbootmem_init, which iterates over all pages in the memory starting at the lowest address until the highest address is reached. On arm64 this ends up in searching a memmory region containing for each single page between lowest and highest available physicall address.Having a sparse memory system there may be some big holes in the memory map. For each page in this holes a lookup is done, which is implemented as a binary search on the available memory blocks.
Adding a
memmap_initfor aarch64 to do the init only for the available memory areas reduces the time needed for initialising memory on startup. On a Renesas R-CAR M3 based system with a total hole of 20GBbootmem_initexecution time is reduced from 378ms to 84ms.Signed-off-by Ralf Hubert <[email protected]>--- arch/arm64/include/asm/pgtable.h | 7 +++++++ arch/arm64/mm/init.c | 24 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+)arch/arm64/include/asm/pgtable.h +0
-0arch/arm64/include/asm/pgtable.h static inline int pte_same(pte_t pte_a, pte_t pte_b) · line 303
Quoted 6 lines
arch/arm64/include/asm/pgtable.h +3
-0@@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)return (lhs == rhs); } +#ifdef CONFIG_SPARSEMEM +/* arch mem_map init routine is needed due to holes in a memmap */ +# define __HAVE_ARCH_MEMMAP_INITThis is not required any more. Its gone with the following commit which also made generic
memmap_init()an weak function currently overridden only on ia64.
dfb3ccd00a0("mm: makememmap_inita proper function")Quoted 28 lines
arch/arm64/include/asm/pgtable.h +4
-0+void memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn); +#endif /* CONFIG_SPARSEMEM */ +/* * Huge pte definitions. */ arch/arm64/mm/init.c +13
-0@@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn) }EXPORT_SYMBOL (pfn_valid); +#ifdef CONFIG_SPARSEMEM +void __meminit +memmap_init (unsigned long size,int nid,unsigned long zone, +unsigned long start_pfn) +{ +struct memblock_region *reg; + +for_each_memblock (memory, reg) { +unsigned long start =memblock_region_memory_base_pfn (reg); +unsigned long end =memblock_region_memory_end_pfn (reg); + +if (start < start_pfn) + start = start_pfn;arch/arm64/mm/init.c int pfn_valid(unsigned long pfn) · line 275
Quoted 10 lines
arch/arm64/mm/init.c +10
-0+if (end > start_pfn + size) + end = start_pfn + size; + +if (start < end) { +memmap_init_zone (end - start, nid, zone, start, +MEMMAP_EARLY ,NULL ); + } + } +} +#endif /* CONFIG_SPARSEMEM */ In generic
mmap_init(), the current high cost comes fromearly_pfn_valid()check for each pfn inmemmap_init_zone()given thatearly_pfn_valid()ispfn_valid()whenCONFIG_SPARSEMEMwhich is known to be expensive on arm64.Though we cannot do anything about pfns which are really present but the high cost for non present pfns should be eliminated. The following check in the above
for_each_memblock()loop can achieve that.if (
reg->flags&MEMBLOCK_NOMAP)continue;
Quoted 6 lines
Quoted 3 lines
MEMBLOCK_NOMAPuniversally should not be initialized into a zone and holes if any should also be universally skipped across platforms. So these changes can be moved into genericmemmap_init()which will benefit other platforms.Not sure if I got this. This is a additional short path for memblocks with the
MEMBLOCK_NOMAPflag set, right? For memblocks without these flag the remaining code still needs to be executed?
Right, they should be initialized in the zone.
Quoted 3 lines
In my case I have 4 memblocks with 1.5GB RAM at
0x40000000,0x48000 0000,0x60000 0000 and0x68000 0000. None of them has theMEMBLOCK_NOMAPflag set.
All of them should be added into the zone with memmap_init_zone() because none of them has the flag.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel