support reserving crashkernel above 4G on arm64 kdump
Subsystems (8)
cover support reserving crashkernel above 4G on arm64 kdump
This patch series enable reserving crashkernel on high memory in arm64.
We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G. Then Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.
Besides, we need to modify kexec-tools:
arm64: support more than one crash kernel regions(see [1])
I post this patch series about one month ago. The previous changes and discussions can be retrived from:
Changes since [v4]
- reimplement memblock_cap_memory_ranges for multiple ranges by Mike.
Changes since [v3]
- Add memblock_cap_memory_ranges back for multiple ranges.
- Fix some compiling warnings.
Changes since [v2]
- Split patch "arm64: kdump: support reserving crashkernel above 4G" as two. Put "move reserve_crashkernel_low() into kexec_core.c" in a separate patch.
Changes since [v1]:
- Move common reserve_crashkernel_low() code into kernel/kexec_core.c.
- Remove memblock_cap_memory_ranges() i added in v1 and implement that in fdt_enforce_memory_region().
There are at most two crash kernel regions, for two crash kernel regions case, we cap the memory range [min(regs[*].start), max(regs[*].end)] and then remove the memory range in the middle.
[1]: http://lists.infradead.org/pipermail/kexec/2019-April/022792.html
[v1]: https://lkml.org/lkml/2019/4/2/1174
[v2]: https://lkml.org/lkml/2019/4/9/86
[v3]: https://lkml.org/lkml/2019/4/9/306
[v4]: https://lkml.org/lkml/2019/4/15/273
Chen Zhou (3):
x86: kdump: move reserve_crashkernel_low() into kexec_core.c
arm64: kdump: support reserving crashkernel above 4G
kdump: update Documentation about crashkernel on arm64
Mike Rapoport (1):
memblock: extend memblock_cap_memory_range to multiple ranges
Documentation/admin-guide/kernel-parameters.txt | 6 +-- arch/arm64/include/asm/kexec.h | 3 ++ arch/arm64/kernel/setup.c | 3 ++ arch/arm64/mm/init.c | 72 +++++++++++++++++++------ arch/x86/include/asm/kexec.h | 3 ++ arch/x86/kernel/setup.c | 66 +++-------------------- include/linux/kexec.h | 5 ++ include/linux/memblock.h | 2 +- kernel/kexec_core.c | 56 +++++++++++++++++++ mm/memblock.c | 44 +++++++-------- 10 files changed, 157 insertions(+), 103 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/4 x86: kdump: move reserve_crashkernel_low() into kexec_core.c
In preparation for supporting reserving crashkernel above 4G in arm64 as x86_64 does, move reserve_crashkernel_low() into kexec/kexec_core.c.
--- arch/x86/include/asm/kexec.h | 3 ++ arch/x86/kernel/setup.c | 66 +++++--------------------------------------- include/linux/kexec.h | 5 ++++ kernel/kexec_core.c | 56 +++++++++++++++++++++++++++++++++++++ 4 files changed, 71 insertions(+), 59 deletions(-)
arch/x86/include/asm/kexec.h +3 -0
@@ -18,6 +18,9 @@# define KEXEC_CONTROL_CODE_MAX_SIZE 2048 +/* 16M alignment for crash kernel regions */ +#define CRASH_ALIGN SZ_16M +#ifndef __ASSEMBLY__#include <linux/string.h>
arch/x86/kernel/setup.c +7 -59
84 lines not on this page · raw message
include/linux/kexec.h +5 -0
@@ -63,6 +63,10 @@#define KEXEC_CORE_NOTE_NAME CRASH_CORE_NOTE_NAME +#ifndef CRASH_ALIGN +#define CRASH_ALIGN SZ_128M +#endif +/* * This structure is used to hold the arguments that are used when loading * kernel binaries.
@@ -281,6 +285,7 @@ extern void __crash_kexec(struct pt_regs *);extern void crash_kexec (struct pt_regs *);int kexec_should_crash (struct task_struct *);int kexec_crash_loaded (void ); +int __init reserve_crashkernel_low (void );void crash_save_cpu (struct pt_regs *regs,int cpu);extern int kimage_crash_copy_vmcoreinfo (struct kimage *image);
kernel/kexec_core.c +56 -0
68 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH 1/4] x86: kdump: move reserve_crashkernel_low() into kexec_core.c
Hello,
Quoted 10 lines · Chen Zhou
In preparation for supporting reserving crashkernel above 4G in arm64 as
x86_64does, movereserve_crashkernel_low()intokexec/kexec_core.c.arch/x86/kernel/setup.c +0
-0
arch/x86/kernel/setup.c static void __init memblock_x86_reserve_range_setup_data(void) · line 468
Quoted 6 lines
arch/x86/kernel/setup.c +0
-3@@ -463,59 +460,6 @@ static void __init memblock_x86_reserve_range_setup_data(void)# define CRASH_ADDR_HIGH_MAX MAXMEM#endif -static int __init reserve_crashkernel_low (void ) -{ -#ifdef CONFIG_X86_64
The behaviour of this #ifdef has disappeared, won't 32bit x86 now try and reserve a chunk of unnecessary 'low' memory?
[...]
Quoted 4 lines
arch/x86/kernel/setup.c +0
-0@@ -579,9 +523,13 @@ static void __init reserve_crashkernel(void)return ; }
arch/x86/kernel/setup.c static void __init reserve_crashkernel(void) · line 532
Quoted 10 lines
arch/x86/kernel/setup.c +7
-3-if (crash_base >= (1ULL <<32 ) &&reserve_crashkernel_low ()) { -memblock_free (crash_base, crash_size); -return ; +if (crash_base >= (1ULL <<32 )) { +if (reserve_crashkernel_low ()) { +memblock_free (crash_base, crash_size); +return ; + } + +insert_resource (&iomem_resource, &crashk_low_res);
Previously reserve_crashkernel_low() was #ifdefed to do nothing if !CONFIG_X86_64, I don't see how 32bit is skipping this reservation...
Quoted 8 lines
arch/x86/kernel/setup.c +0
-0} pr_info("Reserving %ldMB of memory at %ldMB for crashkernel (System RAM: %ldMB)\n",include/linux/kexec.h +0
-0
include/linux/kexec.h line 68
Quoted 6 lines
include/linux/kexec.h +3
-0@@ -63,6 +63,10 @@#define KEXEC_CORE_NOTE_NAME CRASH_CORE_NOTE_NAME +#ifndef CRASH_ALIGN +#define CRASH_ALIGN SZ_128M +#endif
Why 128M? Wouldn't we rather each architecture tells us its minimum alignment?
Quoted 29 lines
kernel/kexec_core.c +14
-0@@ -39,6 +39,8 @@#include <linux/compiler.h> #include <linux/hugetlb.h> #include <linux/frame.h> +#include <linux/memblock.h> +#include <linux/swiotlb.h> #include <asm/page.h> #include <asm/sections.h> @@ -96,6 +98,60 @@ int kexec_crash_loaded(void) }EXPORT_SYMBOL_GPL (kexec_crash_loaded); +int __init reserve_crashkernel_low (void ) +{ +unsigned long long base, low_base =0 , low_size =0 ; +unsigned long total_low_mem; +int ret; + + total_low_mem =memblock_mem_size (1UL << (32 -PAGE_SHIFT )); + +/* crashkernel=Y,low */ + ret =parse_crashkernel_low (boot_command_line, total_low_mem, + &low_size, &base); +if (ret) {
kernel/kexec_core.c int kexec_crash_loaded(void) · line 122
Quoted 10 lines
kernel/kexec_core.c +10
-0+/* +* two parts from lib/swiotlb.c: +* -swiotlb size: user-specified with swiotlb= or default. +* +* -swiotlb overflow buffer: now hardcoded to 32k. We round it +* to 8M for other buffers that may need to stay low too. Also +* make sure we allocate enough extra low memory so that we +* don't run out of DMA buffers for 32-bit devices. +*/ + low_size = max(swiotlb_size_or_default() + (8UL << 20),
SZ_8M?
kernel/kexec_core.c int kexec_crash_loaded(void) · line 123
Quoted 1 line
kernel/kexec_core.c +1
-0+ 256UL << 20);
SZ_256M?
Quoted 20 lines
kernel/kexec_core.c +20
-0+ }else { +/* passed with crashkernel=0,low ? */ +if (!low_size) +return 0 ; + } + + low_base =memblock_find_in_range (0 ,1ULL <<32 , low_size,CRASH_ALIGN ); +if (!low_base) { +pr_err ("Cannot reserve %ldMB crashkernel low memory, please try smaller size.\n" , + (unsigned long )(low_size >>20 )); +return -ENOMEM ; + } + + ret =memblock_reserve (low_base, low_size); +if (ret) { +pr_err ("%s: Error reserving crashkernel low memblock.\n" , + __func__); +return ret; + } +
kernel/kexec_core.c int kexec_crash_loaded(void) · line 153
Quoted 10 lines
kernel/kexec_core.c +10
-0+pr_info ("Reserving %ldMB of low memory at %ldMB for crashkernel (System low RAM: %ldMB)\n" , + (unsigned long )(low_size >>20 ), + (unsigned long )(low_base >>20 ), + (unsigned long )(total_low_mem >>20 )); + + crashk_low_res.start = low_base; + crashk_low_res.end = low_base + low_size -1 ; + +return 0 ; +}
Thanks,
James
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH 1/4] x86: kdump: move reserve_crashkernel_low() into kexec_core.c
Hi James,
Thanks for your review.
Quoted 16 lines · James Morse
Hello,
Quoted 13 lines · Chen Zhou
In preparation for supporting reserving crashkernel above 4G in arm64 as
x86_64does, movereserve_crashkernel_low()intokexec/kexec_core.c.arch/x86/kernel/setup.c +0
-0@@ -463,59 +460,6 @@ static void __init memblock_x86_reserve_range_setup_data(void)# define CRASH_ADDR_HIGH_MAX MAXMEM#endif
Quoted 8 lines
arch/x86/kernel/setup.c static void __init memblock_x86_reserve_range_setup_data(void) · line 468
Quoted 3 lines
arch/x86/kernel/setup.c +0
-3-static int __init reserve_crashkernel_low (void ) -{ -#ifdef CONFIG_X86_64 The behaviour of this #ifdef has disappeared, won't 32bit x86 now try and reserve a chunk of unnecessary 'low' memory?
[...]
At present, reserve_crashkernel_low() is called only when reserving crashkernel above 4G, so i deleted this #ifdef.
If we called reserve_crashkernel_low() at the beginning of reserve_crashkernel(), i need to add it back.
Quoted 31 lines
Quoted 6 lines
arch/x86/kernel/setup.c +0
-0@@ -579,9 +523,13 @@ static void __init reserve_crashkernel(void)return ; }arch/x86/kernel/setup.c static void __init reserve_crashkernel(void) · line 532
Quoted 10 lines
arch/x86/kernel/setup.c +7
-3-if (crash_base >= (1ULL <<32 ) &&reserve_crashkernel_low ()) { -memblock_free (crash_base, crash_size); -return ; +if (crash_base >= (1ULL <<32 )) { +if (reserve_crashkernel_low ()) { +memblock_free (crash_base, crash_size); +return ; + } + +insert_resource (&iomem_resource, &crashk_low_res);Previously
reserve_crashkernel_low()was #ifdefed to do nothing if !CONFIG_X86_64, I don't see how 32bit is skipping this reservation...Quoted 9 lines
arch/x86/kernel/setup.c +0
-0} pr_info("Reserving %ldMB of memory at %ldMB for crashkernel (System RAM: %ldMB)\n",include/linux/kexec.h +0
-0@@ -63,6 +63,10 @@
Quoted 7 lines
include/linux/kexec.h line 68
Quoted 5 lines
include/linux/kexec.h +3
-0#define KEXEC_CORE_NOTE_NAME CRASH_CORE_NOTE_NAME +#ifndef CRASH_ALIGN +#define CRASH_ALIGN SZ_128M +#endif Why 128M? Wouldn't we rather each architecture tells us its minimum alignment?
Yeah, each architecture should tells us its minimum alignment. I added this default size to fix compiling error on some architecture which didn't define it. I will add x86_64 and arm64 restriction on reserve_crashkernel_low() and delete this define.
Quoted 44 lines
Quoted 31 lines
kernel/kexec_core.c +14
-0@@ -39,6 +39,8 @@#include <linux/compiler.h> #include <linux/hugetlb.h> #include <linux/frame.h> +#include <linux/memblock.h> +#include <linux/swiotlb.h> #include <asm/page.h> #include <asm/sections.h> @@ -96,6 +98,60 @@ int kexec_crash_loaded(void) }EXPORT_SYMBOL_GPL (kexec_crash_loaded); +int __init reserve_crashkernel_low (void ) +{ +unsigned long long base, low_base =0 , low_size =0 ; +unsigned long total_low_mem; +int ret; + + total_low_mem =memblock_mem_size (1UL << (32 -PAGE_SHIFT )); + +/* crashkernel=Y,low */ + ret =parse_crashkernel_low (boot_command_line, total_low_mem, + &low_size, &base); +if (ret) {kernel/kexec_core.c int kexec_crash_loaded(void) · line 122
Quoted 10 lines
kernel/kexec_core.c +10
-0+/* +* two parts from lib/swiotlb.c: +* -swiotlb size: user-specified with swiotlb= or default. +* +* -swiotlb overflow buffer: now hardcoded to 32k. We round it +* to 8M for other buffers that may need to stay low too. Also +* make sure we allocate enough extra low memory so that we +* don't run out of DMA buffers for 32-bit devices. +*/ + low_size = max(swiotlb_size_or_default() + (8UL << 20),
SZ_8M?
Quoted 4 lines
kernel/kexec_core.c int kexec_crash_loaded(void) · line 123
Quoted 1 line
kernel/kexec_core.c +1
-0+ 256UL << 20);
SZ_256M?
There is compiling warning "warning: comparison of distinct pointer types lacks a cast" if just use SZ_8M or SZ_256M. We need cast swiotlb_size_or_default() to type int,so i kept the old as in x86_64.
Quoted 35 lines
Quoted 21 lines
kernel/kexec_core.c +20
-0+ }else { +/* passed with crashkernel=0,low ? */ +if (!low_size) +return 0 ; + } + + low_base =memblock_find_in_range (0 ,1ULL <<32 , low_size,CRASH_ALIGN ); +if (!low_base) { +pr_err ("Cannot reserve %ldMB crashkernel low memory, please try smaller size.\n" , + (unsigned long )(low_size >>20 )); +return -ENOMEM ; + } + + ret =memblock_reserve (low_base, low_size); +if (ret) { +pr_err ("%s: Error reserving crashkernel low memblock.\n" , + __func__); +return ret; + } +kernel/kexec_core.c int kexec_crash_loaded(void) · line 153
Quoted 10 lines
kernel/kexec_core.c +10
-0+pr_info ("Reserving %ldMB of low memory at %ldMB for crashkernel (System low RAM: %ldMB)\n" , + (unsigned long )(low_size >>20 ), + (unsigned long )(low_base >>20 ), + (unsigned long )(total_low_mem >>20 )); + + crashk_low_res.start = low_base; + crashk_low_res.end = low_base + low_size -1 ; + +return 0 ; +}Thanks,
Quoted 4 lines
James
.
Thanks,
Chen Zhou
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH 1/4] x86: kdump: move reserve_crashkernel_low() into kexec_core.c
In reply to Chen ZhouHi,
Quoted 3 lines · Chen Zhou
In preparation for supporting reserving crashkernel above 4G in arm64 as
x86_64does, movereserve_crashkernel_low()intokexec/kexec_core.c.
Other than the comments from James, can you move the function into kernel/crash_core.c, we already have some functions moved there for sharing.
Thanks
Dave
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH 1/4] x86: kdump: move reserve_crashkernel_low() into kexec_core.c
Hi Dave,
Quoted 3 lines · Dave Young
Other than the comments from James, can you move the function into
kernel/crash_core.c, we already have some functions moved there for sharing.
Sure.
Thanks,
Chen Zhou
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH 2/4 arm64: kdump: support reserving crashkernel above 4G
In reply to Chen ZhouWhen crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers.
Meanwhile, support crashkernel=X,[high,low] in arm64. When use crashkernel=X parameter, try low memory first and fall back to high memory unless "crashkernel=X,high" is specified.
--- arch/arm64/include/asm/kexec.h | 3 +++ arch/arm64/kernel/setup.c | 3 +++ arch/arm64/mm/init.c | 34 ++++++++++++++++++++++++++++------ 3 files changed, 34 insertions(+), 6 deletions(-)
arch/arm64/include/asm/kexec.h +3 -0
@@ -28,6 +28,9 @@#define KEXEC_ARCH KEXEC_ARCH_AARCH64 +/* 2M alignment for crash kernel regions */ +#define CRASH_ALIGN SZ_2M +#ifndef __ASSEMBLY__/**
arch/arm64/kernel/setup.c +3 -0
@@ -243,6 +243,9 @@ static void __init request_standard_resources(void)request_resource (res, &kernel_data);#ifdef CONFIG_KEXEC_CORE /* Userspace will find "Crash kernel" region in /proc/iomem. */ +if (crashk_low_res.end && crashk_low_res.start >= res->start && + crashk_low_res.end <= res->end) +request_resource (res, &crashk_low_res);if (crashk_res.end && crashk_res.start >= res->start && crashk_res.end <= res->end)request_resource (res, &crashk_res);
arch/arm64/mm/init.c +28 -6
@@ -74,20 +74,37 @@ phys_addr_t arm64_dma_phys_limit __ro_after_init;static void __init reserve_crashkernel (void ) {unsigned long long crash_base, crash_size; +bool high = false;int ret; ret =parse_crashkernel (boot_command_line,memblock_phys_mem_size (), &crash_size, &crash_base);/* no crashkernel= or invalid value specified */ -if (ret || !crash_size) -return ; +if (ret || !crash_size) { +/* crashkernel=X,high */ + ret =parse_crashkernel_high (boot_command_line, +memblock_phys_mem_size (), + &crash_size, &crash_base); +if (ret || !crash_size) +return ; + high = true; + } crash_size =PAGE_ALIGN (crash_size);if (crash_base ==0 ) { -/* Current arm64 boot protocol requires 2MB alignment */ - crash_base =memblock_find_in_range (0 ,ARCH_LOW_ADDRESS_LIMIT , - crash_size,SZ_2M ); +/* +* Try low memory first and fall back to high memory +* unless "crashkernel=size[KMG],high" is specified. +*/ +if (!high) + crash_base =memblock_find_in_range (0 , +ARCH_LOW_ADDRESS_LIMIT , + crash_size,CRASH_ALIGN ); +if (!crash_base) + crash_base =memblock_find_in_range (0 , +memblock_end_of_DRAM (), + crash_size,CRASH_ALIGN );if (crash_base ==0 ) {pr_warn ("cannot allocate crashkernel (size:0x%llx)\n" , crash_size);
@@ -105,13 +122,18 @@ static void __init reserve_crashkernel(void)return ; } -if (!IS_ALIGNED (crash_base,SZ_2M )) { +if (!IS_ALIGNED (crash_base,CRASH_ALIGN )) {pr_warn ("cannot reserve crashkernel: base address is not 2MB aligned\n" );return ; } }memblock_reserve (crash_base, crash_size); +if (crash_base >=SZ_4G &&reserve_crashkernel_low ()) { +memblock_free (crash_base, crash_size); +return ; + } +pr_info ("crashkernel reserved: 0x%016llx - 0x%016llx (%lld MB)\n" , crash_base, crash_base + crash_size, crash_size >>20 );
Signature · 1 line
2.7.4
Re: [PATCH 2/4] arm64: kdump: support reserving crashkernel above 4G
Hello,
Quoted 6 lines · Chen Zhou
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers.
Meanwhile, support crashkernel=X,[high,low] in arm64. When use crashkernel=X parameter, try low memory first and fall back to high memory unless "crashkernel=X,high" is specified.
What is the 'unless crashkernel=...,high' for? I think it would be simpler to relax the ARCH_LOW_ADDRESS_LIMIT if reserve_crashkernel_low() allocated something.
This way "crashkernel=1G" tries to allocate 1G below 4G, but fails if there isn't enough memory. "crashkernel=1G crashkernel=16M,low" allocates 16M below 4G, which is more likely to succeed, if it does it can then place the 1G block anywhere.
Quoted 5 lines
arch/arm64/kernel/setup.c +0
-0
arch/arm64/kernel/setup.c static void __init request_standard_resources(void) · line 251
Quoted 9 lines
arch/arm64/kernel/setup.c +3
-0@@ -243,6 +243,9 @@ static void __init request_standard_resources(void)request_resource (res, &kernel_data);#ifdef CONFIG_KEXEC_CORE /* Userspace will find "Crash kernel" region in /proc/iomem. */ +if (crashk_low_res.end && crashk_low_res.start >= res->start && + crashk_low_res.end <= res->end) +request_resource (res, &crashk_low_res);if (crashk_res.end && crashk_res.start >= res->start && crashk_res.end <= res->end)request_resource (res, &crashk_res);
With both crashk_low_res and crashk_res, we end up with two entries in /proc/iomem called "Crash kernel". Because its sorted by address, and kexec-tools stops searching when it find "Crash kernel", you are always going to get the kernel placed in the lower portion.
I suspect this isn't what you want, can we rename crashk_low_res for arm64 so that existing kexec-tools doesn't use it?
Quoted 52 lines
arch/arm64/mm/init.c +22
-6@@ -74,20 +74,37 @@ phys_addr_t arm64_dma_phys_limit __ro_after_init;static void __init reserve_crashkernel (void ) {unsigned long long crash_base, crash_size; +bool high = false;int ret; ret =parse_crashkernel (boot_command_line,memblock_phys_mem_size (), &crash_size, &crash_base);/* no crashkernel= or invalid value specified */ -if (ret || !crash_size) -return ; +if (ret || !crash_size) { +/* crashkernel=X,high */ + ret =parse_crashkernel_high (boot_command_line, +memblock_phys_mem_size (), + &crash_size, &crash_base); +if (ret || !crash_size) +return ; + high = true; + } crash_size =PAGE_ALIGN (crash_size);if (crash_base ==0 ) { -/* Current arm64 boot protocol requires 2MB alignment */ - crash_base =memblock_find_in_range (0 ,ARCH_LOW_ADDRESS_LIMIT , - crash_size,SZ_2M ); +/* +* Try low memory first and fall back to high memory +* unless "crashkernel=size[KMG],high" is specified. +*/ +if (!high) + crash_base =memblock_find_in_range (0 , +ARCH_LOW_ADDRESS_LIMIT , + crash_size,CRASH_ALIGN ); +if (!crash_base) + crash_base =memblock_find_in_range (0 , +memblock_end_of_DRAM (), + crash_size,CRASH_ALIGN );if (crash_base ==0 ) {pr_warn ("cannot allocate crashkernel (size:0x%llx)\n" , crash_size);@@ -105,13 +122,18 @@ static void __init reserve_crashkernel(void)return ; } -if (!IS_ALIGNED (crash_base,SZ_2M )) {
arch/arm64/mm/init.c static void __init reserve_crashkernel(void) · line 134
Quoted 10 lines
arch/arm64/mm/init.c +4
-0+if (!IS_ALIGNED (crash_base,CRASH_ALIGN )) {pr_warn ("cannot reserve crashkernel: base address is not 2MB aligned\n" );return ; } }memblock_reserve (crash_base, crash_size); +if (crash_base >=SZ_4G &&reserve_crashkernel_low ()) { +memblock_free (crash_base, crash_size); +return ;
This is going to be annoying on platforms that don't have, and don't need memory below 4G. A "crashkernel=...,low" on these system will break crashdump. I don't think we should expect users to know the memory layout. (I'm assuming distro's are going to add a low reservation everywhere, just in case)
I think the 'low' region should be a small optional/best-effort extra, that kexec-tools can't touch.
I'm afraid you've missed the ugly bit of the crashkernel reservation...
arch/arm64/mm/mmu.c::map_mem() marks the crashkernel as 'nomap' during the first pass of page-table generation. This means it isn't mapped in the linear map. It then maps it with page-size mappings, and removes the nomap flag.
This is done so that arch_kexec_protect_crashkres() and arch_kexec_unprotect_crashkres() can remove the valid bits of the crashkernel mapping. This way the old-kernel can't accidentally overwrite the crashkernel. It also saves us if the old-kernel and the crashkernel use different memory attributes for the mapping.
As your low-memory reservation is intended to be used for devices, having it mapped by the old-kernel as cacheable memory is going to cause problems if those CPUs aren't taken offline and go corrupting this memory. (we did crash for a reason after all)
I think the simplest thing to do is mark the low region as 'nomap' in reserve_crashkernel() and always leave it unmapped. We can then describe it via a different string in /proc/iomem, something like "Crash kernel (low)". Older kexec-tools shouldn't use it, (I assume its not using strncmp() in a way that would do this by accident), and newer kexec-tools can know to describe it in the DT, but it can't write to it.
Thanks,
James
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH 2/4] arm64: kdump: support reserving crashkernel above 4G
Hi James,
Quoted 10 lines · James Morse
Hello,
Quoted 6 lines · Chen Zhou
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers.
Meanwhile, support crashkernel=X,[high,low] in arm64. When use crashkernel=X parameter, try low memory first and fall back to high memory unless "crashkernel=X,high" is specified.
Quoted 7 lines
What is the 'unless crashkernel=...,high' for? I think it would be simpler to relax the
ARCH_LOW_ADDRESS_LIMITifreserve_crashkernel_low()allocated something.This way "crashkernel=1G" tries to allocate 1G below 4G, but fails if there isn't enough memory. "crashkernel=1G crashkernel=16M,low" allocates 16M below 4G, which is more likely to succeed, if it does it can then place the 1G block anywhere.
Yeah, this is much simpler.
Quoted 20 lines
Quoted 6 lines
arch/arm64/kernel/setup.c +0
-0arch/arm64/kernel/setup.c static void __init request_standard_resources(void) · line 251
Quoted 9 lines
arch/arm64/kernel/setup.c +3
-0@@ -243,6 +243,9 @@ static void __init request_standard_resources(void)request_resource (res, &kernel_data);#ifdef CONFIG_KEXEC_CORE /* Userspace will find "Crash kernel" region in /proc/iomem. */ +if (crashk_low_res.end && crashk_low_res.start >= res->start && + crashk_low_res.end <= res->end) +request_resource (res, &crashk_low_res);if (crashk_res.end && crashk_res.start >= res->start && crashk_res.end <= res->end)request_resource (res, &crashk_res);With both
crashk_low_resandcrashk_res, we end up with two entries in/proc/iomemcalled "Crash kernel". Because its sorted by address, and kexec-tools stops searching when it find "Crash kernel", you are always going to get the kernel placed in the lower portion.
Quoted 3 lines
I suspect this isn't what you want, can we rename
crashk_low_resfor arm64 so that existing kexec-tools doesn't use it?
In my patchset, in addition to the kernel patches, i also modify the kexec-tools.
arm64: support more than one crash kernel regions(http://lists.infradead.org/pipermail/kexec/2019-April/022792.html).
In kexec-tools patch, we read all the "Crash kernel" entry and load crash kernel high.
Quoted 89 lines
Quoted 53 lines
arch/arm64/mm/init.c +22
-6@@ -74,20 +74,37 @@ phys_addr_t arm64_dma_phys_limit __ro_after_init;static void __init reserve_crashkernel (void ) {unsigned long long crash_base, crash_size; +bool high = false;int ret; ret =parse_crashkernel (boot_command_line,memblock_phys_mem_size (), &crash_size, &crash_base);/* no crashkernel= or invalid value specified */ -if (ret || !crash_size) -return ; +if (ret || !crash_size) { +/* crashkernel=X,high */ + ret =parse_crashkernel_high (boot_command_line, +memblock_phys_mem_size (), + &crash_size, &crash_base); +if (ret || !crash_size) +return ; + high = true; + } crash_size =PAGE_ALIGN (crash_size);if (crash_base ==0 ) { -/* Current arm64 boot protocol requires 2MB alignment */ - crash_base =memblock_find_in_range (0 ,ARCH_LOW_ADDRESS_LIMIT , - crash_size,SZ_2M ); +/* +* Try low memory first and fall back to high memory +* unless "crashkernel=size[KMG],high" is specified. +*/ +if (!high) + crash_base =memblock_find_in_range (0 , +ARCH_LOW_ADDRESS_LIMIT , + crash_size,CRASH_ALIGN ); +if (!crash_base) + crash_base =memblock_find_in_range (0 , +memblock_end_of_DRAM (), + crash_size,CRASH_ALIGN );if (crash_base ==0 ) {pr_warn ("cannot allocate crashkernel (size:0x%llx)\n" , crash_size);@@ -105,13 +122,18 @@ static void __init reserve_crashkernel(void)return ; } -if (!IS_ALIGNED (crash_base,SZ_2M )) {arch/arm64/mm/init.c static void __init reserve_crashkernel(void) · line 134
Quoted 10 lines
arch/arm64/mm/init.c +4
-0+if (!IS_ALIGNED (crash_base,CRASH_ALIGN )) {pr_warn ("cannot reserve crashkernel: base address is not 2MB aligned\n" );return ; } }memblock_reserve (crash_base, crash_size); +if (crash_base >=SZ_4G &&reserve_crashkernel_low ()) { +memblock_free (crash_base, crash_size); +return ;This is going to be annoying on platforms that don't have, and don't need memory below 4G. A "crashkernel=...,low" on these system will break crashdump. I don't think we should expect users to know the memory layout. (I'm assuming distro's are going to add a low reservation everywhere, just in case)
I think the 'low' region should be a small optional/best-effort extra, that kexec-tools can't touch.
I'm afraid you've missed the ugly bit of the crashkernel reservation...
arch/arm64/mm/mmu.c::map_mem() marks the crashkernel as 'nomap' during the first pass of page-table generation. This means it isn't mapped in the linear map. It then maps it with page-size mappings, and removes the nomap flag.
This is done so that
arch_kexec_protect_crashkres()andarch_kexec_unprotect_crashkres()can remove the valid bits of the crashkernel mapping. This way the old-kernel can't accidentally overwrite the crashkernel. It also saves us if the old-kernel and the crashkernel use different memory attributes for the mapping.As your low-memory reservation is intended to be used for devices, having it mapped by the old-kernel as cacheable memory is going to cause problems if those CPUs aren't taken offline and go corrupting this memory. (we did crash for a reason after all)
Quoted 6 lines
I think the simplest thing to do is mark the low region as 'nomap' in
reserve_crashkernel()and always leave it unmapped. We can then describe it via a different string in/proc/iomem, something like "Crash kernel (low)". Older kexec-tools shouldn't use it, (I assume its not usingstrncmp()in a way that would do this by accident), and newer kexec-tools can know to describe it in the DT, but it can't write to it.
I did miss the bit of the crashkernel reservation. I will fix this in next version.
Quoted 7 lines
Thanks,
James
.
Thanks,
Chen Zhou
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH 2/4] arm64: kdump: support reserving crashkernel above 4G
Hi Chen Zhou,
Quoted 37 lines · Chen Zhou
Quoted 8 lines · James Morse
Quoted 6 lines · Chen Zhou
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers.
Meanwhile, support crashkernel=X,[high,low] in arm64. When use crashkernel=X parameter, try low memory first and fall back to high memory unless "crashkernel=X,high" is specified.
Quoted 7 lines
What is the 'unless crashkernel=...,high' for? I think it would be simpler to relax the
ARCH_LOW_ADDRESS_LIMITifreserve_crashkernel_low()allocated something.This way "crashkernel=1G" tries to allocate 1G below 4G, but fails if there isn't enough memory. "crashkernel=1G crashkernel=16M,low" allocates 16M below 4G, which is more likely to succeed, if it does it can then place the 1G block anywhere.
Yeah, this is much simpler.
Quoted 19 lines
Quoted 5 lines
arch/arm64/kernel/setup.c +0
-0arch/arm64/kernel/setup.c static void __init request_standard_resources(void) · line 251
Quoted 9 lines
arch/arm64/kernel/setup.c +3
-0@@ -243,6 +243,9 @@ static void __init request_standard_resources(void)request_resource (res, &kernel_data);#ifdef CONFIG_KEXEC_CORE /* Userspace will find "Crash kernel" region in /proc/iomem. */ +if (crashk_low_res.end && crashk_low_res.start >= res->start && + crashk_low_res.end <= res->end) +request_resource (res, &crashk_low_res);if (crashk_res.end && crashk_res.start >= res->start && crashk_res.end <= res->end)request_resource (res, &crashk_res);With both
crashk_low_resandcrashk_res, we end up with two entries in/proc/iomemcalled "Crash kernel". Because its sorted by address, and kexec-tools stops searching when it find "Crash kernel", you are always going to get the kernel placed in the lower portion.
Quoted 6 lines
Quoted 2 lines
I suspect this isn't what you want, can we rename
crashk_low_resfor arm64 so that existing kexec-tools doesn't use it?In my patchset, in addition to the kernel patches, i also modify the kexec-tools.
arm64: support more than one crash kernel regions(http://lists.infradead.org/pipermail/kexec/2019-April/022792.html).
In kexec-tools patch, we read all the "Crash kernel" entry and load crash kernel high.
But we can't rely on people updating user-space when they update the kernel!
[...]
Quoted 16 lines
Quoted 16 lines
I'm afraid you've missed the ugly bit of the crashkernel reservation...
arch/arm64/mm/mmu.c::map_mem() marks the crashkernel as 'nomap' during the first pass of page-table generation. This means it isn't mapped in the linear map. It then maps it with page-size mappings, and removes the nomap flag.
This is done so that
arch_kexec_protect_crashkres()andarch_kexec_unprotect_crashkres()can remove the valid bits of the crashkernel mapping. This way the old-kernel can't accidentally overwrite the crashkernel. It also saves us if the old-kernel and the crashkernel use different memory attributes for the mapping.As your low-memory reservation is intended to be used for devices, having it mapped by the old-kernel as cacheable memory is going to cause problems if those CPUs aren't taken offline and go corrupting this memory. (we did crash for a reason after all)
Quoted 8 lines
Quoted 5 lines
I think the simplest thing to do is mark the low region as 'nomap' in
reserve_crashkernel()and always leave it unmapped. We can then describe it via a different string in/proc/iomem, something like "Crash kernel (low)". Older kexec-tools shouldn't use it, (I assume its not usingstrncmp()in a way that would do this by accident), and newer kexec-tools can know to describe it in the DT, but it can't write to it.I did miss the bit of the crashkernel reservation. I will fix this in next version.
I think all that is needed is to make the low-region nmap, and describe it via /proc/iomem with a name where nothing will try and use it by accident.
Thanks,
James
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH 3/4 memblock: extend memblock_cap_memory_range to multiple ranges
In reply to Chen ZhouFrom: Mike Rapoport <[email protected]>
The memblock_cap_memory_range() removes all the memory except the range passed to it. Extend this function to receive an array of memblock_regions that should be kept. This allows switching to simple iteration over memblock arrays with 'for_each_mem_range_rev' to remove the unneeded memory.
Enable use of this function in arm64 for reservation of multiple regions for the crash kernel.
--- arch/arm64/mm/init.c | 38 ++++++++++++++++++++++++++++---------- include/linux/memblock.h | 2 +- mm/memblock.c | 44 ++++++++++++++++++++------------------------ 3 files changed, 49 insertions(+), 35 deletions(-)
arch/arm64/mm/init.c +28 -10
@@ -63,6 +63,13 @@ EXPORT_SYMBOL(memstart_addr);phys_addr_t arm64_dma_phys_limit__ro_after_init ; +/* The main usage of linux,usable-memory-range is for crash dump kernel. +* Originally, the number of usable-memory regions is one. Now crash dump +* kernel support at most two crash kernel regions, low_region and high +* region. +*/ +#define MAX_USABLE_RANGES 2 +#ifdef CONFIG_KEXEC_CORE /* * reserve_crashkernel() - reserves memory for crash kernel
@@ -302,9 +309,9 @@ early_param("mem", early_mem);static int __init early_init_dt_scan_usablemem (unsigned long node,const char *uname,int depth,void *data) { -struct memblock_region *usablemem = data; -const __be32 *reg; -int len; +struct memblock_type *usablemem = data; +const __be32 *reg, *endp; +int len, nr =0 ;if (depth !=1 ||strcmp (uname,"chosen" ) !=0 )return 0 ;
@@ -313,22 +320,33 @@ static int __init early_init_dt_scan_usablemem(unsigned long node,if (!reg || (len < (dt_root_addr_cells + dt_root_size_cells)))return 1 ; - usablemem->base =dt_mem_next_cell (dt_root_addr_cells, ®); - usablemem->size =dt_mem_next_cell (dt_root_size_cells, ®); + endp = reg + (len /sizeof (__be32)); +while ((endp - reg) >= (dt_root_addr_cells + dt_root_size_cells)) { +unsigned long base =dt_mem_next_cell (dt_root_addr_cells, ®); +unsigned long size =dt_mem_next_cell (dt_root_size_cells, ®); + +if (memblock_add_range (usablemem, base, size,NUMA_NO_NODE , +MEMBLOCK_NONE )) +return 0 ; +if (++nr >=MAX_USABLE_RANGES ) +break ; + }return 1 ; }static void __init fdt_enforce_memory_region (void ) { -struct memblock_region reg = { - .size =0 , +struct memblock_region usable_regions[MAX_USABLE_RANGES ]; +struct memblock_type usablemem = { + .max =MAX_USABLE_RANGES , + .regions = usable_regions, }; -of_scan_flat_dt (early_init_dt_scan_usablemem, ®); +of_scan_flat_dt (early_init_dt_scan_usablemem, &usablemem); -if (reg.size) -memblock_cap_memory_range (reg.base, reg.size); +if (usablemem.cnt) +memblock_cap_memory_ranges (usablemem.regions, usablemem.cnt); }void __init arm64_memblock_init (void )
include/linux/memblock.h +1 -1
@@ -446,7 +446,7 @@ phys_addr_t memblock_mem_size(unsigned long limit_pfn);phys_addr_t memblock_start_of_DRAM (void );phys_addr_t memblock_end_of_DRAM (void );void memblock_enforce_memory_limit (phys_addr_t memory_limit); -void memblock_cap_memory_range (phys_addr_t base,phys_addr_t size); +void memblock_cap_memory_ranges (struct memblock_region *regions,int count);void memblock_mem_limit_remove_map (phys_addr_t limit);bool memblock_is_memory (phys_addr_t addr);bool memblock_is_map_memory (phys_addr_t addr);
mm/memblock.c +20 -24
63 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH 4/4 kdump: update Documentation about crashkernel on arm64
In reply to Chen ZhouNow we support crashkernel=X,[high,low] on arm64, update the Documentation.
--- Documentation/admin-guide/kernel-parameters.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -3
@@ -705,7 +705,7 @@ memory region [offset, offset + size] for that kernel image. If '@offset' is omitted, then a suitable offset is selected automatically. - [KNL, x86_64] select a region under 4G first, and + [KNL, x86_64, arm64] select a region under 4G first, and fall back to reserve region above 4G when '@offset' hasn't been specified. See Documentation/kdump/kdump.txt for further details.
@@ -718,14 +718,14 @@ Documentation/kdump/kdump.txt for an example. crashkernel=size[KMG],high - [KNL, x86_64] range could be above 4G. Allow kernel + [KNL, x86_64, arm64] range could be above 4G. Allow kernel to allocate physical memory region from top, so could be above 4G if system have more than 4G ram installed. Otherwise memory region will be allocated below 4G, if available. It will be ignored if crashkernel=X is specified. crashkernel=size[KMG],low - [KNL, x86_64] range under 4G. When crashkernel=X,high + [KNL, x86_64, arm64] range under 4G. When crashkernel=X,high is passed, kernel could allocate physical memory region above 4G, that cause second kernel crash on system that require some amount of low memory, e.g. swiotlb
Signature · 1 line
2.7.4
Re: [PATCH 4/4] kdump: update Documentation about crashkernel on arm64
Quoted 29 lines · Chen Zhou
Now we support crashkernel=X,[high,low] on arm64, update the Documentation.
Signed-off-by Chen Zhou <[email protected]>--- Documentation/admin-guide/kernel-parameters.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)Documentation/admin-guide/kernel-parameters.txt +2
-2@@ -705,7 +705,7 @@ memory region [offset, offset + size] for that kernel image. If '@offset' is omitted, then a suitable offset is selected automatically. - [KNL, x86_64] select a region under 4G first, and + [KNL, x86_64, arm64] select a region under 4G first, and fall back to reserve region above 4G when '@offset' hasn't been specified. See Documentation/kdump/kdump.txt for further details.@@ -718,14 +718,14 @@ Documentation/kdump/kdump.txt for an example. crashkernel=size[KMG],high - [KNL, x86_64] range could be above 4G. Allow kernel + [KNL, x86_64, arm64] range could be above 4G. Allow kernel to allocate physical memory region from top, so could
Documentation/admin-guide/kernel-parameters.txt line 731
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +1
-1be above 4G if system have more than 4G ram installed. Otherwise memory region will be allocated below 4G, if available. It will be ignored if crashkernel=X is specified. crashkernel=size[KMG],low - [KNL, x86_64] range under 4G. When crashkernel=X,high + [KNL, x86_64, arm64] range under 4G. When crashkernel=X,high is passed, kernel could allocate physical memory region above 4G, that cause second kernel crash on system that require some amount of low memory, e.g. swiotlb
IMO, it is a good time to update 'Documentation/kdump/kdump.txt' with this patchset itself for both x86_64 and arm64, where we still specify only the old format for 'crashkernel' boot-argument:
Section: Boot into System Kernel
=======================
On arm64, use "crashkernel=Y[@X]". Note that the start address of the kernel, X if explicitly specified, must be aligned to 2MiB (0x200000).
...
We can update this to add the new crashkernel=size[KMG],low or crashkernel=size[KMG],high format as well.
Thanks,
Bhupesh
Re: [PATCH 4/4] kdump: update Documentation about crashkernel on arm64
Quoted 49 lines · Bhupesh Sharma
Quoted 28 lines · Chen Zhou
Now we support crashkernel=X,[high,low] on arm64, update the Documentation.
Signed-off-by Chen Zhou <[email protected]>--- Documentation/admin-guide/kernel-parameters.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)Documentation/admin-guide/kernel-parameters.txt +2
-2@@ -705,7 +705,7 @@ memory region [offset, offset + size] for that kernel image. If '@offset' is omitted, then a suitable offset is selected automatically. - [KNL, x86_64] select a region under 4G first, and + [KNL, x86_64, arm64] select a region under 4G first, and fall back to reserve region above 4G when '@offset' hasn't been specified. See Documentation/kdump/kdump.txt for further details.@@ -718,14 +718,14 @@ Documentation/kdump/kdump.txt for an example. crashkernel=size[KMG],high - [KNL, x86_64] range could be above 4G. Allow kernel + [KNL, x86_64, arm64] range could be above 4G. Allow kernel to allocate physical memory region from top, so could be above 4G if system have more than 4G ram installed.Documentation/admin-guide/kernel-parameters.txt line 731
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +1
-1Otherwise memory region will be allocated below 4G, if available. It will be ignored if crashkernel=X is specified. crashkernel=size[KMG],low - [KNL, x86_64] range under 4G. When crashkernel=X,high + [KNL, x86_64, arm64] range under 4G. When crashkernel=X,high is passed, kernel could allocate physical memory region above 4G, that cause second kernel crash on system that require some amount of low memory, e.g. swiotlbIMO, it is a good time to update '
Documentation/kdump/kdump.txt' with this patchset itself for bothx86_64and arm64, where we still specify only the old format for 'crashkernel' boot-argument:Section: Boot into System Kernel
=======================On arm64, use "crashkernel=Y[@X]". Note that the start address of the kernel, X if explicitly specified, must be aligned to 2MiB (
0x200000).
...
Quoted 6 lines
We can update this to add the new crashkernel=size[KMG],low or crashkernel=size[KMG],high format as well.
Thanks,
Bhupesh.
Sure, we can also update here.
Thanks,
Chen Zhou
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
+Cc kexec-list.
Hi Chen,
I think we are still in the quiet period of the merge cycle, but this is a change which will be useful for systems like HPE Apollo where we are looking at reserving crashkernel across a larger range.
Some comments inline and in respective patch threads..
Quoted 1 line · Chen Zhou
This patch series enable reserving crashkernel on high memory in arm64.
Please fix the patch subject, it should be v5.
Also please Cc the kexec-list ([email protected]) for future versions to allow wider review of the patchset.
Quoted 4 lines
We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.
... ^^ required
s/capture kernel will boot failure because of no low memory./capture kernel boot will fail because there is no low memory available for allocation.
Quoted 6 lines
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G. Then Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.
Please use consistent naming for the second kernel, better to use crash dump kernel.
I have tested this on my HPE Apollo machine and with crashkernel=886M,high syntax, I can get the board to reserve a larger memory range for the crashkernel (i.e. 886M):
# dmesg | grep -i crash [ 0.000000] kexec_core: Reserving 256MB of low memory at 3560MB for crashkernel (System low RAM: 2029MB) [ 0.000000] crashkernel reserved: 0x0000000bc5a00000 - 0x0000000bfd000000 (886 MB)
kexec/kdump can also work also work fine on the board.
So, with the changes suggested in this cover letter and individual patches, please feel free to add:
Thanks,
Bhupesh
Quoted 52 lines
Besides, we need to modify kexec-tools:
arm64: support more than one crash kernelregions(see [1])I post this patch series about one month ago. The previous changes and discussions can be retrived from:
Changes since [v4]
- reimplementmemblock_cap_memory_rangesfor multiple ranges by Mike.Changes since [v3]
- Addmemblock_cap_memory_rangesback for multiple ranges.
- Fix some compiling warnings.Changes since [v2]
- Split patch "arm64: kdump: support reserving crashkernel above 4G" as two. Put "movereserve_crashkernel_low()intokexec_core.c" in a separate patch.Changes since [v1]:
- Move commonreserve_crashkernel_low()code intokernel/kexec_core.c.
- Removememblock_cap_memory_ranges()i added in v1 and implement that infdt_enforce_memory_region().
There are at most two crash kernel regions, for two crash kernel regions case, we cap the memory range [min(regs[*].start),max(regs[*].end)] and then remove the memory range in the middle.[1]: http://lists.infradead.org/pipermail/kexec/2019-April/022792.html
[v1]: https://lkml.org/lkml/2019/4/2/1174
[v2]: https://lkml.org/lkml/2019/4/9/86
[v3]: https://lkml.org/lkml/2019/4/9/306
[v4]: https://lkml.org/lkml/2019/4/15/273Chen Zhou (3):
x86: kdump: movereserve_crashkernel_low()intokexec_core.c
arm64: kdump: support reserving crashkernel above 4G
kdump: update Documentation about crashkernel on arm64Mike Rapoport (1):
memblock: extendmemblock_cap_memory_rangeto multiple rangesDocumentation/admin-guide/kernel-parameters.txt | 6 +-- arch/arm64/include/asm/kexec.h | 3 ++ arch/arm64/kernel/setup.c | 3 ++ arch/arm64/mm/init.c | 72 +++++++++++++++++++------ arch/x86/include/asm/kexec.h | 3 ++ arch/x86/kernel/setup.c | 66 +++-------------------- include/linux/kexec.h | 5 ++ include/linux/memblock.h | 2 +- kernel/kexec_core.c | 56 +++++++++++++++++++ mm/memblock.c | 44 +++++++-------- 10 files changed, 157 insertions(+), 103 deletions(-)
Hi Bhupesh,
Quoted 38 lines · Bhupesh Sharma
+Cc kexec-list.
Hi Chen,
I think we are still in the quiet period of the merge cycle, but this is a change which will be useful for systems like HPE Apollo where we are looking at reserving crashkernel across a larger range.
Some comments inline and in respective patch threads..
Quoted 1 line · Chen Zhou
This patch series enable reserving crashkernel on high memory in arm64.
Please fix the patch subject, it should be v5.
Also please Cc the kexec-list ([email protected]) for future versions to allow wider review of the patchset.Quoted 4 lines
We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.
... ^^ required
s/capture kernel will boot failure because of no low memory./capture kernel boot will fail because there is no low memory available for allocation.
Quoted 6 lines
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G. Then Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.Please use consistent naming for the second kernel, better to use crash dump kernel.
I have tested this on my HPE Apollo machine and with crashkernel=886M,high syntax, I can get the board to reserve a larger memory range for the crashkernel (i.e. 886M):
# dmesg | grep -i crash [ 0.000000] kexec_core: Reserving 256MB of low memory at 3560MB for crashkernel (System low RAM: 2029MB) [ 0.000000] crashkernel reserved: 0x0000000bc5a00000 - 0x0000000bfd000000 (886 MB)
Quoted 9 lines
kexec/kdump can also work also work fine on the board.
So, with the changes suggested in this cover letter and individual patches, please feel free to add:
Reviewed-and-Tested-by Bhupesh Sharma <[email protected]>Thanks,
Bhupesh
Thanks for you review and test. I will fix these later.
Thanks,
Chen Zhou
Quoted 54 lines
Quoted 40 lines
Besides, we need to modify kexec-tools:
arm64: support more than one crash kernelregions(see [1])I post this patch series about one month ago. The previous changes and discussions can be retrived from:
Changes since [v4]
- reimplementmemblock_cap_memory_rangesfor multiple ranges by Mike.Changes since [v3]
- Addmemblock_cap_memory_rangesback for multiple ranges.
- Fix some compiling warnings.Changes since [v2]
- Split patch "arm64: kdump: support reserving crashkernel above 4G" as two. Put "movereserve_crashkernel_low()intokexec_core.c" in a separate patch.Changes since [v1]:
- Move commonreserve_crashkernel_low()code intokernel/kexec_core.c.
- Removememblock_cap_memory_ranges()i added in v1 and implement that infdt_enforce_memory_region().
There are at most two crash kernel regions, for two crash kernel regions case, we cap the memory range [min(regs[*].start),max(regs[*].end)] and then remove the memory range in the middle.[1]: http://lists.infradead.org/pipermail/kexec/2019-April/022792.html
[v1]: https://lkml.org/lkml/2019/4/2/1174
[v2]: https://lkml.org/lkml/2019/4/9/86
[v3]: https://lkml.org/lkml/2019/4/9/306
[v4]: https://lkml.org/lkml/2019/4/15/273Chen Zhou (3):
x86: kdump: movereserve_crashkernel_low()intokexec_core.c
arm64: kdump: support reserving crashkernel above 4G
kdump: update Documentation about crashkernel on arm64Mike Rapoport (1):
memblock: extendmemblock_cap_memory_rangeto multiple rangesQuoted 12 lines
Documentation/admin-guide/kernel-parameters.txt | 6 +-- arch/arm64/include/asm/kexec.h | 3 ++ arch/arm64/kernel/setup.c | 3 ++ arch/arm64/mm/init.c | 72 +++++++++++++++++++------ arch/x86/include/asm/kexec.h | 3 ++ arch/x86/kernel/setup.c | 66 +++-------------------- include/linux/kexec.h | 5 ++ include/linux/memblock.h | 2 +- kernel/kexec_core.c | 56 +++++++++++++++++++ mm/memblock.c | 44 +++++++-------- 10 files changed, 157 insertions(+), 103 deletions(-)
Quoted 2 lines
.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hi Catalin,
Sorry to ping you.
What's your suggestion about this patch series? I am looking forward to your replay.
Thanks,
Chen Zhou
Quoted 110 lines · Chen Zhou
Hi Bhupesh,
Quoted 38 lines · Bhupesh Sharma
+Cc kexec-list.
Hi Chen,
I think we are still in the quiet period of the merge cycle, but this is a change which will be useful for systems like HPE Apollo where we are looking at reserving crashkernel across a larger range.
Some comments inline and in respective patch threads..
Quoted 1 line · Chen Zhou
This patch series enable reserving crashkernel on high memory in arm64.
Please fix the patch subject, it should be v5.
Also please Cc the kexec-list ([email protected]) for future versions to allow wider review of the patchset.Quoted 4 lines
We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.
... ^^ required
s/capture kernel will boot failure because of no low memory./capture kernel boot will fail because there is no low memory available for allocation.
Quoted 6 lines
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G. Then Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.Please use consistent naming for the second kernel, better to use crash dump kernel.
I have tested this on my HPE Apollo machine and with crashkernel=886M,high syntax, I can get the board to reserve a larger memory range for the crashkernel (i.e. 886M):
# dmesg | grep -i crash [ 0.000000] kexec_core: Reserving 256MB of low memory at 3560MB for crashkernel (System low RAM: 2029MB) [ 0.000000] crashkernel reserved: 0x0000000bc5a00000 - 0x0000000bfd000000 (886 MB)Quoted 9 lines
kexec/kdump can also work also work fine on the board.
So, with the changes suggested in this cover letter and individual patches, please feel free to add:
Reviewed-and-Tested-by Bhupesh Sharma <[email protected]>Thanks,
BhupeshThanks for you review and test. I will fix these later.
Thanks,
Chen ZhouQuoted 54 lines
Quoted 52 lines
Besides, we need to modify kexec-tools:
arm64: support more than one crash kernelregions(see [1])I post this patch series about one month ago. The previous changes and discussions can be retrived from:
Changes since [v4]
- reimplementmemblock_cap_memory_rangesfor multiple ranges by Mike.Changes since [v3]
- Addmemblock_cap_memory_rangesback for multiple ranges.
- Fix some compiling warnings.Changes since [v2]
- Split patch "arm64: kdump: support reserving crashkernel above 4G" as two. Put "movereserve_crashkernel_low()intokexec_core.c" in a separate patch.Changes since [v1]:
- Move commonreserve_crashkernel_low()code intokernel/kexec_core.c.
- Removememblock_cap_memory_ranges()i added in v1 and implement that infdt_enforce_memory_region().
There are at most two crash kernel regions, for two crash kernel regions case, we cap the memory range [min(regs[*].start),max(regs[*].end)] and then remove the memory range in the middle.[1]: http://lists.infradead.org/pipermail/kexec/2019-April/022792.html
[v1]: https://lkml.org/lkml/2019/4/2/1174
[v2]: https://lkml.org/lkml/2019/4/9/86
[v3]: https://lkml.org/lkml/2019/4/9/306
[v4]: https://lkml.org/lkml/2019/4/15/273Chen Zhou (3):
x86: kdump: movereserve_crashkernel_low()intokexec_core.c
arm64: kdump: support reserving crashkernel above 4G
kdump: update Documentation about crashkernel on arm64Mike Rapoport (1):
memblock: extendmemblock_cap_memory_rangeto multiple rangesDocumentation/admin-guide/kernel-parameters.txt | 6 +-- arch/arm64/include/asm/kexec.h | 3 ++ arch/arm64/kernel/setup.c | 3 ++ arch/arm64/mm/init.c | 72 +++++++++++++++++++------ arch/x86/include/asm/kexec.h | 3 ++ arch/x86/kernel/setup.c | 66 +++-------------------- include/linux/kexec.h | 5 ++ include/linux/memblock.h | 2 +- kernel/kexec_core.c | 56 +++++++++++++++++++ mm/memblock.c | 44 +++++++-------- 10 files changed, 157 insertions(+), 103 deletions(-)
Quoted 6 lines
Quoted 2 lines
.
.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hi!
Quoted 8 lines · Chen Zhou
We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G.
This is a good argument for supporting the 'crashkernel=...,low' version. What is the 'crashkernel=...,high' version for?
Wouldn't it be simpler to relax the ARCH_LOW_ADDRESS_LIMIT if we see 'crashkernel=...,low' in the kernel cmdline?
I don't see what the 'crashkernel=...,high' variant is giving us, it just complicates the flow of reserve_crashkernel().
If we called reserve_crashkernel_low() at the beginning of reserve_crashkernel() we could use crashk_low_res.end to change some limit variable from ARCH_LOW_ADDRESS_LIMIT to memblock_end_of_DRAM().
I think this is a simpler change that gives you what you want.
Quoted 4 lines
Then
Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.
Won't this break if your kdump kernel doesn't know what the extra parameters are? Or if it expects two ranges, but only gets one? These DT properties should be treated as ABI between kernel versions, we can't really change it like this.
I think the 'low' region is an optional-extra, that is never mapped by the first kernel. I think the simplest thing to do is to add an 'linux,low-memory-range' that we memblock_add() after memblock_cap_memory_range() has been called.
If its missing, or the new kernel doesn't know what its for, everything keeps working.
Quoted 5 lines
Besides, we need to modify kexec-tools:
arm64: support more than one crash kernelregions(see [1])I post this patch series about one month ago. The previous changes and discussions can be retrived from:
Ah, this wasn't obvious as you've stopped numbering the series. Please label the next one 'v6' so that we can describe this as 'v5'. (duplicate numbering would be even more confusing!)
Thanks,
James
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 18 lines · James Morse
Hi!
Quoted 8 lines · Chen Zhou
We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G.
This is a good argument for supporting the 'crashkernel=...,low' version. What is the 'crashkernel=...,high' version for?
Wouldn't it be simpler to relax the
ARCH_LOW_ADDRESS_LIMITif we see 'crashkernel=...,low' in the kernel cmdline?
Quoted 7 lines
I don't see what the 'crashkernel=...,high' variant is giving us, it just complicates the flow of
reserve_crashkernel().If we called
reserve_crashkernel_low()at the beginning ofreserve_crashkernel()we could use crashk_low_res.end to change some limit variable fromARCH_LOW_ADDRESS_LIMITtomemblock_end_of_DRAM().
I think this is a simpler change that gives you what you want.
According to your suggestions, we should do like this:
1. call reserve_crashkernel_low() at the beginning of reserve_crashkernel()
2. mark the low region as 'nomap'
3. use crashk_low_res.end to change some limit variable from ARCH_LOW_ADDRESS_LIMIT to memblock_end_of_DRAM()
4. rename crashk_low_res as "Crash kernel (low)" for arm64
5. add an 'linux,low-memory-range' node in DT
Do i understand correctly?
Quoted 17 lines
Quoted 6 lines
Then
Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.Won't this break if your kdump kernel doesn't know what the extra parameters are? Or if it expects two ranges, but only gets one? These DT properties should be treated as ABI between kernel versions, we can't really change it like this.
I think the 'low' region is an optional-extra, that is never mapped by the first kernel. I think the simplest thing to do is to add an 'linux,low-memory-range' that we
memblock_add()aftermemblock_cap_memory_range()has been called.
If its missing, or the new kernel doesn't know what its for, everything keeps working.
Quoted 9 lines
Quoted 5 lines
Besides, we need to modify kexec-tools:
arm64: support more than one crash kernelregions(see [1])I post this patch series about one month ago. The previous changes and discussions can be retrived from:
Ah, this wasn't obvious as you've stopped numbering the series. Please label the next one 'v6' so that we can describe this as 'v5'. (duplicate numbering would be even more confusing!)
ok.
Quoted 7 lines
Thanks,
James
.
Thanks,
Chen Zhou
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hi Chen Zhou,
Quoted 25 lines · Chen Zhou
Quoted 16 lines · James Morse
Quoted 8 lines · Chen Zhou
We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.
When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G.
This is a good argument for supporting the 'crashkernel=...,low' version. What is the 'crashkernel=...,high' version for?
Wouldn't it be simpler to relax the
ARCH_LOW_ADDRESS_LIMITif we see 'crashkernel=...,low' in the kernel cmdline?Quoted 7 lines
I don't see what the 'crashkernel=...,high' variant is giving us, it just complicates the flow of
reserve_crashkernel().If we called
reserve_crashkernel_low()at the beginning ofreserve_crashkernel()we could use crashk_low_res.end to change some limit variable fromARCH_LOW_ADDRESS_LIMITtomemblock_end_of_DRAM().
I think this is a simpler change that gives you what you want.
Quoted 8 lines
According to your suggestions, we should do like this:
1. callreserve_crashkernel_low()at the beginning ofreserve_crashkernel()
2. mark the low region as 'nomap'
3. use crashk_low_res.end to change some limit variable fromARCH_LOW_ADDRESS_LIMITtomemblock_end_of_DRAM()
4. renamecrashk_low_resas "Crash kernel (low)" for arm645. add an 'linux,low-memory-range' node in DT
(This bit would happen in kexec-tools)
Quoted 1 line
Do i understand correctly?
Yes, I think this is simpler and still gives you what you want.
It also leaves the existing behaviour unchanged, which helps with keeping compatibility with existing user-space and older kdump kernels.
Thanks,
James
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec