Skip to contentSkip to conversation

support reserving crashkernel above 4G on arm64 kdump

PATCH In review Chen Zhou 4 participants 20 messages May 7, 2019 – Jun 13, 2019

Subsystems (8)
lore
Share & tools
lore

Chen Zhou

author
#0 lore raw

cover support reserving crashkernel above 4G on arm64 kdump

This patch series enable reserving crashkernel on high memory in arm64.

We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G. Then Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.

Besides, we need to modify kexec-tools:
arm64: support more than one crash kernel regions(see [1])

I post this patch series about one month ago. The previous changes and discussions can be retrived from:

Changes since [v4]
- reimplement memblock_cap_memory_ranges for multiple ranges by Mike.

Changes since [v3]
- Add memblock_cap_memory_ranges back for multiple ranges.
- Fix some compiling warnings.

Changes since [v2]
- Split patch "arm64: kdump: support reserving crashkernel above 4G" as two. Put "move reserve_crashkernel_low() into kexec_core.c" in a separate patch.

Changes since [v1]:
- Move common reserve_crashkernel_low() code into kernel/kexec_core.c.
- Remove memblock_cap_memory_ranges() i added in v1 and implement that in fdt_enforce_memory_region().
There are at most two crash kernel regions, for two crash kernel regions case, we cap the memory range [min(regs[*].start), max(regs[*].end)] and then remove the memory range in the middle.

[1]: http://lists.infradead.org/pipermail/kexec/2019-April/022792.html
[v1]: https://lkml.org/lkml/2019/4/2/1174
[v2]: https://lkml.org/lkml/2019/4/9/86
[v3]: https://lkml.org/lkml/2019/4/9/306
[v4]: https://lkml.org/lkml/2019/4/15/273

Chen Zhou (3):
x86: kdump: move reserve_crashkernel_low() into kexec_core.c
arm64: kdump: support reserving crashkernel above 4G
kdump: update Documentation about crashkernel on arm64

Mike Rapoport (1):
memblock: extend memblock_cap_memory_range to multiple ranges

 Documentation/admin-guide/kernel-parameters.txt |  6 +--
 arch/arm64/include/asm/kexec.h                  |  3 ++
 arch/arm64/kernel/setup.c                       |  3 ++
 arch/arm64/mm/init.c                            | 72 +++++++++++++++++++------
 arch/x86/include/asm/kexec.h                    |  3 ++
 arch/x86/kernel/setup.c                         | 66 +++--------------------
 include/linux/kexec.h                           |  5 ++
 include/linux/memblock.h                        |  2 +-
 kernel/kexec_core.c                             | 56 +++++++++++++++++++
 mm/memblock.c                                   | 44 +++++++--------
 10 files changed, 157 insertions(+), 103 deletions(-)
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH 1/4 x86: kdump: move reserve_crashkernel_low() into kexec_core.c

In preparation for supporting reserving crashkernel above 4G in arm64 as x86_64 does, move reserve_crashkernel_low() into kexec/kexec_core.c.

---
 arch/x86/include/asm/kexec.h |  3 ++
 arch/x86/kernel/setup.c      | 66 +++++---------------------------------------
 include/linux/kexec.h        |  5 ++++
 kernel/kexec_core.c          | 56 +++++++++++++++++++++++++++++++++++++
 4 files changed, 71 insertions(+), 59 deletions(-)
arch/x86/include/asm/kexec.h +3 -0
index 003f2da..c51f293 100644
@@ -18,6 +18,9 @@
 
 # define KEXEC_CONTROL_CODE_MAX_SIZE	2048
 
+/* 16M alignment for crash kernel regions */
+#define CRASH_ALIGN		SZ_16M
+
 #ifndef __ASSEMBLY__
 
 #include <linux/string.h>
arch/x86/kernel/setup.c +7 -59

84 lines not on this page · raw message

include/linux/kexec.h +5 -0
index b9b1bc5..096ad63 100644
@@ -63,6 +63,10 @@
 
 #define KEXEC_CORE_NOTE_NAME	CRASH_CORE_NOTE_NAME
 
+#ifndef CRASH_ALIGN
+#define CRASH_ALIGN SZ_128M
+#endif
+
 /*
  * This structure is used to hold the arguments that are used when loading
  * kernel binaries.
@@ -281,6 +285,7 @@ extern void __crash_kexec(struct pt_regs *);
 extern void crash_kexec(struct pt_regs *);
 int kexec_should_crash(struct task_struct *);
 int kexec_crash_loaded(void);
+int __init reserve_crashkernel_low(void);
 void crash_save_cpu(struct pt_regs *regs, int cpu);
 extern int kimage_crash_copy_vmcoreinfo(struct kimage *image);
 
kernel/kexec_core.c +56 -0

68 lines not on this page · raw message

Signature · 1 line
2.7.4
#2 lore raw

Re: [PATCH 1/4] x86: kdump: move reserve_crashkernel_low() into kexec_core.c

Hello,

Quoted 10 lines · Chen Zhou

In preparation for supporting reserving crashkernel above 4G in arm64 as x86_64 does, move reserve_crashkernel_low() into kexec/kexec_core.c.

arch/x86/kernel/setup.c +0 -0
index 905dae8..9ee33b6 100644

arch/x86/kernel/setup.c static void __init memblock_x86_reserve_range_setup_data(void) · line 468

Quoted 6 lines
arch/x86/kernel/setup.c +0 -3
@@ -463,59 +460,6 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 # define CRASH_ADDR_HIGH_MAX	MAXMEM
 #endif
 
-static int __init reserve_crashkernel_low(void)
-{
-#ifdef CONFIG_X86_64

The behaviour of this #ifdef has disappeared, won't 32bit x86 now try and reserve a chunk of unnecessary 'low' memory?

[...]

Quoted 4 lines
arch/x86/kernel/setup.c +0 -0
@@ -579,9 +523,13 @@ static void __init reserve_crashkernel(void)
 		return;
 	}
 

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void) · line 532

Quoted 10 lines
arch/x86/kernel/setup.c +7 -3
-	if (crash_base >= (1ULL << 32) && reserve_crashkernel_low()) {
-		memblock_free(crash_base, crash_size);
-		return;
+	if (crash_base >= (1ULL << 32)) {
+		if (reserve_crashkernel_low()) {
+			memblock_free(crash_base, crash_size);
+			return;
+		}
+
+		insert_resource(&iomem_resource, &crashk_low_res);

Previously reserve_crashkernel_low() was #ifdefed to do nothing if !CONFIG_X86_64, I don't see how 32bit is skipping this reservation...

Quoted 8 lines
arch/x86/kernel/setup.c +0 -0
 	}
 
 	pr_info("Reserving %ldMB of memory at %ldMB for crashkernel (System RAM: %ldMB)\n",
include/linux/kexec.h +0 -0
index b9b1bc5..096ad63 100644

include/linux/kexec.h line 68

Quoted 6 lines
include/linux/kexec.h +3 -0
@@ -63,6 +63,10 @@
 
 #define KEXEC_CORE_NOTE_NAME	CRASH_CORE_NOTE_NAME
 
+#ifndef CRASH_ALIGN
+#define CRASH_ALIGN SZ_128M
+#endif

Why 128M? Wouldn't we rather each architecture tells us its minimum alignment?

Quoted 29 lines
kernel/kexec_core.c +14 -0
index d714044..3492abd 100644
@@ -39,6 +39,8 @@
 #include <linux/compiler.h>
 #include <linux/hugetlb.h>
 #include <linux/frame.h>
+#include <linux/memblock.h>
+#include <linux/swiotlb.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
@@ -96,6 +98,60 @@ int kexec_crash_loaded(void)
 }
 EXPORT_SYMBOL_GPL(kexec_crash_loaded);
 
+int __init reserve_crashkernel_low(void)
+{
+	unsigned long long base, low_base = 0, low_size = 0;
+	unsigned long total_low_mem;
+	int ret;
+
+	total_low_mem = memblock_mem_size(1UL << (32 - PAGE_SHIFT));
+
+	/* crashkernel=Y,low */
+	ret = parse_crashkernel_low(boot_command_line, total_low_mem,
+			&low_size, &base);
+	if (ret) {

kernel/kexec_core.c int kexec_crash_loaded(void) · line 122

Quoted 10 lines
kernel/kexec_core.c +10 -0
+		/*
+		 * two parts from lib/swiotlb.c:
+		 * -swiotlb size: user-specified with swiotlb= or default.
+		 *
+		 * -swiotlb overflow buffer: now hardcoded to 32k. We round it
+		 * to 8M for other buffers that may need to stay low too. Also
+		 * make sure we allocate enough extra low memory so that we
+		 * don't run out of DMA buffers for 32-bit devices.
+		 */
+		low_size = max(swiotlb_size_or_default() + (8UL << 20),

SZ_8M?

kernel/kexec_core.c int kexec_crash_loaded(void) · line 123

Quoted 1 line
kernel/kexec_core.c +1 -0
+				256UL << 20);

SZ_256M?

Quoted 20 lines
kernel/kexec_core.c +20 -0
+	} else {
+		/* passed with crashkernel=0,low ? */
+		if (!low_size)
+			return 0;
+	}
+
+	low_base = memblock_find_in_range(0, 1ULL << 32, low_size, CRASH_ALIGN);
+	if (!low_base) {
+		pr_err("Cannot reserve %ldMB crashkernel low memory, please try smaller size.\n",
+		       (unsigned long)(low_size >> 20));
+		return -ENOMEM;
+	}
+
+	ret = memblock_reserve(low_base, low_size);
+	if (ret) {
+		pr_err("%s: Error reserving crashkernel low memblock.\n",
+				__func__);
+		return ret;
+	}
+

kernel/kexec_core.c int kexec_crash_loaded(void) · line 153

Quoted 10 lines
kernel/kexec_core.c +10 -0
+	pr_info("Reserving %ldMB of low memory at %ldMB for crashkernel (System low RAM: %ldMB)\n",
+		(unsigned long)(low_size >> 20),
+		(unsigned long)(low_base >> 20),
+		(unsigned long)(total_low_mem >> 20));
+
+	crashk_low_res.start = low_base;
+	crashk_low_res.end   = low_base + low_size - 1;
+
+	return 0;
+}

Thanks,

James

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#3 lore raw

Re: [PATCH 1/4] x86: kdump: move reserve_crashkernel_low() into kexec_core.c

Hi James,

Thanks for your review.

Quoted 16 lines · James Morse

Hello,

Quoted 13 lines · Chen Zhou

In preparation for supporting reserving crashkernel above 4G in arm64 as x86_64 does, move reserve_crashkernel_low() into kexec/kexec_core.c.

arch/x86/kernel/setup.c +0 -0
index 905dae8..9ee33b6 100644
@@ -463,59 +460,6 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 # define CRASH_ADDR_HIGH_MAX	MAXMEM
 #endif
 
Quoted 8 lines

arch/x86/kernel/setup.c static void __init memblock_x86_reserve_range_setup_data(void) · line 468

Quoted 3 lines
arch/x86/kernel/setup.c +0 -3
-static int __init reserve_crashkernel_low(void)
-{
-#ifdef CONFIG_X86_64

The behaviour of this #ifdef has disappeared, won't 32bit x86 now try and reserve a chunk of unnecessary 'low' memory?

[...]

At present, reserve_crashkernel_low() is called only when reserving crashkernel above 4G, so i deleted this #ifdef.
If we called reserve_crashkernel_low() at the beginning of reserve_crashkernel(), i need to add it back.

Quoted 31 lines
Quoted 6 lines
arch/x86/kernel/setup.c +0 -0
@@ -579,9 +523,13 @@ static void __init reserve_crashkernel(void)
 		return;
 	}
 

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void) · line 532

Quoted 10 lines
arch/x86/kernel/setup.c +7 -3
-	if (crash_base >= (1ULL << 32) && reserve_crashkernel_low()) {
-		memblock_free(crash_base, crash_size);
-		return;
+	if (crash_base >= (1ULL << 32)) {
+		if (reserve_crashkernel_low()) {
+			memblock_free(crash_base, crash_size);
+			return;
+		}
+
+		insert_resource(&iomem_resource, &crashk_low_res);

Previously reserve_crashkernel_low() was #ifdefed to do nothing if !CONFIG_X86_64, I don't see how 32bit is skipping this reservation...

Quoted 9 lines
arch/x86/kernel/setup.c +0 -0
 	}
 
 	pr_info("Reserving %ldMB of memory at %ldMB for crashkernel (System RAM: %ldMB)\n",
include/linux/kexec.h +0 -0
index b9b1bc5..096ad63 100644
@@ -63,6 +63,10 @@
 
Quoted 7 lines

include/linux/kexec.h line 68

Quoted 5 lines
include/linux/kexec.h +3 -0
 #define KEXEC_CORE_NOTE_NAME	CRASH_CORE_NOTE_NAME
 
+#ifndef CRASH_ALIGN
+#define CRASH_ALIGN SZ_128M
+#endif

Why 128M? Wouldn't we rather each architecture tells us its minimum alignment?

Yeah, each architecture should tells us its minimum alignment. I added this default size to fix compiling error on some architecture which didn't define it. I will add x86_64 and arm64 restriction on reserve_crashkernel_low() and delete this define.

Quoted 44 lines
Quoted 31 lines
kernel/kexec_core.c +14 -0
index d714044..3492abd 100644
@@ -39,6 +39,8 @@
 #include <linux/compiler.h>
 #include <linux/hugetlb.h>
 #include <linux/frame.h>
+#include <linux/memblock.h>
+#include <linux/swiotlb.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
@@ -96,6 +98,60 @@ int kexec_crash_loaded(void)
 }
 EXPORT_SYMBOL_GPL(kexec_crash_loaded);
 
+int __init reserve_crashkernel_low(void)
+{
+	unsigned long long base, low_base = 0, low_size = 0;
+	unsigned long total_low_mem;
+	int ret;
+
+	total_low_mem = memblock_mem_size(1UL << (32 - PAGE_SHIFT));
+
+	/* crashkernel=Y,low */
+	ret = parse_crashkernel_low(boot_command_line, total_low_mem,
+			&low_size, &base);
+	if (ret) {

kernel/kexec_core.c int kexec_crash_loaded(void) · line 122

Quoted 10 lines
kernel/kexec_core.c +10 -0
+		/*
+		 * two parts from lib/swiotlb.c:
+		 * -swiotlb size: user-specified with swiotlb= or default.
+		 *
+		 * -swiotlb overflow buffer: now hardcoded to 32k. We round it
+		 * to 8M for other buffers that may need to stay low too. Also
+		 * make sure we allocate enough extra low memory so that we
+		 * don't run out of DMA buffers for 32-bit devices.
+		 */
+		low_size = max(swiotlb_size_or_default() + (8UL << 20),

SZ_8M?

Quoted 4 lines

kernel/kexec_core.c int kexec_crash_loaded(void) · line 123

Quoted 1 line
kernel/kexec_core.c +1 -0
+				256UL << 20);

SZ_256M?

There is compiling warning "warning: comparison of distinct pointer types lacks a cast" if just use SZ_8M or SZ_256M. We need cast swiotlb_size_or_default() to type int,so i kept the old as in x86_64.

Quoted 35 lines
Quoted 21 lines
kernel/kexec_core.c +20 -0
+	} else {
+		/* passed with crashkernel=0,low ? */
+		if (!low_size)
+			return 0;
+	}
+
+	low_base = memblock_find_in_range(0, 1ULL << 32, low_size, CRASH_ALIGN);
+	if (!low_base) {
+		pr_err("Cannot reserve %ldMB crashkernel low memory, please try smaller size.\n",
+		       (unsigned long)(low_size >> 20));
+		return -ENOMEM;
+	}
+
+	ret = memblock_reserve(low_base, low_size);
+	if (ret) {
+		pr_err("%s: Error reserving crashkernel low memblock.\n",
+				__func__);
+		return ret;
+	}
+

kernel/kexec_core.c int kexec_crash_loaded(void) · line 153

Quoted 10 lines
kernel/kexec_core.c +10 -0
+	pr_info("Reserving %ldMB of low memory at %ldMB for crashkernel (System low RAM: %ldMB)\n",
+		(unsigned long)(low_size >> 20),
+		(unsigned long)(low_base >> 20),
+		(unsigned long)(total_low_mem >> 20));
+
+	crashk_low_res.start = low_base;
+	crashk_low_res.end   = low_base + low_size - 1;
+
+	return 0;
+}

Thanks,

Quoted 4 lines

James

.

Thanks,
Chen Zhou

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw

Re: [PATCH 1/4] x86: kdump: move reserve_crashkernel_low() into kexec_core.c

In reply to Chen Zhou

Hi,

Quoted 3 lines · Chen Zhou

In preparation for supporting reserving crashkernel above 4G in arm64 as x86_64 does, move reserve_crashkernel_low() into kexec/kexec_core.c.

Other than the comments from James, can you move the function into kernel/crash_core.c, we already have some functions moved there for sharing.

Thanks
Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#5 lore raw

Re: [PATCH 1/4] x86: kdump: move reserve_crashkernel_low() into kexec_core.c

Hi Dave,

Quoted 3 lines · Dave Young

Other than the comments from James, can you move the function into kernel/crash_core.c, we already have some functions moved there for sharing.

Sure.

Thanks,
Chen Zhou

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#6 lore raw

PATCH 2/4 arm64: kdump: support reserving crashkernel above 4G

In reply to Chen Zhou

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers.

Meanwhile, support crashkernel=X,[high,low] in arm64. When use crashkernel=X parameter, try low memory first and fall back to high memory unless "crashkernel=X,high" is specified.

---
 arch/arm64/include/asm/kexec.h |  3 +++
 arch/arm64/kernel/setup.c      |  3 +++
 arch/arm64/mm/init.c           | 34 ++++++++++++++++++++++++++++------
 3 files changed, 34 insertions(+), 6 deletions(-)
arch/arm64/include/asm/kexec.h +3 -0
index 67e4cb7..32949bf 100644
@@ -28,6 +28,9 @@
 
 #define KEXEC_ARCH KEXEC_ARCH_AARCH64
 
+/* 2M alignment for crash kernel regions */
+#define CRASH_ALIGN	SZ_2M
+
 #ifndef __ASSEMBLY__
 
 /**
arch/arm64/kernel/setup.c +3 -0
index 413d566..82cd9a0 100644
@@ -243,6 +243,9 @@ static void __init request_standard_resources(void)
 			request_resource(res, &kernel_data);
 #ifdef CONFIG_KEXEC_CORE
 		/* Userspace will find "Crash kernel" region in /proc/iomem. */
+		if (crashk_low_res.end && crashk_low_res.start >= res->start &&
+		    crashk_low_res.end <= res->end)
+			request_resource(res, &crashk_low_res);
 		if (crashk_res.end && crashk_res.start >= res->start &&
 		    crashk_res.end <= res->end)
 			request_resource(res, &crashk_res);
arch/arm64/mm/init.c +28 -6
index d2adffb..3fcd739 100644
@@ -74,20 +74,37 @@ phys_addr_t arm64_dma_phys_limit __ro_after_init;
 static void __init reserve_crashkernel(void)
 {
 	unsigned long long crash_base, crash_size;
+	bool high = false;
 	int ret;
 
 	ret = parse_crashkernel(boot_command_line, memblock_phys_mem_size(),
 				&crash_size, &crash_base);
 	/* no crashkernel= or invalid value specified */
-	if (ret || !crash_size)
-		return;
+	if (ret || !crash_size) {
+		/* crashkernel=X,high */
+		ret = parse_crashkernel_high(boot_command_line,
+				memblock_phys_mem_size(),
+				&crash_size, &crash_base);
+		if (ret || !crash_size)
+			return;
+		high = true;
+	}
 
 	crash_size = PAGE_ALIGN(crash_size);
 
 	if (crash_base == 0) {
-		/* Current arm64 boot protocol requires 2MB alignment */
-		crash_base = memblock_find_in_range(0, ARCH_LOW_ADDRESS_LIMIT,
-				crash_size, SZ_2M);
+		/*
+		 * Try low memory first and fall back to high memory
+		 * unless "crashkernel=size[KMG],high" is specified.
+		 */
+		if (!high)
+			crash_base = memblock_find_in_range(0,
+					ARCH_LOW_ADDRESS_LIMIT,
+					crash_size, CRASH_ALIGN);
+		if (!crash_base)
+			crash_base = memblock_find_in_range(0,
+					memblock_end_of_DRAM(),
+					crash_size, CRASH_ALIGN);
 		if (crash_base == 0) {
 			pr_warn("cannot allocate crashkernel (size:0x%llx)\n",
 				crash_size);
@@ -105,13 +122,18 @@ static void __init reserve_crashkernel(void)
 			return;
 		}
 
-		if (!IS_ALIGNED(crash_base, SZ_2M)) {
+		if (!IS_ALIGNED(crash_base, CRASH_ALIGN)) {
 			pr_warn("cannot reserve crashkernel: base address is not 2MB aligned\n");
 			return;
 		}
 	}
 	memblock_reserve(crash_base, crash_size);
 
+	if (crash_base >= SZ_4G && reserve_crashkernel_low()) {
+		memblock_free(crash_base, crash_size);
+		return;
+	}
+
 	pr_info("crashkernel reserved: 0x%016llx - 0x%016llx (%lld MB)\n",
 		crash_base, crash_base + crash_size, crash_size >> 20);
 
Signature · 1 line
2.7.4
#7 lore raw

Re: [PATCH 2/4] arm64: kdump: support reserving crashkernel above 4G

Hello,

Quoted 6 lines · Chen Zhou

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers.

Meanwhile, support crashkernel=X,[high,low] in arm64. When use crashkernel=X parameter, try low memory first and fall back to high memory unless "crashkernel=X,high" is specified.

What is the 'unless crashkernel=...,high' for? I think it would be simpler to relax the ARCH_LOW_ADDRESS_LIMIT if reserve_crashkernel_low() allocated something.

This way "crashkernel=1G" tries to allocate 1G below 4G, but fails if there isn't enough memory. "crashkernel=1G crashkernel=16M,low" allocates 16M below 4G, which is more likely to succeed, if it does it can then place the 1G block anywhere.

Quoted 5 lines
arch/arm64/kernel/setup.c +0 -0
index 413d566..82cd9a0 100644

arch/arm64/kernel/setup.c static void __init request_standard_resources(void) · line 251

Quoted 9 lines
arch/arm64/kernel/setup.c +3 -0
@@ -243,6 +243,9 @@ static void __init request_standard_resources(void)
 			request_resource(res, &kernel_data);
 #ifdef CONFIG_KEXEC_CORE
 		/* Userspace will find "Crash kernel" region in /proc/iomem. */
+		if (crashk_low_res.end && crashk_low_res.start >= res->start &&
+		    crashk_low_res.end <= res->end)
+			request_resource(res, &crashk_low_res);
 		if (crashk_res.end && crashk_res.start >= res->start &&
 		    crashk_res.end <= res->end)
 			request_resource(res, &crashk_res);

With both crashk_low_res and crashk_res, we end up with two entries in /proc/iomem called "Crash kernel". Because its sorted by address, and kexec-tools stops searching when it find "Crash kernel", you are always going to get the kernel placed in the lower portion.

I suspect this isn't what you want, can we rename crashk_low_res for arm64 so that existing kexec-tools doesn't use it?

Quoted 52 lines
arch/arm64/mm/init.c +22 -6
index d2adffb..3fcd739 100644
@@ -74,20 +74,37 @@ phys_addr_t arm64_dma_phys_limit __ro_after_init;
 static void __init reserve_crashkernel(void)
 {
 	unsigned long long crash_base, crash_size;
+	bool high = false;
 	int ret;
 
 	ret = parse_crashkernel(boot_command_line, memblock_phys_mem_size(),
 				&crash_size, &crash_base);
 	/* no crashkernel= or invalid value specified */
-	if (ret || !crash_size)
-		return;
+	if (ret || !crash_size) {
+		/* crashkernel=X,high */
+		ret = parse_crashkernel_high(boot_command_line,
+				memblock_phys_mem_size(),
+				&crash_size, &crash_base);
+		if (ret || !crash_size)
+			return;
+		high = true;
+	}
 
 	crash_size = PAGE_ALIGN(crash_size);
 
 	if (crash_base == 0) {
-		/* Current arm64 boot protocol requires 2MB alignment */
-		crash_base = memblock_find_in_range(0, ARCH_LOW_ADDRESS_LIMIT,
-				crash_size, SZ_2M);
+		/*
+		 * Try low memory first and fall back to high memory
+		 * unless "crashkernel=size[KMG],high" is specified.
+		 */
+		if (!high)
+			crash_base = memblock_find_in_range(0,
+					ARCH_LOW_ADDRESS_LIMIT,
+					crash_size, CRASH_ALIGN);
+		if (!crash_base)
+			crash_base = memblock_find_in_range(0,
+					memblock_end_of_DRAM(),
+					crash_size, CRASH_ALIGN);
 		if (crash_base == 0) {
 			pr_warn("cannot allocate crashkernel (size:0x%llx)\n",
 				crash_size);
@@ -105,13 +122,18 @@ static void __init reserve_crashkernel(void)
 			return;
 		}
 
-		if (!IS_ALIGNED(crash_base, SZ_2M)) {

arch/arm64/mm/init.c static void __init reserve_crashkernel(void) · line 134

Quoted 10 lines
arch/arm64/mm/init.c +4 -0
+		if (!IS_ALIGNED(crash_base, CRASH_ALIGN)) {
 			pr_warn("cannot reserve crashkernel: base address is not 2MB aligned\n");
 			return;
 		}
 	}
 	memblock_reserve(crash_base, crash_size);
 
+	if (crash_base >= SZ_4G && reserve_crashkernel_low()) {
+		memblock_free(crash_base, crash_size);
+		return;

This is going to be annoying on platforms that don't have, and don't need memory below 4G. A "crashkernel=...,low" on these system will break crashdump. I don't think we should expect users to know the memory layout. (I'm assuming distro's are going to add a low reservation everywhere, just in case)

I think the 'low' region should be a small optional/best-effort extra, that kexec-tools can't touch.

I'm afraid you've missed the ugly bit of the crashkernel reservation...

arch/arm64/mm/mmu.c::map_mem() marks the crashkernel as 'nomap' during the first pass of page-table generation. This means it isn't mapped in the linear map. It then maps it with page-size mappings, and removes the nomap flag.

This is done so that arch_kexec_protect_crashkres() and arch_kexec_unprotect_crashkres() can remove the valid bits of the crashkernel mapping. This way the old-kernel can't accidentally overwrite the crashkernel. It also saves us if the old-kernel and the crashkernel use different memory attributes for the mapping.

As your low-memory reservation is intended to be used for devices, having it mapped by the old-kernel as cacheable memory is going to cause problems if those CPUs aren't taken offline and go corrupting this memory. (we did crash for a reason after all)

I think the simplest thing to do is mark the low region as 'nomap' in reserve_crashkernel() and always leave it unmapped. We can then describe it via a different string in /proc/iomem, something like "Crash kernel (low)". Older kexec-tools shouldn't use it, (I assume its not using strncmp() in a way that would do this by accident), and newer kexec-tools can know to describe it in the DT, but it can't write to it.

Thanks,

James

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#8 lore raw

Re: [PATCH 2/4] arm64: kdump: support reserving crashkernel above 4G

Hi James,

Quoted 10 lines · James Morse

Hello,

Quoted 6 lines · Chen Zhou

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers.

Meanwhile, support crashkernel=X,[high,low] in arm64. When use crashkernel=X parameter, try low memory first and fall back to high memory unless "crashkernel=X,high" is specified.

Quoted 7 lines

What is the 'unless crashkernel=...,high' for? I think it would be simpler to relax the ARCH_LOW_ADDRESS_LIMIT if reserve_crashkernel_low() allocated something.

This way "crashkernel=1G" tries to allocate 1G below 4G, but fails if there isn't enough memory. "crashkernel=1G crashkernel=16M,low" allocates 16M below 4G, which is more likely to succeed, if it does it can then place the 1G block anywhere.

Yeah, this is much simpler.

Quoted 20 lines
Quoted 6 lines
arch/arm64/kernel/setup.c +0 -0
index 413d566..82cd9a0 100644

arch/arm64/kernel/setup.c static void __init request_standard_resources(void) · line 251

Quoted 9 lines
arch/arm64/kernel/setup.c +3 -0
@@ -243,6 +243,9 @@ static void __init request_standard_resources(void)
 			request_resource(res, &kernel_data);
 #ifdef CONFIG_KEXEC_CORE
 		/* Userspace will find "Crash kernel" region in /proc/iomem. */
+		if (crashk_low_res.end && crashk_low_res.start >= res->start &&
+		    crashk_low_res.end <= res->end)
+			request_resource(res, &crashk_low_res);
 		if (crashk_res.end && crashk_res.start >= res->start &&
 		    crashk_res.end <= res->end)
 			request_resource(res, &crashk_res);

With both crashk_low_res and crashk_res, we end up with two entries in /proc/iomem called "Crash kernel". Because its sorted by address, and kexec-tools stops searching when it find "Crash kernel", you are always going to get the kernel placed in the lower portion.

Quoted 3 lines

I suspect this isn't what you want, can we rename crashk_low_res for arm64 so that existing kexec-tools doesn't use it?

In my patchset, in addition to the kernel patches, i also modify the kexec-tools.
arm64: support more than one crash kernel regions(http://lists.infradead.org/pipermail/kexec/2019-April/022792.html).
In kexec-tools patch, we read all the "Crash kernel" entry and load crash kernel high.

Quoted 89 lines
Quoted 53 lines
arch/arm64/mm/init.c +22 -6
index d2adffb..3fcd739 100644
@@ -74,20 +74,37 @@ phys_addr_t arm64_dma_phys_limit __ro_after_init;
 static void __init reserve_crashkernel(void)
 {
 	unsigned long long crash_base, crash_size;
+	bool high = false;
 	int ret;
 
 	ret = parse_crashkernel(boot_command_line, memblock_phys_mem_size(),
 				&crash_size, &crash_base);
 	/* no crashkernel= or invalid value specified */
-	if (ret || !crash_size)
-		return;
+	if (ret || !crash_size) {
+		/* crashkernel=X,high */
+		ret = parse_crashkernel_high(boot_command_line,
+				memblock_phys_mem_size(),
+				&crash_size, &crash_base);
+		if (ret || !crash_size)
+			return;
+		high = true;
+	}
 
 	crash_size = PAGE_ALIGN(crash_size);
 
 	if (crash_base == 0) {
-		/* Current arm64 boot protocol requires 2MB alignment */
-		crash_base = memblock_find_in_range(0, ARCH_LOW_ADDRESS_LIMIT,
-				crash_size, SZ_2M);
+		/*
+		 * Try low memory first and fall back to high memory
+		 * unless "crashkernel=size[KMG],high" is specified.
+		 */
+		if (!high)
+			crash_base = memblock_find_in_range(0,
+					ARCH_LOW_ADDRESS_LIMIT,
+					crash_size, CRASH_ALIGN);
+		if (!crash_base)
+			crash_base = memblock_find_in_range(0,
+					memblock_end_of_DRAM(),
+					crash_size, CRASH_ALIGN);
 		if (crash_base == 0) {
 			pr_warn("cannot allocate crashkernel (size:0x%llx)\n",
 				crash_size);
@@ -105,13 +122,18 @@ static void __init reserve_crashkernel(void)
 			return;
 		}
 
-		if (!IS_ALIGNED(crash_base, SZ_2M)) {

arch/arm64/mm/init.c static void __init reserve_crashkernel(void) · line 134

Quoted 10 lines
arch/arm64/mm/init.c +4 -0
+		if (!IS_ALIGNED(crash_base, CRASH_ALIGN)) {
 			pr_warn("cannot reserve crashkernel: base address is not 2MB aligned\n");
 			return;
 		}
 	}
 	memblock_reserve(crash_base, crash_size);
 
+	if (crash_base >= SZ_4G && reserve_crashkernel_low()) {
+		memblock_free(crash_base, crash_size);
+		return;

This is going to be annoying on platforms that don't have, and don't need memory below 4G. A "crashkernel=...,low" on these system will break crashdump. I don't think we should expect users to know the memory layout. (I'm assuming distro's are going to add a low reservation everywhere, just in case)

I think the 'low' region should be a small optional/best-effort extra, that kexec-tools can't touch.

I'm afraid you've missed the ugly bit of the crashkernel reservation...

arch/arm64/mm/mmu.c::map_mem() marks the crashkernel as 'nomap' during the first pass of page-table generation. This means it isn't mapped in the linear map. It then maps it with page-size mappings, and removes the nomap flag.

This is done so that arch_kexec_protect_crashkres() and arch_kexec_unprotect_crashkres() can remove the valid bits of the crashkernel mapping. This way the old-kernel can't accidentally overwrite the crashkernel. It also saves us if the old-kernel and the crashkernel use different memory attributes for the mapping.

As your low-memory reservation is intended to be used for devices, having it mapped by the old-kernel as cacheable memory is going to cause problems if those CPUs aren't taken offline and go corrupting this memory. (we did crash for a reason after all)

Quoted 6 lines

I think the simplest thing to do is mark the low region as 'nomap' in reserve_crashkernel() and always leave it unmapped. We can then describe it via a different string in /proc/iomem, something like "Crash kernel (low)". Older kexec-tools shouldn't use it, (I assume its not using strncmp() in a way that would do this by accident), and newer kexec-tools can know to describe it in the DT, but it can't write to it.

I did miss the bit of the crashkernel reservation. I will fix this in next version.

Quoted 7 lines

Thanks,

James

.

Thanks,
Chen Zhou

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw

Re: [PATCH 2/4] arm64: kdump: support reserving crashkernel above 4G

Hi Chen Zhou,

Quoted 37 lines · Chen Zhou
Quoted 8 lines · James Morse
Quoted 6 lines · Chen Zhou

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers.

Meanwhile, support crashkernel=X,[high,low] in arm64. When use crashkernel=X parameter, try low memory first and fall back to high memory unless "crashkernel=X,high" is specified.

Quoted 7 lines

What is the 'unless crashkernel=...,high' for? I think it would be simpler to relax the ARCH_LOW_ADDRESS_LIMIT if reserve_crashkernel_low() allocated something.

This way "crashkernel=1G" tries to allocate 1G below 4G, but fails if there isn't enough memory. "crashkernel=1G crashkernel=16M,low" allocates 16M below 4G, which is more likely to succeed, if it does it can then place the 1G block anywhere.

Yeah, this is much simpler.

Quoted 19 lines
Quoted 5 lines
arch/arm64/kernel/setup.c +0 -0
index 413d566..82cd9a0 100644

arch/arm64/kernel/setup.c static void __init request_standard_resources(void) · line 251

Quoted 9 lines
arch/arm64/kernel/setup.c +3 -0
@@ -243,6 +243,9 @@ static void __init request_standard_resources(void)
 			request_resource(res, &kernel_data);
 #ifdef CONFIG_KEXEC_CORE
 		/* Userspace will find "Crash kernel" region in /proc/iomem. */
+		if (crashk_low_res.end && crashk_low_res.start >= res->start &&
+		    crashk_low_res.end <= res->end)
+			request_resource(res, &crashk_low_res);
 		if (crashk_res.end && crashk_res.start >= res->start &&
 		    crashk_res.end <= res->end)
 			request_resource(res, &crashk_res);

With both crashk_low_res and crashk_res, we end up with two entries in /proc/iomem called "Crash kernel". Because its sorted by address, and kexec-tools stops searching when it find "Crash kernel", you are always going to get the kernel placed in the lower portion.

Quoted 6 lines
Quoted 2 lines

I suspect this isn't what you want, can we rename crashk_low_res for arm64 so that existing kexec-tools doesn't use it?

In my patchset, in addition to the kernel patches, i also modify the kexec-tools.
arm64: support more than one crash kernel regions(http://lists.infradead.org/pipermail/kexec/2019-April/022792.html).
In kexec-tools patch, we read all the "Crash kernel" entry and load crash kernel high.

But we can't rely on people updating user-space when they update the kernel!

[...]

Quoted 16 lines
Quoted 16 lines

I'm afraid you've missed the ugly bit of the crashkernel reservation...

arch/arm64/mm/mmu.c::map_mem() marks the crashkernel as 'nomap' during the first pass of page-table generation. This means it isn't mapped in the linear map. It then maps it with page-size mappings, and removes the nomap flag.

This is done so that arch_kexec_protect_crashkres() and arch_kexec_unprotect_crashkres() can remove the valid bits of the crashkernel mapping. This way the old-kernel can't accidentally overwrite the crashkernel. It also saves us if the old-kernel and the crashkernel use different memory attributes for the mapping.

As your low-memory reservation is intended to be used for devices, having it mapped by the old-kernel as cacheable memory is going to cause problems if those CPUs aren't taken offline and go corrupting this memory. (we did crash for a reason after all)

Quoted 8 lines
Quoted 5 lines

I think the simplest thing to do is mark the low region as 'nomap' in reserve_crashkernel() and always leave it unmapped. We can then describe it via a different string in /proc/iomem, something like "Crash kernel (low)". Older kexec-tools shouldn't use it, (I assume its not using strncmp() in a way that would do this by accident), and newer kexec-tools can know to describe it in the DT, but it can't write to it.

I did miss the bit of the crashkernel reservation. I will fix this in next version.

I think all that is needed is to make the low-region nmap, and describe it via /proc/iomem with a name where nothing will try and use it by accident.

Thanks,

James

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#10 lore raw

PATCH 3/4 memblock: extend memblock_cap_memory_range to multiple ranges

In reply to Chen Zhou

From: Mike Rapoport <[email protected]>

The memblock_cap_memory_range() removes all the memory except the range passed to it. Extend this function to receive an array of memblock_regions that should be kept. This allows switching to simple iteration over memblock arrays with 'for_each_mem_range_rev' to remove the unneeded memory.

Enable use of this function in arm64 for reservation of multiple regions for the crash kernel.

---
 arch/arm64/mm/init.c     | 38 ++++++++++++++++++++++++++++----------
 include/linux/memblock.h |  2 +-
 mm/memblock.c            | 44 ++++++++++++++++++++------------------------
 3 files changed, 49 insertions(+), 35 deletions(-)
arch/arm64/mm/init.c +28 -10
index 3fcd739..2d8f302 100644
@@ -63,6 +63,13 @@ EXPORT_SYMBOL(memstart_addr);
 
 phys_addr_t arm64_dma_phys_limit __ro_after_init;
 
+/* The main usage of linux,usable-memory-range is for crash dump kernel.
+ * Originally, the number of usable-memory regions is one. Now crash dump
+ * kernel support at most two crash kernel regions, low_region and high
+ * region.
+ */
+#define MAX_USABLE_RANGES	2
+
 #ifdef CONFIG_KEXEC_CORE
 /*
  * reserve_crashkernel() - reserves memory for crash kernel
@@ -302,9 +309,9 @@ early_param("mem", early_mem);
 static int __init early_init_dt_scan_usablemem(unsigned long node,
 		const char *uname, int depth, void *data)
 {
-	struct memblock_region *usablemem = data;
-	const __be32 *reg;
-	int len;
+	struct memblock_type *usablemem = data;
+	const __be32 *reg, *endp;
+	int len, nr = 0;
 
 	if (depth != 1 || strcmp(uname, "chosen") != 0)
 		return 0;
@@ -313,22 +320,33 @@ static int __init early_init_dt_scan_usablemem(unsigned long node,
 	if (!reg || (len < (dt_root_addr_cells + dt_root_size_cells)))
 		return 1;
 
-	usablemem->base = dt_mem_next_cell(dt_root_addr_cells, &reg);
-	usablemem->size = dt_mem_next_cell(dt_root_size_cells, &reg);
+	endp = reg + (len / sizeof(__be32));
+	while ((endp - reg) >= (dt_root_addr_cells + dt_root_size_cells)) {
+		unsigned long base = dt_mem_next_cell(dt_root_addr_cells, &reg);
+		unsigned long size = dt_mem_next_cell(dt_root_size_cells, &reg);
+
+		if (memblock_add_range(usablemem, base, size, NUMA_NO_NODE,
+				       MEMBLOCK_NONE))
+			return 0;
+		if (++nr >= MAX_USABLE_RANGES)
+			break;
+	}
 
 	return 1;
 }
 
 static void __init fdt_enforce_memory_region(void)
 {
-	struct memblock_region reg = {
-		.size = 0,
+	struct memblock_region usable_regions[MAX_USABLE_RANGES];
+	struct memblock_type usablemem = {
+		.max = MAX_USABLE_RANGES,
+		.regions = usable_regions,
 	};
 
-	of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
+	of_scan_flat_dt(early_init_dt_scan_usablemem, &usablemem);
 
-	if (reg.size)
-		memblock_cap_memory_range(reg.base, reg.size);
+	if (usablemem.cnt)
+		memblock_cap_memory_ranges(usablemem.regions, usablemem.cnt);
 }
 
 void __init arm64_memblock_init(void)
include/linux/memblock.h +1 -1
index 676d390..526e279 100644
@@ -446,7 +446,7 @@ phys_addr_t memblock_mem_size(unsigned long limit_pfn);
 phys_addr_t memblock_start_of_DRAM(void);
 phys_addr_t memblock_end_of_DRAM(void);
 void memblock_enforce_memory_limit(phys_addr_t memory_limit);
-void memblock_cap_memory_range(phys_addr_t base, phys_addr_t size);
+void memblock_cap_memory_ranges(struct memblock_region *regions, int count);
 void memblock_mem_limit_remove_map(phys_addr_t limit);
 bool memblock_is_memory(phys_addr_t addr);
 bool memblock_is_map_memory(phys_addr_t addr);
mm/memblock.c +20 -24

63 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#11 lore raw

PATCH 4/4 kdump: update Documentation about crashkernel on arm64

In reply to Chen Zhou

Now we support crashkernel=X,[high,low] on arm64, update the Documentation.

---
 Documentation/admin-guide/kernel-parameters.txt | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -3
index 268b10a..03a08aa 100644
@@ -705,7 +705,7 @@
 			memory region [offset, offset + size] for that kernel
 			image. If '@offset' is omitted, then a suitable offset
 			is selected automatically.
-			[KNL, x86_64] select a region under 4G first, and
+			[KNL, x86_64, arm64] select a region under 4G first, and
 			fall back to reserve region above 4G when '@offset'
 			hasn't been specified.
 			See Documentation/kdump/kdump.txt for further details.
@@ -718,14 +718,14 @@
 			Documentation/kdump/kdump.txt for an example.
 
 	crashkernel=size[KMG],high
-			[KNL, x86_64] range could be above 4G. Allow kernel
+			[KNL, x86_64, arm64] range could be above 4G. Allow kernel
 			to allocate physical memory region from top, so could
 			be above 4G if system have more than 4G ram installed.
 			Otherwise memory region will be allocated below 4G, if
 			available.
 			It will be ignored if crashkernel=X is specified.
 	crashkernel=size[KMG],low
-			[KNL, x86_64] range under 4G. When crashkernel=X,high
+			[KNL, x86_64, arm64] range under 4G. When crashkernel=X,high
 			is passed, kernel could allocate physical memory region
 			above 4G, that cause second kernel crash on system
 			that require some amount of low memory, e.g. swiotlb
Signature · 1 line
2.7.4
#12 lore raw

Re: [PATCH 4/4] kdump: update Documentation about crashkernel on arm64

Quoted 29 lines · Chen Zhou

Now we support crashkernel=X,[high,low] on arm64, update the Documentation.

---
  Documentation/admin-guide/kernel-parameters.txt | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -2
index 268b10a..03a08aa 100644
@@ -705,7 +705,7 @@
  			memory region [offset, offset + size] for that kernel
  			image. If '@offset' is omitted, then a suitable offset
  			is selected automatically.
-			[KNL, x86_64] select a region under 4G first, and
+			[KNL, x86_64, arm64] select a region under 4G first, and
  			fall back to reserve region above 4G when '@offset'
  			hasn't been specified.
  			See Documentation/kdump/kdump.txt for further details.
@@ -718,14 +718,14 @@
  			Documentation/kdump/kdump.txt for an example.
 
  	crashkernel=size[KMG],high
-			[KNL, x86_64] range could be above 4G. Allow kernel
+			[KNL, x86_64, arm64] range could be above 4G. Allow kernel
  			to allocate physical memory region from top, so could

Documentation/admin-guide/kernel-parameters.txt line 731

Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +1 -1
  			be above 4G if system have more than 4G ram installed.
  			Otherwise memory region will be allocated below 4G, if
  			available.
  			It will be ignored if crashkernel=X is specified.
  	crashkernel=size[KMG],low
-			[KNL, x86_64] range under 4G. When crashkernel=X,high
+			[KNL, x86_64, arm64] range under 4G. When crashkernel=X,high
  			is passed, kernel could allocate physical memory region
  			above 4G, that cause second kernel crash on system
  			that require some amount of low memory, e.g. swiotlb

IMO, it is a good time to update 'Documentation/kdump/kdump.txt' with this patchset itself for both x86_64 and arm64, where we still specify only the old format for 'crashkernel' boot-argument:

Section: Boot into System Kernel

         =======================

On arm64, use "crashkernel=Y[@X]". Note that the start address of the kernel, X if explicitly specified, must be aligned to 2MiB (0x200000).
...

We can update this to add the new crashkernel=size[KMG],low or crashkernel=size[KMG],high format as well.

Thanks,
Bhupesh

author
#13 lore raw

Re: [PATCH 4/4] kdump: update Documentation about crashkernel on arm64

Quoted 49 lines · Bhupesh Sharma
Quoted 28 lines · Chen Zhou

Now we support crashkernel=X,[high,low] on arm64, update the Documentation.

---
  Documentation/admin-guide/kernel-parameters.txt | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -2
index 268b10a..03a08aa 100644
@@ -705,7 +705,7 @@
              memory region [offset, offset + size] for that kernel
              image. If '@offset' is omitted, then a suitable offset
              is selected automatically.
-            [KNL, x86_64] select a region under 4G first, and
+            [KNL, x86_64, arm64] select a region under 4G first, and
              fall back to reserve region above 4G when '@offset'
              hasn't been specified.
              See Documentation/kdump/kdump.txt for further details.
@@ -718,14 +718,14 @@
              Documentation/kdump/kdump.txt for an example.
        crashkernel=size[KMG],high
-            [KNL, x86_64] range could be above 4G. Allow kernel
+            [KNL, x86_64, arm64] range could be above 4G. Allow kernel
              to allocate physical memory region from top, so could
              be above 4G if system have more than 4G ram installed.

Documentation/admin-guide/kernel-parameters.txt line 731

Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +1 -1
              Otherwise memory region will be allocated below 4G, if
              available.
              It will be ignored if crashkernel=X is specified.
      crashkernel=size[KMG],low
-            [KNL, x86_64] range under 4G. When crashkernel=X,high
+            [KNL, x86_64, arm64] range under 4G. When crashkernel=X,high
              is passed, kernel could allocate physical memory region
              above 4G, that cause second kernel crash on system
              that require some amount of low memory, e.g. swiotlb
 

IMO, it is a good time to update 'Documentation/kdump/kdump.txt' with this patchset itself for both x86_64 and arm64, where we still specify only the old format for 'crashkernel' boot-argument:

Section: Boot into System Kernel

         =======================

On arm64, use "crashkernel=Y[@X]". Note that the start address of the kernel, X if explicitly specified, must be aligned to 2MiB (0x200000).
...

Quoted 6 lines

We can update this to add the new crashkernel=size[KMG],low or crashkernel=size[KMG],high format as well.

Thanks,
Bhupesh

.

Sure, we can also update here.

Thanks,
Chen Zhou

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#14 lore raw
In reply to Chen Zhou

+Cc kexec-list.

Hi Chen,

I think we are still in the quiet period of the merge cycle, but this is a change which will be useful for systems like HPE Apollo where we are looking at reserving crashkernel across a larger range.

Some comments inline and in respective patch threads..

Quoted 1 line · Chen Zhou

This patch series enable reserving crashkernel on high memory in arm64.

Please fix the patch subject, it should be v5.
Also please Cc the kexec-list ([email protected]) for future versions to allow wider review of the patchset.

Quoted 4 lines

We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.

... ^^ required

s/capture kernel will boot failure because of no low memory./capture kernel boot will fail because there is no low memory available for allocation.

Quoted 6 lines

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G. Then Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.

Please use consistent naming for the second kernel, better to use crash dump kernel.

I have tested this on my HPE Apollo machine and with crashkernel=886M,high syntax, I can get the board to reserve a larger memory range for the crashkernel (i.e. 886M):

# dmesg | grep -i crash
[    0.000000] kexec_core: Reserving 256MB of low memory at 3560MB for crashkernel (System low RAM: 2029MB)
[    0.000000] crashkernel reserved: 0x0000000bc5a00000 - 0x0000000bfd000000 (886 MB)

kexec/kdump can also work also work fine on the board.

So, with the changes suggested in this cover letter and individual patches, please feel free to add:

Thanks,
Bhupesh

Quoted 52 lines

Besides, we need to modify kexec-tools:
arm64: support more than one crash kernel regions(see [1])

I post this patch series about one month ago. The previous changes and discussions can be retrived from:

Changes since [v4]
- reimplement memblock_cap_memory_ranges for multiple ranges by Mike.

Changes since [v3]
- Add memblock_cap_memory_ranges back for multiple ranges.
- Fix some compiling warnings.

Changes since [v2]
- Split patch "arm64: kdump: support reserving crashkernel above 4G" as two. Put "move reserve_crashkernel_low() into kexec_core.c" in a separate patch.

Changes since [v1]:
- Move common reserve_crashkernel_low() code into kernel/kexec_core.c.
- Remove memblock_cap_memory_ranges() i added in v1 and implement that in fdt_enforce_memory_region().
There are at most two crash kernel regions, for two crash kernel regions case, we cap the memory range [min(regs[*].start), max(regs[*].end)] and then remove the memory range in the middle.

[1]: http://lists.infradead.org/pipermail/kexec/2019-April/022792.html
[v1]: https://lkml.org/lkml/2019/4/2/1174
[v2]: https://lkml.org/lkml/2019/4/9/86
[v3]: https://lkml.org/lkml/2019/4/9/306
[v4]: https://lkml.org/lkml/2019/4/15/273

Chen Zhou (3):
x86: kdump: move reserve_crashkernel_low() into kexec_core.c
arm64: kdump: support reserving crashkernel above 4G
kdump: update Documentation about crashkernel on arm64

Mike Rapoport (1):
memblock: extend memblock_cap_memory_range to multiple ranges

  Documentation/admin-guide/kernel-parameters.txt |  6 +--
  arch/arm64/include/asm/kexec.h                  |  3 ++
  arch/arm64/kernel/setup.c                       |  3 ++
  arch/arm64/mm/init.c                            | 72 +++++++++++++++++++------
  arch/x86/include/asm/kexec.h                    |  3 ++
  arch/x86/kernel/setup.c                         | 66 +++--------------------
  include/linux/kexec.h                           |  5 ++
  include/linux/memblock.h                        |  2 +-
  kernel/kexec_core.c                             | 56 +++++++++++++++++++
  mm/memblock.c                                   | 44 +++++++--------
  10 files changed, 157 insertions(+), 103 deletions(-)
author
#15 lore raw

Hi Bhupesh,

Quoted 38 lines · Bhupesh Sharma

+Cc kexec-list.

Hi Chen,

I think we are still in the quiet period of the merge cycle, but this is a change which will be useful for systems like HPE Apollo where we are looking at reserving crashkernel across a larger range.

Some comments inline and in respective patch threads..

Quoted 1 line · Chen Zhou

This patch series enable reserving crashkernel on high memory in arm64.

Please fix the patch subject, it should be v5.
Also please Cc the kexec-list ([email protected]) for future versions to allow wider review of the patchset.

Quoted 4 lines

We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.

... ^^ required

s/capture kernel will boot failure because of no low memory./capture kernel boot will fail because there is no low memory available for allocation.

Quoted 6 lines

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G. Then Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.

Please use consistent naming for the second kernel, better to use crash dump kernel.

I have tested this on my HPE Apollo machine and with crashkernel=886M,high syntax, I can get the board to reserve a larger memory range for the crashkernel (i.e. 886M):

# dmesg | grep -i crash
[    0.000000] kexec_core: Reserving 256MB of low memory at 3560MB for crashkernel (System low RAM: 2029MB)
[    0.000000] crashkernel reserved: 0x0000000bc5a00000 - 0x0000000bfd000000 (886 MB)
Quoted 9 lines

kexec/kdump can also work also work fine on the board.

So, with the changes suggested in this cover letter and individual patches, please feel free to add:

Thanks,
Bhupesh

Thanks for you review and test. I will fix these later.

Thanks,
Chen Zhou

Quoted 54 lines
Quoted 40 lines

Besides, we need to modify kexec-tools:
arm64: support more than one crash kernel regions(see [1])

I post this patch series about one month ago. The previous changes and discussions can be retrived from:

Changes since [v4]
- reimplement memblock_cap_memory_ranges for multiple ranges by Mike.

Changes since [v3]
- Add memblock_cap_memory_ranges back for multiple ranges.
- Fix some compiling warnings.

Changes since [v2]
- Split patch "arm64: kdump: support reserving crashkernel above 4G" as two. Put "move reserve_crashkernel_low() into kexec_core.c" in a separate patch.

Changes since [v1]:
- Move common reserve_crashkernel_low() code into kernel/kexec_core.c.
- Remove memblock_cap_memory_ranges() i added in v1 and implement that in fdt_enforce_memory_region().
There are at most two crash kernel regions, for two crash kernel regions case, we cap the memory range [min(regs[*].start), max(regs[*].end)] and then remove the memory range in the middle.

[1]: http://lists.infradead.org/pipermail/kexec/2019-April/022792.html
[v1]: https://lkml.org/lkml/2019/4/2/1174
[v2]: https://lkml.org/lkml/2019/4/9/86
[v3]: https://lkml.org/lkml/2019/4/9/306
[v4]: https://lkml.org/lkml/2019/4/15/273

Chen Zhou (3):
x86: kdump: move reserve_crashkernel_low() into kexec_core.c
arm64: kdump: support reserving crashkernel above 4G
kdump: update Documentation about crashkernel on arm64

Mike Rapoport (1):
memblock: extend memblock_cap_memory_range to multiple ranges

Quoted 12 lines
  Documentation/admin-guide/kernel-parameters.txt |  6 +--
  arch/arm64/include/asm/kexec.h                  |  3 ++
  arch/arm64/kernel/setup.c                       |  3 ++
  arch/arm64/mm/init.c                            | 72 +++++++++++++++++++------
  arch/x86/include/asm/kexec.h                    |  3 ++
  arch/x86/kernel/setup.c                         | 66 +++--------------------
  include/linux/kexec.h                           |  5 ++
  include/linux/memblock.h                        |  2 +-
  kernel/kexec_core.c                             | 56 +++++++++++++++++++
  mm/memblock.c                                   | 44 +++++++--------
  10 files changed, 157 insertions(+), 103 deletions(-)
Quoted 2 lines

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#16 lore raw

Hi Catalin,

Sorry to ping you.
What's your suggestion about this patch series? I am looking forward to your replay.

Thanks,
Chen Zhou

Quoted 110 lines · Chen Zhou

Hi Bhupesh,

Quoted 38 lines · Bhupesh Sharma

+Cc kexec-list.

Hi Chen,

I think we are still in the quiet period of the merge cycle, but this is a change which will be useful for systems like HPE Apollo where we are looking at reserving crashkernel across a larger range.

Some comments inline and in respective patch threads..

Quoted 1 line · Chen Zhou

This patch series enable reserving crashkernel on high memory in arm64.

Please fix the patch subject, it should be v5.
Also please Cc the kexec-list ([email protected]) for future versions to allow wider review of the patchset.

Quoted 4 lines

We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.

... ^^ required

s/capture kernel will boot failure because of no low memory./capture kernel boot will fail because there is no low memory available for allocation.

Quoted 6 lines

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G. Then Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.

Please use consistent naming for the second kernel, better to use crash dump kernel.

I have tested this on my HPE Apollo machine and with crashkernel=886M,high syntax, I can get the board to reserve a larger memory range for the crashkernel (i.e. 886M):

# dmesg | grep -i crash
[    0.000000] kexec_core: Reserving 256MB of low memory at 3560MB for crashkernel (System low RAM: 2029MB)
[    0.000000] crashkernel reserved: 0x0000000bc5a00000 - 0x0000000bfd000000 (886 MB)
Quoted 9 lines

kexec/kdump can also work also work fine on the board.

So, with the changes suggested in this cover letter and individual patches, please feel free to add:

Thanks,
Bhupesh

Thanks for you review and test. I will fix these later.

Thanks,
Chen Zhou

Quoted 54 lines
Quoted 52 lines

Besides, we need to modify kexec-tools:
arm64: support more than one crash kernel regions(see [1])

I post this patch series about one month ago. The previous changes and discussions can be retrived from:

Changes since [v4]
- reimplement memblock_cap_memory_ranges for multiple ranges by Mike.

Changes since [v3]
- Add memblock_cap_memory_ranges back for multiple ranges.
- Fix some compiling warnings.

Changes since [v2]
- Split patch "arm64: kdump: support reserving crashkernel above 4G" as two. Put "move reserve_crashkernel_low() into kexec_core.c" in a separate patch.

Changes since [v1]:
- Move common reserve_crashkernel_low() code into kernel/kexec_core.c.
- Remove memblock_cap_memory_ranges() i added in v1 and implement that in fdt_enforce_memory_region().
There are at most two crash kernel regions, for two crash kernel regions case, we cap the memory range [min(regs[*].start), max(regs[*].end)] and then remove the memory range in the middle.

[1]: http://lists.infradead.org/pipermail/kexec/2019-April/022792.html
[v1]: https://lkml.org/lkml/2019/4/2/1174
[v2]: https://lkml.org/lkml/2019/4/9/86
[v3]: https://lkml.org/lkml/2019/4/9/306
[v4]: https://lkml.org/lkml/2019/4/15/273

Chen Zhou (3):
x86: kdump: move reserve_crashkernel_low() into kexec_core.c
arm64: kdump: support reserving crashkernel above 4G
kdump: update Documentation about crashkernel on arm64

Mike Rapoport (1):
memblock: extend memblock_cap_memory_range to multiple ranges

  Documentation/admin-guide/kernel-parameters.txt |  6 +--
  arch/arm64/include/asm/kexec.h                  |  3 ++
  arch/arm64/kernel/setup.c                       |  3 ++
  arch/arm64/mm/init.c                            | 72 +++++++++++++++++++------
  arch/x86/include/asm/kexec.h                    |  3 ++
  arch/x86/kernel/setup.c                         | 66 +++--------------------
  include/linux/kexec.h                           |  5 ++
  include/linux/memblock.h                        |  2 +-
  kernel/kexec_core.c                             | 56 +++++++++++++++++++
  mm/memblock.c                                   | 44 +++++++--------
  10 files changed, 157 insertions(+), 103 deletions(-)
Quoted 6 lines
Quoted 2 lines

.

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#17 lore raw
In reply to Chen Zhou

Hi!

Quoted 8 lines · Chen Zhou

We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G.

This is a good argument for supporting the 'crashkernel=...,low' version. What is the 'crashkernel=...,high' version for?

Wouldn't it be simpler to relax the ARCH_LOW_ADDRESS_LIMIT if we see 'crashkernel=...,low' in the kernel cmdline?

I don't see what the 'crashkernel=...,high' variant is giving us, it just complicates the flow of reserve_crashkernel().

If we called reserve_crashkernel_low() at the beginning of reserve_crashkernel() we could use crashk_low_res.end to change some limit variable from ARCH_LOW_ADDRESS_LIMIT to memblock_end_of_DRAM().
I think this is a simpler change that gives you what you want.

Quoted 4 lines

Then
Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.

Won't this break if your kdump kernel doesn't know what the extra parameters are? Or if it expects two ranges, but only gets one? These DT properties should be treated as ABI between kernel versions, we can't really change it like this.

I think the 'low' region is an optional-extra, that is never mapped by the first kernel. I think the simplest thing to do is to add an 'linux,low-memory-range' that we memblock_add() after memblock_cap_memory_range() has been called.
If its missing, or the new kernel doesn't know what its for, everything keeps working.

Quoted 5 lines

Besides, we need to modify kexec-tools:
arm64: support more than one crash kernel regions(see [1])

I post this patch series about one month ago. The previous changes and discussions can be retrived from:

Ah, this wasn't obvious as you've stopped numbering the series. Please label the next one 'v6' so that we can describe this as 'v5'. (duplicate numbering would be even more confusing!)

Thanks,

James

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#18 lore raw
Quoted 18 lines · James Morse

Hi!

Quoted 8 lines · Chen Zhou

We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G.

This is a good argument for supporting the 'crashkernel=...,low' version. What is the 'crashkernel=...,high' version for?

Wouldn't it be simpler to relax the ARCH_LOW_ADDRESS_LIMIT if we see 'crashkernel=...,low' in the kernel cmdline?

Quoted 7 lines

I don't see what the 'crashkernel=...,high' variant is giving us, it just complicates the flow of reserve_crashkernel().

If we called reserve_crashkernel_low() at the beginning of reserve_crashkernel() we could use crashk_low_res.end to change some limit variable from ARCH_LOW_ADDRESS_LIMIT to memblock_end_of_DRAM().
I think this is a simpler change that gives you what you want.

According to your suggestions, we should do like this:
1. call reserve_crashkernel_low() at the beginning of reserve_crashkernel()
2. mark the low region as 'nomap'
3. use crashk_low_res.end to change some limit variable from ARCH_LOW_ADDRESS_LIMIT to memblock_end_of_DRAM()
4. rename crashk_low_res as "Crash kernel (low)" for arm64
5. add an 'linux,low-memory-range' node in DT

Do i understand correctly?

Quoted 17 lines
Quoted 6 lines

Then
Crash dump kernel reads more than one crash kernel regions via a dtb property under node /chosen,
linux,usable-memory-range = <BASE1 SIZE1 [BASE2 SIZE2]>.

Won't this break if your kdump kernel doesn't know what the extra parameters are? Or if it expects two ranges, but only gets one? These DT properties should be treated as ABI between kernel versions, we can't really change it like this.

I think the 'low' region is an optional-extra, that is never mapped by the first kernel. I think the simplest thing to do is to add an 'linux,low-memory-range' that we memblock_add() after memblock_cap_memory_range() has been called.
If its missing, or the new kernel doesn't know what its for, everything keeps working.

Quoted 9 lines
Quoted 5 lines

Besides, we need to modify kexec-tools:
arm64: support more than one crash kernel regions(see [1])

I post this patch series about one month ago. The previous changes and discussions can be retrived from:

Ah, this wasn't obvious as you've stopped numbering the series. Please label the next one 'v6' so that we can describe this as 'v5'. (duplicate numbering would be even more confusing!)

ok.

Quoted 7 lines

Thanks,

James

.

Thanks,
Chen Zhou

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#19 lore raw

Hi Chen Zhou,

Quoted 25 lines · Chen Zhou
Quoted 16 lines · James Morse
Quoted 8 lines · Chen Zhou

We use crashkernel=X to reserve crashkernel below 4G, which will fail when there is no enough memory. Currently, crashkernel=Y@X can be used to reserve crashkernel above 4G, in this case, if swiotlb or DMA buffers are requierd, capture kernel will boot failure because of no low memory.

When crashkernel is reserved above 4G in memory, kernel should reserve some amount of low memory for swiotlb and some DMA buffers. So there may be two crash kernel regions, one is below 4G, the other is above 4G.

This is a good argument for supporting the 'crashkernel=...,low' version. What is the 'crashkernel=...,high' version for?

Wouldn't it be simpler to relax the ARCH_LOW_ADDRESS_LIMIT if we see 'crashkernel=...,low' in the kernel cmdline?

Quoted 7 lines

I don't see what the 'crashkernel=...,high' variant is giving us, it just complicates the flow of reserve_crashkernel().

If we called reserve_crashkernel_low() at the beginning of reserve_crashkernel() we could use crashk_low_res.end to change some limit variable from ARCH_LOW_ADDRESS_LIMIT to memblock_end_of_DRAM().
I think this is a simpler change that gives you what you want.

Quoted 8 lines

According to your suggestions, we should do like this:
1. call reserve_crashkernel_low() at the beginning of reserve_crashkernel()
2. mark the low region as 'nomap'
3. use crashk_low_res.end to change some limit variable from ARCH_LOW_ADDRESS_LIMIT to memblock_end_of_DRAM()
4. rename crashk_low_res as "Crash kernel (low)" for arm64

5. add an 'linux,low-memory-range' node in DT

(This bit would happen in kexec-tools)

Quoted 1 line

Do i understand correctly?

Yes, I think this is simpler and still gives you what you want.
It also leaves the existing behaviour unchanged, which helps with keeping compatibility with existing user-space and older kdump kernels.

Thanks,

James

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec