kdump: crashkernel reservation from CMA
Subsystems (13)
v2 cover kdump: crashkernel reservation from CMA
Hi,
this series implements a way to reserve additional crash kernel memory using CMA.
Link to the v1 discussion:
https://lore.kernel.org/lkml/[email protected]/
See below for the changes since v1 and how concerns from the discussion have been addressed.
Currently, all the memory for the crash kernel is not usable by the 1st (production) kernel. It is also unmapped so that it can't be corrupted by the fault that will eventually trigger the crash. This makes sense for the memory actually used by the kexec-loaded crash kernel image and initrd and the data prepared during the load (vmcoreinfo, ...). However, the reserved space needs to be much larger than that to provide enough run-time memory for the crash kernel and the kdump userspace. Estimating the amount of memory to reserve is difficult. Being too careful makes kdump likely to end in OOM, being too generous takes even more memory from the production system. Also, the reservation only allows reserving a single contiguous block (or two with the "low" suffix). I've seen systems where this fails because the physical memory is fragmented.
By reserving additional crashkernel memory from CMA, the main crashkernel reservation can be just large enough to fit the kernel and initrd image, minimizing the memory taken away from the production system. Most of the run-time memory for the crash kernel will be memory previously available to userspace in the production system. As this memory is no longer wasted, the reservation can be done with a generous margin, making kdump more reliable. Kernel memory that we need to preserve for dumping is never allocated from CMA. User data is typically not dumped by makedumpfile. When dumping of user data is intended this new CMA reservation cannot be used.
There are five patches in this series:
The first adds a new ",cma" suffix to the recenly introduced generic crashkernel parsing code. parse_crashkernel() takes one more argument to store the cma reservation size.
The second patch implements reserve_crashkernel_cma() which performs the reservation. If the requested size is not available in a single range, multiple smaller ranges will be reserved.
The third patch updates Documentation/, explicitly mentioning the potential DMA corruption of the CMA-reserved memory.
The fourth patch adds a short delay before booting the kdump kernel, allowing pending DMA transfers to finish.
The fifth patch enables the functionality for x86 as a proof of concept. There are just three things every arch needs to do:
- call reserve_crashkernel_cma()
- include the CMA-reserved ranges in the physical memory map
- exclude the CMA-reserved ranges from the memory available through /proc/vmcore by excluding them from the vmcoreinfo PT_LOAD ranges.
Adding other architectures is easy and I can do that as soon as this series is merged.
With this series applied, specifying
crashkernel=100M craskhernel=1G,cma
on the command line will make a standard crashkernel reservation of 100M, where kexec will load the kernel and initrd.
An additional 1G will be reserved from CMA, still usable by the production system. The crash kernel will have 1.1G memory available. The 100M can be reliably predicted based on the size of the kernel and initrd.
The new cma suffix is completely optional. When no crashkernel=size,cma is specified, everything works as before.
---
Changes since v1:
The key concern raised in the v1 discussion was that pages in the CMA region may be pinned and used for a DMA transfer, potentially corrupting the new kernel's memory. When the cma suffix is used, kdump may be less reliable and the corruption hard to debug
This v2 series addresses this concern in two ways:
1) Clearly stating the potential problem in the updated Documentation and setting the expectation (patch 3/5)
Documentation now explicitly states that:
- the risk of kdump failure is increased
- the CMA reservation is intended for users who can not or don't want to sacrifice enough memory for a standard crashkernel reservation and who prefer less reliable kdump to no kdump at all
This is consistent with the documentation of the crash_kexec_post_notifiers option, which can also increase the risk of kdump failure, yet may be the only way to use kdump on some systems. And just like the crash_kexec_post_notifiers option, the cma crashkernel suffix is completely optional:
the series has zero effect when the suffix is not used.
2) Giving DMA time to finish before booting the kdump kernel (patch 4/5)
Pages can be pinned for long term use using the FOLL_LONGTERM flag. Then they are migrated outside the CMA region. Pinning without this flag shows that the intent of their user is to only use them for short-lived DMA transfers.
Delay the boot of the kdump kernel when the CMA reservation is used, giving potential pending DMA transfers time to finish.
Other minor changes since v1:
- updated for 6.14-rc2
- moved #ifdefs and #defines to header files
- added __always_unused in parse_crashkernel() to silence a false unused variable warning
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
PATCH v2 1/5 Add a new optional ",cma" suffix to the crashkernel= command line option
Add a new cma_size parameter to parse_crashkernel(). When not NULL, call __parse_crashkernel to parse the CMA reservation size from "crashkernel=size,cma" and store it in cma_size.
Set cma_size to NULL in all calls to parse_crashkernel().
--- arch/arm/kernel/setup.c | 2 +- arch/arm64/mm/init.c | 2 +- arch/loongarch/kernel/setup.c | 2 +- arch/mips/kernel/setup.c | 2 +- arch/powerpc/kernel/fadump.c | 2 +- arch/powerpc/kexec/core.c | 2 +- arch/powerpc/mm/nohash/kaslr_booke.c | 2 +- arch/riscv/mm/init.c | 2 +- arch/s390/kernel/setup.c | 2 +- arch/sh/kernel/machine_kexec.c | 2 +- arch/x86/kernel/setup.c | 2 +- include/linux/crash_reserve.h | 3 ++- kernel/crash_reserve.c | 22 +++++++++++++++++----- 13 files changed, 30 insertions(+), 17 deletions(-)
arch/arm/kernel/setup.c +1 -1
@@ -1004,7 +1004,7 @@ static void __init reserve_crashkernel(void) total_mem =get_total_mem (); ret =parse_crashkernel (boot_command_line, total_mem, &crash_size, &crash_base, -NULL ,NULL ); +NULL ,NULL ,NULL );/* invalid value specified or crashkernel=0 */ if (ret || !crash_size)return ;
arch/arm64/mm/init.c +1 -1
@@ -107,7 +107,7 @@ static void __init arch_reserve_crashkernel(void) ret =parse_crashkernel (cmdline,memblock_phys_mem_size (), &crash_size, &crash_base, - &low_size, &high); + &low_size,NULL , &high);if (ret)return ;
arch/loongarch/kernel/setup.c +1 -1
@@ -266,7 +266,7 @@ static void __init arch_reserve_crashkernel(void)return ; ret =parse_crashkernel (cmdline,memblock_phys_mem_size (), - &crash_size, &crash_base, &low_size, &high); + &crash_size, &crash_base, &low_size,NULL , &high);if (ret)return ;
arch/mips/kernel/setup.c +1 -1
@@ -458,7 +458,7 @@ static void __init mips_parse_crashkernel(void) total_mem =memblock_phys_mem_size (); ret =parse_crashkernel (boot_command_line, total_mem, &crash_size, &crash_base, -NULL ,NULL ); +NULL ,NULL ,NULL );if (ret !=0 || crash_size <=0 )return ;
arch/powerpc/kernel/fadump.c +1 -1
@@ -334,7 +334,7 @@ static __init u64 fadump_calculate_reserve_size(void)* memory at a predefined offset. */ ret =parse_crashkernel (boot_command_line,memblock_phys_mem_size (), - &size, &base,NULL ,NULL ); + &size, &base,NULL ,NULL ,NULL );if (ret ==0 && size >0 ) {unsigned long max_size;
arch/powerpc/kexec/core.c +1 -1
@@ -66,7 +66,7 @@ void __init reserve_crashkernel(void) total_mem_sz = memory_limit ? memory_limit :memblock_phys_mem_size ();/* use common parsing */ ret =parse_crashkernel (boot_command_line, total_mem_sz, - &crash_size, &crash_base,NULL ,NULL ); + &crash_size, &crash_base,NULL ,NULL ,NULL );if (ret ==0 && crash_size >0 ) { crashk_res.start = crash_base; crashk_res.end = crash_base + crash_size -1 ;
arch/powerpc/mm/nohash/kaslr_booke.c +1 -1
@@ -178,7 +178,7 @@ static void __init get_crash_kernel(void *fdt, unsigned long size)int ret; ret =parse_crashkernel (boot_command_line, size, &crash_size, - &crash_base,NULL ,NULL ); + &crash_base,NULL ,NULL ,NULL );if (ret !=0 || crash_size ==0 )return ;if (crash_base ==0 )
arch/riscv/mm/init.c +1 -1
@@ -1405,7 +1405,7 @@ static void __init arch_reserve_crashkernel(void) ret =parse_crashkernel (cmdline,memblock_phys_mem_size (), &crash_size, &crash_base, - &low_size, &high); + &low_size,NULL , &high);if (ret)return ;
arch/s390/kernel/setup.c +1 -1
@@ -607,7 +607,7 @@ static void __init reserve_crashkernel(void)int rc; rc =parse_crashkernel (boot_command_line, ident_map_size, - &crash_size, &crash_base,NULL ,NULL ); + &crash_size, &crash_base,NULL ,NULL ,NULL ); crash_base =ALIGN (crash_base,KEXEC_CRASH_MEM_ALIGN ); crash_size =ALIGN (crash_size,KEXEC_CRASH_MEM_ALIGN );
arch/sh/kernel/machine_kexec.c +1 -1
@@ -146,7 +146,7 @@ void __init reserve_crashkernel(void)return ; ret =parse_crashkernel (boot_command_line,memblock_phys_mem_size (), - &crash_size, &crash_base,NULL ,NULL ); + &crash_size, &crash_base,NULL ,NULL ,NULL );if (ret ==0 && crash_size >0 ) { crashk_res.start = crash_base; crashk_res.end = crash_base + crash_size -1 ;
arch/x86/kernel/setup.c +1 -1
@@ -481,7 +481,7 @@ static void __init arch_reserve_crashkernel(void) ret =parse_crashkernel (cmdline,memblock_phys_mem_size (), &crash_size, &crash_base, - &low_size, &high); + &low_size,NULL , &high);if (ret)return ;
include/linux/crash_reserve.h +2 -1
@@ -16,7 +16,8 @@ extern struct resource crashk_low_res;int __init parse_crashkernel (char *cmdline,unsigned long long system_ram,unsigned long long *crash_size,unsigned long long *crash_base, -unsigned long long *low_size,bool *high); +unsigned long long *low_size,unsigned long long *cma_size, +bool *high);#ifdef CONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION #ifndef DEFAULT_CRASH_KERNEL_LOW_SIZE
kernel/crash_reserve.c +17 -5
@@ -172,17 +172,19 @@ static int __init parse_crashkernel_simple(char *cmdline,#define SUFFIX_HIGH 0#define SUFFIX_LOW 1 -#define SUFFIX_NULL 2 +#define SUFFIX_CMA 2 +#define SUFFIX_NULL 3static __initdata char *suffix_tbl[] = { - [SUFFIX_HIGH ] =",high" , - [SUFFIX_LOW ] =",low" , - [SUFFIX_NULL ] =NULL , + [SUFFIX_HIGH ] =",high" , + [SUFFIX_LOW ] =",low" , + [SUFFIX_CMA ] =",cma" , + [SUFFIX_NULL ] =NULL , };/* * That function parses "suffix" crashkernel command lines like * -* crashkernel=size,[high|low] +* crashkernel=size,[high|low|cma] * * It returns 0 on success and -EINVAL on failure. */
@@ -298,9 +300,11 @@ int __init parse_crashkernel(char *cmdline,unsigned long long *crash_size, unsigned long long *crash_base, unsigned long long *low_size, + unsigned long long *cma_size, bool *high) { int ret; +unsigned long long __always_unused cma_base;/* crashkernel=X[@offset] */ ret = __parse_crashkernel(cmdline, system_ram, crash_size,
@@ -331,6 +335,14 @@ int __init parse_crashkernel(char *cmdline, *high = true; } + +/* +* optional CMA reservation +* cma_base is ignored +*/ +if (cma_size) +__parse_crashkernel (cmdline,0 , cma_size, + &cma_base, suffix_tbl[SUFFIX_CMA ]);#endif if (!*crash_size) ret = -EINVAL ;
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
Re: [PATCH v2 1/5] Add a new optional ",cma" suffix to the crashkernel= command line option
On 02/20/25 at 05:51pm, Jiri Bohac wrote:
......snip...
Quoted 10 lines
kernel/crash_reserve.c +1
-1@@ -172,17 +172,19 @@ static int __init parse_crashkernel_simple(char *cmdline,#define SUFFIX_HIGH 0#define SUFFIX_LOW 1 -#define SUFFIX_NULL 2 +#define SUFFIX_CMA 2
kernel/crash_reserve.c static int __init parse_crashkernel_simple(char *cmdline, · line 182
Quoted 10 lines
kernel/crash_reserve.c +5
-3+#define SUFFIX_NULL 3static __initdata char *suffix_tbl[] = { - [SUFFIX_HIGH ] =",high" , - [SUFFIX_LOW ] =",low" , - [SUFFIX_NULL ] =NULL , + [SUFFIX_HIGH ] =",high" , + [SUFFIX_LOW ] =",low" , + [SUFFIX_CMA ] =",cma" , + [SUFFIX_NULL ] =NULL , };
Seems the old style looks a little better.
......
PATCH v2 2/5 kdump: implement reserve_crashkernel_cma
In reply to Jiri Bohacreserve_crashkernel_cma() reserves CMA ranges for the crash kernel. If allocating the requested size fails, try to reserve in smaller blocks.
Store the reserved ranges in the crashk_cma_ranges array and the number of ranges in crashk_cma_cnt.
--- include/linux/crash_reserve.h | 12 +++++++++ kernel/crash_reserve.c | 49 +++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+)
include/linux/crash_reserve.h +12 -0
@@ -13,12 +13,24 @@*/ extern struct resource crashk_res;extern struct resource crashk_low_res; +extern struct range crashk_cma_ranges[]; +#if defined(CONFIG_CMA ) && defined(CONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION ) +#define CRASHKERNEL_CMA +#define CRASHKERNEL_CMA_RANGES_MAX 4 +extern int crashk_cma_cnt; +#else +#define crashk_cma_cnt 0 +#define CRASHKERNEL_CMA_RANGES_MAX 0 +#endif +int __init parse_crashkernel (char *cmdline,unsigned long long system_ram,unsigned long long *crash_size,unsigned long long *crash_base,unsigned long long *low_size,unsigned long long *cma_size,bool *high); +void __init reserve_crashkernel_cma (unsigned long long cma_size); +#ifdef CONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION #ifndef DEFAULT_CRASH_KERNEL_LOW_SIZE #define DEFAULT_CRASH_KERNEL_LOW_SIZE (128UL << 20)
kernel/crash_reserve.c +49 -0
61 lines not on this page · raw message
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
PATCH v2 3/5 kdump, documentation: describe craskernel CMA reservation
In reply to Jiri BohacDescribe the new crashkernel ",cma" suffix in Documentation/
--- Documentation/admin-guide/kdump/kdump.rst | 19 +++++++++++++++++-- .../admin-guide/kernel-parameters.txt | 14 ++++++++++++++ 2 files changed, 31 insertions(+), 2 deletions(-)
Documentation/admin-guide/kdump/kdump.rst +17 -2
@@ -213,8 +213,7 @@ Dump-capture kernel config options (Arch Dependent, i386 and x86_64) kernel. Otherwise it should be the start of memory region reserved for - second kernel using boot parameter "crashkernel=Y@X". Here X is - start of memory region reserved for dump-capture kernel. + second kernel using boot parameter "crashkernel=Y@X". Here X is start of memory region reserved for dump-capture kernel. Generally X is 16MB (0x1000000). So you can set CONFIG_PHYSICAL_START=0x1000000
@@ -315,6 +314,22 @@ crashkernel syntax crashkernel=0,low +4) crashkernel=size,cma + + Reserves additional memory from CMA. A standard crashkernel reservation, as + described above, is still needed, but can be just big enough to hold the + kernel and initrd. All the memory the crash kernel needs for its runtime and + for running the kdump userspace processes can be provided by this CMA + reservation, re-using memory available to the production system's userspace. + Because of this, the CMA reservation should not be used if kdump is configured + to dump userspace memory - the re-used memory ranges won't be in the vmcore. + + This option increases the risk of a kdump failure: DMA transfers configured + by the first kernel may end up corrupting the second kernel's memory. This + reservation method is intended for systems that can't afford to sacrifice + enough memory for standard crashkernel reservation and where less reliable + kdump is preferrable to no kdump at all. + Boot into System Kernel ----------------------- 1) Update the boot loader (such as grub, yaboot, or lilo) configuration
Documentation/admin-guide/kernel-parameters.txt +14 -0
@@ -987,6 +987,20 @@ 0: to disable low allocation. It will be ignored when crashkernel=X,high is not used or memory reserved is below 4G. + crashkernel=size[KMG],cma + [KNL, X86] Reserve additional crash kernel memory from CMA. + This reservation is usable by the first system's userspace, + so this should not be used if dumping of userspace + memory is intended. A standard crashkernel reservation, + as described above, is still needed to hold the crash + kernel and initrd. + This option increases the risk of a kdump failure: DMA + transfers configured by the first kernel may end up + corrupting the second kernel's memory. This reservation + method is intended for systems that can't afford to + sacrifice enough memory for standard crashkernel + reservation and where less reliable kdump is preferrable + to no kdump at all. cryptomgr.notests [KNL] Disable crypto self-tests
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
Re: [PATCH v2 3/5] kdump, documentation: describe craskernel CMA reservation
On 02/20/25 at 05:54pm, Jiri Bohac wrote:
......
Quoted 11 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0@@ -987,6 +987,20 @@ 0: to disable low allocation. It will be ignored when crashkernel=X,high is not used or memory reserved is below 4G. + crashkernel=size[KMG],cma + [KNL, X86] Reserve additional crash kernel memory from CMA. + This reservation is usable by the first system's userspace,
Documentation/admin-guide/kernel-parameters.txt line 1002
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +10
-0+ so this should not be used if dumping of userspace + memory is intended. A standard crashkernel reservation, + as described above, is still needed to hold the crash + kernel and initrd. + This option increases the risk of a kdump failure: DMA + transfers configured by the first kernel may end up + corrupting the second kernel's memory. This reservation + method is intended for systems that can't afford to + sacrifice enough memory for standard crashkernel + reservation and where less reliable kdump is preferrable
There's trailing whitespace at the end of above line, checkpatch should be run to check it.
Quoted 9 lines
Documentation/admin-guide/kernel-parameters.txt +1
-0+ to no kdump at all. cryptomgr.notests [KNL] Disable crypto self-testsSignature · 3 lines
-- Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
PATCH v2 4/5 kdump: wait for DMA to finish when using CMA
In reply to Jiri BohacWhen re-using the CMA area for kdump there is a risk of pending DMA into pinned user pages in the CMA area.
Pages that are pinned long-term are migrated away from CMA, so these are not a concern. Pages pinned without FOLL_LONGTERM remain in the CMA and may possibly be the source or destination of a pending DMA transfer.
Although there is no clear specification how long a page may be pinned without FOLL_LONGTERM, pinning without the flag shows an intent of the caller to only use the memory for short-lived DMA transfers, not a transfer initiated by a device asynchronously at a random time in the future.
Add a delay of CMA_DMA_TIMEOUT_MSEC milliseconds before starting the kdump kernel, giving such short-lived DMA transfers time to finish before the CMA memory is re-used by the kdump kernel.
Set CMA_DMA_TIMEOUT_MSEC to 1000 (one second) - chosen arbitrarily as both a huge margin for a DMA transfer, yet not increasing the kdump time significantly.
--- include/linux/crash_core.h | 5 +++++ kernel/crash_core.c | 10 ++++++++++ 2 files changed, 15 insertions(+)
include/linux/crash_core.h +5 -0
@@ -56,6 +56,11 @@ static inline unsigned int crash_get_elfcorehdr_size(void) { return 0; }/* Alignment required for elf header segment */ #define ELF_CORE_HEADER_ALIGN 4096 +/* Time to wait for possible DMA to finish before starting the kdump kernel +* when a CMA reservation is used +*/ +#define CMA_DMA_TIMEOUT_MSEC 1000 +extern int crash_exclude_mem_range (struct crash_mem *mem,unsigned long long mstart,unsigned long long mend);
kernel/crash_core.c +10 -0
@@ -21,6 +21,7 @@#include <linux/reboot.h> #include <linux/btf.h> #include <linux/objtool.h> +#include <linux/delay.h> #include <asm/page.h> #include <asm/sections.h>
@@ -97,6 +98,14 @@ int kexec_crash_loaded(void) }EXPORT_SYMBOL_GPL (kexec_crash_loaded); +static void crash_cma_clear_pending_dma (void ) +{ +if (!crashk_cma_cnt) +return ; + +mdelay (CMA_DMA_TIMEOUT_MSEC ); +} +/* * No panic_cpu check version of crash_kexec(). This function is called * only when panic_cpu holds the current CPU number; this is the only CPU
@@ -116,6 +125,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)if (kexec_crash_image) {struct pt_regs fixed_regs; +crash_cma_clear_pending_dma ();crash_setup_regs (&fixed_regs, regs);crash_save_vmcoreinfo ();machine_crash_shutdown (&fixed_regs);
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
Re: [PATCH v2 4/5] kdump: wait for DMA to finish when using CMA
Quoted 70 lines · Jiri Bohac
When re-using the CMA area for kdump there is a risk of pending DMA into pinned user pages in the CMA area.
Pages that are pinned long-term are migrated away from CMA, so these are not a concern. Pages pinned without
FOLL_LONGTERMremain in the CMA and may possibly be the source or destination of a pending DMA transfer.Although there is no clear specification how long a page may be pinned without
FOLL_LONGTERM, pinning without the flag shows an intent of the caller to only use the memory for short-lived DMA transfers, not a transfer initiated by a device asynchronously at a random time in the future.Add a delay of
CMA_DMA_TIMEOUT_MSECmilliseconds before starting the kdump kernel, giving such short-lived DMA transfers time to finish before the CMA memory is re-used by the kdump kernel.Set
CMA_DMA_TIMEOUT_MSECto 1000 (one second) - chosen arbitrarily as both a huge margin for a DMA transfer, yet not increasing the kdump time significantly.Signed-off-by Jiri Bohac <[email protected]>--- include/linux/crash_core.h | 5 +++++ kernel/crash_core.c | 10 ++++++++++ 2 files changed, 15 insertions(+)include/linux/crash_core.h +5
-0@@ -56,6 +56,11 @@ static inline unsigned int crash_get_elfcorehdr_size(void) { return 0; }/* Alignment required for elf header segment */ #define ELF_CORE_HEADER_ALIGN 4096 +/* Time to wait for possible DMA to finish before starting the kdump kernel +* when a CMA reservation is used +*/ +#define CMA_DMA_TIMEOUT_MSEC 1000 +extern int crash_exclude_mem_range (struct crash_mem *mem,unsigned long long mstart,unsigned long long mend);kernel/crash_core.c +9
-0@@ -21,6 +21,7 @@#include <linux/reboot.h> #include <linux/btf.h> #include <linux/objtool.h> +#include <linux/delay.h> #include <asm/page.h> #include <asm/sections.h> @@ -97,6 +98,14 @@ int kexec_crash_loaded(void) }EXPORT_SYMBOL_GPL (kexec_crash_loaded); +static void crash_cma_clear_pending_dma (void ) +{ +if (!crashk_cma_cnt) +return ; + +mdelay (CMA_DMA_TIMEOUT_MSEC ); +} +/* * No panic_cpu check version of crash_kexec(). This function is called * only when panic_cpu holds the current CPU number; this is the only CPU
kernel/crash_core.c void __noclone __crash_kexec(struct pt_regs *regs) · line 128
Quoted 4 lines
kernel/crash_core.c +1
-0@@ -116,6 +125,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)if (kexec_crash_image) {struct pt_regs fixed_regs; +crash_cma_clear_pending_dma ();
This could be too ideal, I am not sure if it's a good way. When crash triggered, we need do the urgent and necessary thing as soon as possible, then shutdown all CPU to avoid further damage. This one second of waiting could give the strayed system too much time. My personal opinion.
Quoted 8 lines
kernel/crash_core.c +0
-0crash_setup_regs (&fixed_regs, regs);crash_save_vmcoreinfo ();machine_crash_shutdown (&fixed_regs);Signature · 3 lines
-- Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
Re: [PATCH v2 4/5] kdump: wait for DMA to finish when using CMA
Quoted 16 lines · Baoquan He
Quoted 15 lines · Jiri Bohac
kernel/crash_core.c +8
-0+static void crash_cma_clear_pending_dma (void ) +{ +if (!crashk_cma_cnt) +return ; + +mdelay (CMA_DMA_TIMEOUT_MSEC ); +} +/* * No panic_cpu check version of crash_kexec(). This function is called * only when panic_cpu holds the current CPU number; this is the only CPU @@ -116,6 +125,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)if (kexec_crash_image) {struct pt_regs fixed_regs;
Quoted 7 lines
kernel/crash_core.c void __noclone __crash_kexec(struct pt_regs *regs) · line 128
Quoted 1 line
kernel/crash_core.c +1
-0+crash_cma_clear_pending_dma ();This could be too ideal, I am not sure if it's a good way. When crash triggered, we need do the urgent and necessary thing as soon as possible, then shutdown all CPU to avoid further damage. This one second of waiting could give the strayed system too much time. My personal opinion.
Good point! I think it makes sense to move the call to crash_cma_clear_pending_dma() past the call of machine_crash_shutdown where all the shutdown happens, like this:
Quoted 3 lines
kernel/crash_core.c void __noclone __crash_kexec(struct pt_regs *regs) · line 131
Quoted 3 lines
kernel/crash_core.c +0
-0crash_setup_regs (&fixed_regs, regs);crash_save_vmcoreinfo ();machine_crash_shutdown (&fixed_regs);
+ crash_cma_clear_pending_dma();
I'll post a v3 with this change included.
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
PATCH v2 5/5 x86: implement crashkernel cma reservation
In reply to Jiri BohacImplement the crashkernel CMA reservation for x86:
- enable parsing of the cma suffix by parse_crashkernel()
- reserve memory with reserve_crashkernel_cma()
- add the CMA-reserved ranges to the e820 map for the crash kernel
- exclude the CMA-reserved ranges from vmcore
--- arch/x86/kernel/crash.c | 26 ++++++++++++++++++++++---- arch/x86/kernel/setup.c | 5 +++-- 2 files changed, 25 insertions(+), 6 deletions(-)
arch/x86/kernel/crash.c +22 -4
@@ -163,10 +163,10 @@ static struct crash_mem *fill_up_crash_elf_data(void)return NULL ;/* -* Exclusion of crash region and/or crashk_low_res may cause -* another range split. So add extra two slots here. +* Exclusion of crash region, crashk_low_res and/or crashk_cma_ranges +* may cause range splits. So add extra slots here. */ - nr_ranges +=2 ; + nr_ranges +=2 + crashk_cma_cnt; cmem =vzalloc (struct_size (cmem, ranges, nr_ranges));if (!cmem)return NULL ;
@@ -184,6 +184,7 @@ static struct crash_mem *fill_up_crash_elf_data(void)static int elf_header_exclude_ranges (struct crash_mem *cmem) {int ret =0 ; +int i;/* Exclude the low 1M because it is always reserved */ ret =crash_exclude_mem_range (cmem,0 ,SZ_1M -1 );
@@ -198,8 +199,17 @@ static int elf_header_exclude_ranges(struct crash_mem *cmem)if (crashk_low_res.end) ret =crash_exclude_mem_range (cmem, crashk_low_res.start, crashk_low_res.end); +if (ret) +return ret; -return ret; +for (i =0 ; i < crashk_cma_cnt; ++i) { + ret =crash_exclude_mem_range (cmem, crashk_cma_ranges[i].start, + crashk_cma_ranges[i].end); +if (ret) +return ret; + } + +return 0 ; }static int prepare_elf64_ram_headers_callback (struct resource *res,void *arg)
@@ -352,6 +362,14 @@ int crash_setup_memmap_entries(struct kimage *image, struct boot_params *params)add_e820_entry (params, &ei); } +for (i =0 ; i < crashk_cma_cnt; ++i) { + ei.addr = crashk_cma_ranges[i].start; + ei.size = crashk_cma_ranges[i].end - + crashk_cma_ranges[i].start +1 ; + ei.type =E820_TYPE_RAM ; +add_e820_entry (params, &ei); + } + out:vfree (cmem);return ret;
arch/x86/kernel/setup.c +3 -2
@@ -471,7 +471,7 @@ static void __init memblock_x86_reserve_range_setup_data(void)static void __init arch_reserve_crashkernel (void ) { -unsigned long long crash_base, crash_size, low_size =0 ; +unsigned long long crash_base, crash_size, low_size =0 , cma_size =0 ;char *cmdline = boot_command_line;bool high = false;int ret;
@@ -481,7 +481,7 @@ static void __init arch_reserve_crashkernel(void) ret =parse_crashkernel (cmdline,memblock_phys_mem_size (), &crash_size, &crash_base, - &low_size,NULL , &high); + &low_size, &cma_size, &high);if (ret)return ;
@@ -492,6 +492,7 @@ static void __init arch_reserve_crashkernel(void)reserve_crashkernel_generic (cmdline, crash_size, crash_base, low_size, high); +reserve_crashkernel_cma (cma_size); }static struct resource standard_io_resources[] = {
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
On 02/20/25 at 05:48pm, Jiri Bohac wrote:
......snip...
Quoted 19 lines
---
Changes since v1:The key concern raised in the v1 discussion was that pages in the CMA region may be pinned and used for a DMA transfer, potentially corrupting the new kernel's memory. When the cma suffix is used, kdump may be less reliable and the corruption hard to debug
This v2 series addresses this concern in two ways:
1) Clearly stating the potential problem in the updated Documentation and setting the expectation (patch 3/5)
Documentation now explicitly states that:
- the risk of kdump failure is increased
- the CMA reservation is intended for users who can not or don't want to sacrifice enough memory for a standard crashkernel reservation and who prefer less reliable kdump to no kdump at all
Quoted 6 lines
This is consistent with the documentation of the
crash_kexec_post_notifiersoption, which can also increase the risk of kdump failure, yet may be the only way to use kdump on some systems. And just like thecrash_kexec_post_notifiersoption, the cma crashkernel suffix is completely optional:
the series has zero effect when the suffix is not used.
Thanks for the effort to investigate and add clear note about the potential risk in document. Except of the 1 second waiting for short term pined page for DMA, the whole series looks good to me. Hope other people can also give comment to evaluate the risk of waiting, I will wait another week to add my personal ACK.
Thanks
Baoquan
Quoted 24 lines
2) Giving DMA time to finish before booting the kdump kernel (patch 4/5)
Pages can be pinned for long term use using the
FOLL_LONGTERMflag. Then they are migrated outside the CMA region. Pinning without this flag shows that the intent of their user is to only use them for short-lived DMA transfers.Delay the boot of the kdump kernel when the CMA reservation is used, giving potential pending DMA transfers time to finish.
Other minor changes since v1:
- updated for 6.14-rc2
- moved #ifdefs and #defines to header files
- added__always_unusedinparse_crashkernel()to silence a false unused variable warningSignature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
Quoted 25 lines · Jiri Bohac
Hi,
this series implements a way to reserve additional crash kernel memory using CMA.
Link to the v1 discussion:
https://lore.kernel.org/lkml/[email protected]/
See below for the changes since v1 and how concerns from the discussion have been addressed.Currently, all the memory for the crash kernel is not usable by the 1st (production) kernel. It is also unmapped so that it can't be corrupted by the fault that will eventually trigger the crash. This makes sense for the memory actually used by the kexec-loaded crash kernel image and initrd and the data prepared during the load (vmcoreinfo, ...). However, the reserved space needs to be much larger than that to provide enough run-time memory for the crash kernel and the kdump userspace. Estimating the amount of memory to reserve is difficult. Being too careful makes kdump likely to end in OOM, being too generous takes even more memory from the production system. Also, the reservation only allows reserving a single contiguous block (or two with the "low" suffix). I've seen systems where this fails because the physical memory is fragmented.
Quoted 11 lines
By reserving additional crashkernel memory from CMA, the main crashkernel reservation can be just large enough to fit the kernel and initrd image, minimizing the memory taken away from the production system. Most of the run-time memory for the crash kernel will be memory previously available to userspace in the production system. As this memory is no longer wasted, the reservation can be done with a generous margin, making kdump more reliable. Kernel memory that we need to preserve for dumping is never allocated from CMA. User data is typically not dumped by makedumpfile. When dumping of user data is intended this new CMA reservation cannot be used.
Hi,
I'll note that your comment about "user space" is currently the case, but will likely not hold in the long run. The assumption you are making is that only user-space memory will be allocated from MIGRATE_CMA, which is not necessarily the case. Any movable allocation will end up in there.
Besides LRU folios (user space memory and the pagecache), we already support migration of some kernel allocations using the non-lru migration framework. Such allocations (which use __GFP_MOVABLE, see __SetPageMovable()) currently only include
* memory balloon: pages we never want to dump either way
* zsmalloc (->zpool): only used by zswap (-> compressed LRU pages)
* z3fold (->zpool): only used by zswap (-> compressed LRU pages)
Just imagine if we support migration of other kernel allocations, such as user page tables. The dump would be missing important information.
Once that happens, it will become a lot harder to judge whether CMA can be used or not. At least, the kernel could bail out/warn for these kernel configs.
Quoted 11 lines
There are five patches in this series:
The first adds a new ",cma" suffix to the recenly introduced generic crashkernel parsing code.
parse_crashkernel()takes one more argument to store the cma reservation size.The second patch implements
reserve_crashkernel_cma()which performs the reservation. If the requested size is not available in a single range, multiple smaller ranges will be reserved.
Quoted 5 lines
The third patch updates
Documentation/, explicitly mentioning the potential DMA corruption of the CMA-reserved memory.The fourth patch adds a short delay before booting the kdump kernel, allowing pending DMA transfers to finish.
What does "short" mean? At least in theory, long-term pinning is forbidden for MIGRATE_CMA, so we should not have such pages mapped into an iommu where DMA can happily keep going on for quite a while.
But that assumes that our old kernel is not buggy, and doesn't end up mapping these pages into an IOMMU where DMA will just continue. I recall that DRM might currently be a problem, described here [1].
If kdump starts not working as expected in case our old kernel is buggy, doesn't that partially destroy the purpose of kdump (-> debug bugs in the old kernel)?
[1] https://lore.kernel.org/all/[email protected]/T/#u
Signature · 3 lines
Cheers, David / dhildenb
Quoted 41 lines · David Hildenbrand
Quoted 25 lines · Jiri Bohac
Hi,
this series implements a way to reserve additional crash kernel memory using CMA.
Link to the v1 discussion:
https://lore.kernel.org/lkml/[email protected]/
See below for the changes since v1 and how concerns from the discussion have been addressed.Currently, all the memory for the crash kernel is not usable by the 1st (production) kernel. It is also unmapped so that it can't be corrupted by the fault that will eventually trigger the crash. This makes sense for the memory actually used by the kexec-loaded crash kernel image and initrd and the data prepared during the load (vmcoreinfo, ...). However, the reserved space needs to be much larger than that to provide enough run-time memory for the crash kernel and the kdump userspace. Estimating the amount of memory to reserve is difficult. Being too careful makes kdump likely to end in OOM, being too generous takes even more memory from the production system. Also, the reservation only allows reserving a single contiguous block (or two with the "low" suffix). I've seen systems where this fails because the physical memory is fragmented.
Quoted 11 lines
By reserving additional crashkernel memory from CMA, the main crashkernel reservation can be just large enough to fit the kernel and initrd image, minimizing the memory taken away from the production system. Most of the run-time memory for the crash kernel will be memory previously available to userspace in the production system. As this memory is no longer wasted, the reservation can be done with a generous margin, making kdump more reliable. Kernel memory that we need to preserve for dumping is never allocated from CMA. User data is typically not dumped by makedumpfile. When dumping of user data is intended this new CMA reservation cannot be used.
Hi,
Quoted 9 lines
I'll note that your comment about "user space" is currently the case, but will likely not hold in the long run. The assumption you are making is that only user-space memory will be allocated from
MIGRATE_CMA, which is not necessarily the case. Any movable allocation will end up in there.Besides LRU folios (user space memory and the pagecache), we already support migration of some kernel allocations using the non-lru migration framework. Such allocations (which use
__GFP_MOVABLE, see__SetPageMovable()) currently only include
* memory balloon: pages we never want to dump either way
* zsmalloc (->zpool): only used by zswap (-> compressed LRU pages)
* z3fold (->zpool): only used by zswap (-> compressed LRU pages)Just imagine if we support migration of other kernel allocations, such as user page tables. The dump would be missing important information.
IOMMUFD is a near-term candidate for user page tables with multi-stage iommu support with going through upstream review atm. Just saying, that David's case will be a norm in high-end VMs with performance-enhanced guest-driven iommu support (for GPUs).
Quoted 2 lines
Once that happens, it will become a lot harder to judge whether CMA can be used or not. At least, the kernel could bail out/warn for these kernel configs.
I don't think the aforementioned focus is to use CMA, but given its performance benefits, it won't take long to be the next perf improvement step taken.
Quoted 11 lines
Quoted 11 lines
There are five patches in this series:
The first adds a new ",cma" suffix to the recenly introduced generic crashkernel parsing code.
parse_crashkernel()takes one more argument to store the cma reservation size.The second patch implements
reserve_crashkernel_cma()which performs the reservation. If the requested size is not available in a single range, multiple smaller ranges will be reserved.
Quoted 9 lines
Quoted 5 lines
The third patch updates
Documentation/, explicitly mentioning the potential DMA corruption of the CMA-reserved memory.The fourth patch adds a short delay before booting the kdump kernel, allowing pending DMA transfers to finish.
What does "short" mean? At least in theory, long-term pinning is forbidden for
MIGRATE_CMA, so we should not have such pages mapped into an iommu where DMA can happily keep going on for quite a while.
Hmmm, in the case I mentioned above, should there be a kexec hook in multi-stage IOMMU support for the hypervisor/VMM to invalidate/shut-off stage 2 mappings asap (a multi-microsecond process) so DMA termination from VMs is stunted ? is that already done today (due to 'simple', single-stage, device assignment in a VM)?
Quoted 7 lines
But that assumes that our old kernel is not buggy, and doesn't end up mapping these pages into an IOMMU where DMA will just continue. I recall that DRM might currently be a problem, described here [1].
If kdump starts not working as expected in case our old kernel is buggy, doesn't that partially destroy the purpose of kdump (-> debug bugs in the old kernel)?
Quoted 51 lines · Donald Dutile
Quoted 48 lines · David Hildenbrand
Quoted 25 lines · Jiri Bohac
Hi,
this series implements a way to reserve additional crash kernel memory using CMA.
Link to the v1 discussion:
https://lore.kernel.org/lkml/[email protected]/
See below for the changes since v1 and how concerns from the discussion have been addressed.Currently, all the memory for the crash kernel is not usable by the 1st (production) kernel. It is also unmapped so that it can't be corrupted by the fault that will eventually trigger the crash. This makes sense for the memory actually used by the kexec-loaded crash kernel image and initrd and the data prepared during the load (vmcoreinfo, ...). However, the reserved space needs to be much larger than that to provide enough run-time memory for the crash kernel and the kdump userspace. Estimating the amount of memory to reserve is difficult. Being too careful makes kdump likely to end in OOM, being too generous takes even more memory from the production system. Also, the reservation only allows reserving a single contiguous block (or two with the "low" suffix). I've seen systems where this fails because the physical memory is fragmented.
Quoted 11 lines
By reserving additional crashkernel memory from CMA, the main crashkernel reservation can be just large enough to fit the kernel and initrd image, minimizing the memory taken away from the production system. Most of the run-time memory for the crash kernel will be memory previously available to userspace in the production system. As this memory is no longer wasted, the reservation can be done with a generous margin, making kdump more reliable. Kernel memory that we need to preserve for dumping is never allocated from CMA. User data is typically not dumped by makedumpfile. When dumping of user data is intended this new CMA reservation cannot be used.
Hi,
I'll note that your comment about "user space" is currently the case, but will likely not hold in the long run. The assumption you are making is that only user-space memory will be allocated from
MIGRATE_CMA, which is not necessarily the case. Any movable allocation will end up in there.Besides LRU folios (user space memory and the pagecache), we already support migration of some kernel allocations using the non-lru migration framework. Such allocations (which use
__GFP_MOVABLE, see__SetPageMovable()) currently only include
* memory balloon: pages we never want to dump either way
* zsmalloc (->zpool): only used by zswap (-> compressed LRU pages)
* z3fold (->zpool): only used by zswap (-> compressed LRU pages)
Quoted 4 lines
Quoted 2 lines
Just imagine if we support migration of other kernel allocations, such as user page tables. The dump would be missing important information.
IOMMUFD is a near-term candidate for user page tables with multi-stage iommu support with going through upstream review atm. Just saying, that David's case will be a norm in high-end VMs with performance-enhanced guest-driven iommu support (for GPUs).
Thank both for valuable inputs, David and Don. I agree that we may argue not every system have ballon or enabling swap for now, while future extending of migration on other kernel allocation could become obstacle we can't detour.
If we have known for sure this feature could be a bad code, we may need to stop it in advance.
Thoughts, Jiri?
Quoted 36 lines
Quoted 3 lines
Once that happens, it will become a lot harder to judge whether CMA can be used or not. At least, the kernel could bail out/warn for these kernel configs.
I don't think the aforementioned focus is to use CMA, but given its performance benefits, it won't take long to be the next perf improvement step taken.
Quoted 11 lines
Quoted 11 lines
There are five patches in this series:
The first adds a new ",cma" suffix to the recenly introduced generic crashkernel parsing code.
parse_crashkernel()takes one more argument to store the cma reservation size.The second patch implements
reserve_crashkernel_cma()which performs the reservation. If the requested size is not available in a single range, multiple smaller ranges will be reserved.Quoted 9 lines
Quoted 5 lines
The third patch updates
Documentation/, explicitly mentioning the potential DMA corruption of the CMA-reserved memory.The fourth patch adds a short delay before booting the kdump kernel, allowing pending DMA transfers to finish.
What does "short" mean? At least in theory, long-term pinning is forbidden for
MIGRATE_CMA, so we should not have such pages mapped into an iommu where DMA can happily keep going on for quite a while.Hmmm, in the case I mentioned above, should there be a kexec hook in multi-stage IOMMU support for the hypervisor/VMM to invalidate/shut-off stage 2 mappings asap (a multi-microsecond process) so DMA termination from VMs is stunted ? is that already done today (due to 'simple', single-stage, device assignment in a VM)?
Quoted 7 lines
But that assumes that our old kernel is not buggy, and doesn't end up mapping these pages into an IOMMU where DMA will just continue. I recall that DRM might currently be a problem, described here [1].
If kdump starts not working as expected in case our old kernel is buggy, doesn't that partially destroy the purpose of kdump (-> debug bugs in the old kernel)?
Quoted 57 lines · Baoquan He
Quoted 51 lines · Donald Dutile
Quoted 48 lines · David Hildenbrand
Quoted 25 lines · Jiri Bohac
Hi,
this series implements a way to reserve additional crash kernel memory using CMA.
Link to the v1 discussion:
https://lore.kernel.org/lkml/[email protected]/
See below for the changes since v1 and how concerns from the discussion have been addressed.Currently, all the memory for the crash kernel is not usable by the 1st (production) kernel. It is also unmapped so that it can't be corrupted by the fault that will eventually trigger the crash. This makes sense for the memory actually used by the kexec-loaded crash kernel image and initrd and the data prepared during the load (vmcoreinfo, ...). However, the reserved space needs to be much larger than that to provide enough run-time memory for the crash kernel and the kdump userspace. Estimating the amount of memory to reserve is difficult. Being too careful makes kdump likely to end in OOM, being too generous takes even more memory from the production system. Also, the reservation only allows reserving a single contiguous block (or two with the "low" suffix). I've seen systems where this fails because the physical memory is fragmented.
Quoted 11 lines
By reserving additional crashkernel memory from CMA, the main crashkernel reservation can be just large enough to fit the kernel and initrd image, minimizing the memory taken away from the production system. Most of the run-time memory for the crash kernel will be memory previously available to userspace in the production system. As this memory is no longer wasted, the reservation can be done with a generous margin, making kdump more reliable. Kernel memory that we need to preserve for dumping is never allocated from CMA. User data is typically not dumped by makedumpfile. When dumping of user data is intended this new CMA reservation cannot be used.
Hi,
I'll note that your comment about "user space" is currently the case, but will likely not hold in the long run. The assumption you are making is that only user-space memory will be allocated from
MIGRATE_CMA, which is not necessarily the case. Any movable allocation will end up in there.Besides LRU folios (user space memory and the pagecache), we already support migration of some kernel allocations using the non-lru migration framework. Such allocations (which use
__GFP_MOVABLE, see__SetPageMovable()) currently only include
* memory balloon: pages we never want to dump either way
* zsmalloc (->zpool): only used by zswap (-> compressed LRU pages)
* z3fold (->zpool): only used by zswap (-> compressed LRU pages)Quoted 4 lines
Quoted 2 lines
Just imagine if we support migration of other kernel allocations, such as user page tables. The dump would be missing important information.
IOMMUFD is a near-term candidate for user page tables with multi-stage iommu support with going through upstream review atm. Just saying, that David's case will be a norm in high-end VMs with performance-enhanced guest-driven iommu support (for GPUs).
Quoted 7 lines
Thank both for valuable inputs, David and Don. I agree that we may argue not every system have ballon or enabling swap for now, while future extending of migration on other kernel allocation could become obstacle we can't detour.
If we have known for sure this feature could be a bad code, we may need to stop it in advance.
Sorry for the late reply.
I think we just have to be careful to document it properly -- especially the shortcomings and that this feature might become a problem in the future. Movable user-space page tables getting placed on CMA memory would probably not be a problem if we don't care about ... user-space data either way.
The whole "Direct I/O takes max 1s" part is a bit shaky. Maybe it could be configurable how long to wait? 10s is certainly "safer".
But maybe, in the target use case: VMs, direct I/O will not be that common.
Signature · 3 lines
Cheers, David / dhildenb
On Wed 28-05-25 23:01:04, David Hildenbrand wrote:
[...]
Quoted 4 lines
I think we just have to be careful to document it properly -- especially the shortcomings and that this feature might become a problem in the future. Movable user-space page tables getting placed on CMA memory would probably not be a problem if we don't care about ... user-space data either way.
I think makedumpfile could refuse to capture a dump if userspace memory is requested to enforce this.
Quoted 2 lines
The whole "Direct I/O takes max 1s" part is a bit shaky. Maybe it could be configurable how long to wait? 10s is certainly "safer".
Quite honestly we will never know and rather than making this configurable I would go with reasonably large. Couple of seconds certainly do not matter for the kdump situations but I would go as far as minutes.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 10 lines · Michal Hocko
On Wed 28-05-25 23:01:04, David Hildenbrand wrote:
[...]Quoted 4 lines
I think we just have to be careful to document it properly -- especially the shortcomings and that this feature might become a problem in the future. Movable user-space page tables getting placed on CMA memory would probably not be a problem if we don't care about ... user-space data either way.
I think makedumpfile could refuse to capture a dump if userspace memory is requested to enforce this.
Quoted 6 lines
Quoted 2 lines
The whole "Direct I/O takes max 1s" part is a bit shaky. Maybe it could be configurable how long to wait? 10s is certainly "safer".
Quite honestly we will never know and rather than making this configurable I would go with reasonably large. Couple of seconds certainly do not matter for the kdump situations but I would go as far
typo
s@I would go@I would not go@
Quoted 1 line
as minutes.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 9 lines · Michal Hocko
On Wed 28-05-25 23:01:04, David Hildenbrand wrote:
[...]Quoted 4 lines
I think we just have to be careful to document it properly -- especially the shortcomings and that this feature might become a problem in the future. Movable user-space page tables getting placed on CMA memory would probably not be a problem if we don't care about ... user-space data either way.
I think makedumpfile could refuse to capture a dump if userspace memory is requested to enforce this.
Yeah, it will be tricky once we support placing other memory on CMA regions. E.g., there was the discussion of making some slab allocations movable.
But probably, in such a configuration, we would later simply refuse to active CMA kdump.
Quoted 8 lines
Quoted 3 lines
The whole "Direct I/O takes max 1s" part is a bit shaky. Maybe it could be configurable how long to wait? 10s is certainly "safer".
Quite honestly we will never know and rather than making this configurable I would go with reasonably large. Couple of seconds certainly do not matter for the kdump situations but I would go as far as minutes.
I recall that somebody raised that kdump downtime might be problematic (might affect service downtime?).
So I would just add a kconfig option with a default of 10s.
But even better if we can avoid the kconfig and just make it 10s for all setups.
I would not suggest having a different (runtime/boottime) way of configuring this.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Michal Hocko
On Wed 28-05-25 23:01:04, David Hildenbrand wrote:
[...]Quoted 4 lines
I think we just have to be careful to document it properly -- especially the shortcomings and that this feature might become a problem in the future. Movable user-space page tables getting placed on CMA memory would probably not be a problem if we don't care about ... user-space data either way.
Quoted 8 lines
Quoted 2 lines
I think makedumpfile could refuse to capture a dump if userspace memory is requested to enforce this.
Yeah, it will be tricky once we support placing other memory on CMA regions. E.g., there was the discussion of making some slab allocations movable.
But probably, in such a configuration, we would later simply refuse to active CMA kdump.
Or we can make the kdump CMA region more special and only allow GFP_HIGHUSER_MOVABLE allocations from that. Anyaway I think we should deal with this once we get there.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
The whole "Direct I/O takes max 1s" part is a bit shaky. Maybe it could be configurable how long to wait? 10s is certainly "safer".
Quite honestly we will never know and rather than making this configurable I would go with reasonably large. Couple of seconds certainly do not matter for the kdump situations but I would go as far as minutes.
Quoted 4 lines
I recall that somebody raised that kdump downtime might be problematic (might affect service downtime?).
So I would just add a kconfig option with a default of 10s.
kconfig option usually doesn't really work for distro kernels. I am personally not really keen on having a tuning knob because there is a risk of cargo cult based tuning we have seen in other areas. That might make it hard to remove the knob later on. Fundamentally we should have 2 situations though. Either we know that the HW is sane and then we shouldn't really need any sleep or the HW might misbehave and then we need to wait _some_ time. If our initial guess is incorrect then we can always increase it and we would learn about that through bug reports.
All that being said I would go with an additional parameter to the kdump cma setup - e.g. cma_sane_dma that would skip waiting and use 10s otherwise. That would make the optimized behavior opt in, we do not need to support all sorts of timeouts and also learn if this is not sufficient.
Makes sense?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 15 lines · Michal Hocko
Quoted 14 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko
On Wed 28-05-25 23:01:04, David Hildenbrand wrote:
[...]Quoted 4 lines
I think we just have to be careful to document it properly -- especially the shortcomings and that this feature might become a problem in the future. Movable user-space page tables getting placed on CMA memory would probably not be a problem if we don't care about ... user-space data either way.
I think makedumpfile could refuse to capture a dump if userspace memory is requested to enforce this.
Yeah, it will be tricky once we support placing other memory on CMA regions. E.g., there was the discussion of making some slab allocations movable.
Quoted 6 lines
Quoted 2 lines
But probably, in such a configuration, we would later simply refuse to active CMA kdump.
Or we can make the kdump CMA region more special and only allow
GFP_HIGHUSER_MOVABLEallocations from that. Anyaway I think we should deal with this once we get there.
Might be doable. When migrating (e.g., compacting) pages we'd have to make sure to also not migrate these pages into the CMA regions. Might be a bit more tricky, but likely solvable.
Quoted 14 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
The whole "Direct I/O takes max 1s" part is a bit shaky. Maybe it could be configurable how long to wait? 10s is certainly "safer".
Quite honestly we will never know and rather than making this configurable I would go with reasonably large. Couple of seconds certainly do not matter for the kdump situations but I would go as far as minutes.
Quoted 4 lines
I recall that somebody raised that kdump downtime might be problematic (might affect service downtime?).
So I would just add a kconfig option with a default of 10s.
Quoted 8 lines
kconfig option usually doesn't really work for distro kernels. I am personally not really keen on having a tuning knob because there is a risk of cargo cult based tuning we have seen in other areas. That might make it hard to remove the knob later on. Fundamentally we should have 2 situations though. Either we know that the HW is sane and then we shouldn't really need any sleep or the HW might misbehave and then we need to wait _some_ time. If our initial guess is incorrect then we can always increase it and we would learn about that through bug reports.
kconfigs are usually much easier to alter/remove than other tunables in my experience.
But yeah, it would have to go for the setting that works for all supported hw (iow, conservative timeout).
Quoted 8 lines
All that being said I would go with an additional parameter to the kdump cma setup - e.g.
cma_sane_dmathat would skip waiting and use 10s otherwise. That would make the optimized behavior opt in, we do not need to support all sorts of timeouts and also learn if this is not sufficient.Makes sense?
Just so I understand correctly, you mean extending the "crashkernel=" option with a boolean parameter? If set, e.g., wait 1s, otherwise magic number 10?
Signature · 3 lines
Cheers, David / dhildenb
Quoted 1 line · David Hildenbrand
On 30.05.25 10:28, Michal Hocko wrote:
[...]
Quoted 10 lines
Quoted 7 lines
All that being said I would go with an additional parameter to the kdump cma setup - e.g.
cma_sane_dmathat would skip waiting and use 10s otherwise. That would make the optimized behavior opt in, we do not need to support all sorts of timeouts and also learn if this is not sufficient.Makes sense?
Just so I understand correctly, you mean extending the "crashkernel=" option with a boolean parameter? If set, e.g., wait 1s, otherwise magic number 10?
crashkernel=1G,cma,cma_sane_dma # no wait on transition crashkernel=1G,cma # wait on transition with e.g. 10s timeout
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 9 lines · Michal Hocko
Quoted 1 line · David Hildenbrand
On 30.05.25 10:28, Michal Hocko wrote:
[...]
Quoted 6 lines
Quoted 6 lines
All that being said I would go with an additional parameter to the kdump cma setup - e.g.
cma_sane_dmathat would skip waiting and use 10s otherwise. That would make the optimized behavior opt in, we do not need to support all sorts of timeouts and also learn if this is not sufficient.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Makes sense?
Just so I understand correctly, you mean extending the "crashkernel=" option with a boolean parameter? If set, e.g., wait 1s, otherwise magic number 10?
crashkernel=1G,cma,cma_sane_dma # no wait on transition
But is no wait ok? I mean, any O_DIRECT with any device would at least take a bit, no?
Of course, there is a short time between the crash and actually triggerying kdump.
Quoted 1 line
crashkernel=1G,cma # wait on transition with e.g. 10s timeout
In general, would work for me.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 15 lines · David Hildenbrand
Quoted 14 lines · Michal Hocko
Quoted 1 line · David Hildenbrand
On 30.05.25 10:28, Michal Hocko wrote:
[...]
Quoted 10 lines
Quoted 7 lines
All that being said I would go with an additional parameter to the kdump cma setup - e.g.
cma_sane_dmathat would skip waiting and use 10s otherwise. That would make the optimized behavior opt in, we do not need to support all sorts of timeouts and also learn if this is not sufficient.Makes sense?
Just so I understand correctly, you mean extending the "crashkernel=" option with a boolean parameter? If set, e.g., wait 1s, otherwise magic number 10?
Quoted 7 lines
Quoted 1 line
crashkernel=1G,cma,cma_sane_dma # no wait on transition
But is no wait ok? I mean, any
O_DIRECTwith any device would at least take a bit, no?Of course, there is a short time between the crash and actually triggerying kdump.
This is something we can test for and if we need a short timeout in this case as well then it is just trivial to add it. I am much more concerned about those potentially unpredictable DMA transfers that could take too long and it is impossible to test for those and therefore we need to overshoot.
Quoted 8 lines
Quoted 1 line
crashkernel=1G,cma # wait on transition with e.g. 10s timeout
In general, would work for me.
Signature · 3 lines
Cheers, David / dhildenb
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 21 lines · Michal Hocko
Quoted 20 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko
Quoted 1 line · David Hildenbrand
On 30.05.25 10:28, Michal Hocko wrote:
[...]
Quoted 6 lines
Quoted 6 lines
All that being said I would go with an additional parameter to the kdump cma setup - e.g.
cma_sane_dmathat would skip waiting and use 10s otherwise. That would make the optimized behavior opt in, we do not need to support all sorts of timeouts and also learn if this is not sufficient.Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Makes sense?
Just so I understand correctly, you mean extending the "crashkernel=" option with a boolean parameter? If set, e.g., wait 1s, otherwise magic number 10?
crashkernel=1G,cma,cma_sane_dma # no wait on transition
But is no wait ok? I mean, any
O_DIRECTwith any device would at least take a bit, no?
Quoted 8 lines
Quoted 2 lines
Of course, there is a short time between the crash and actually triggerying kdump.
This is something we can test for and if we need a short timeout in this case as well then it is just trivial to add it. I am much more concerned about those potentially unpredictable DMA transfers that could take too long and it is impossible to test for those and therefore we need to overshoot.
Agreed.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 20 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko
Quoted 1 line · David Hildenbrand
On 30.05.25 10:28, Michal Hocko wrote:
[...]
Quoted 6 lines
Quoted 6 lines
All that being said I would go with an additional parameter to the kdump cma setup - e.g.
cma_sane_dmathat would skip waiting and use 10s otherwise. That would make the optimized behavior opt in, we do not need to support all sorts of timeouts and also learn if this is not sufficient.Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Makes sense?
Just so I understand correctly, you mean extending the "crashkernel=" option with a boolean parameter? If set, e.g., wait 1s, otherwise magic number 10?
crashkernel=1G,cma,cma_sane_dma # no wait on transition
But is no wait ok? I mean, any
O_DIRECTwith any device would at least take a bit, no?
Quoted 6 lines
Of course, there is a short time between the crash and actually triggerying kdump.
Quoted 1 line
crashkernel=1G,cma # wait on transition with e.g. 10s timeout
In general, would work for me.
I don't like extending the crashkernel= syntax like this. It would make hooking into the generic parsing code in parse_crashkernel() really ugly. The syntax is already convoluted as is and hard enough to explain in the documentation.
Also I don't see how adding a boolean knob is better than adding one that allows setting any arbitrary timeout. It has less flexibility and all the drawbacks of having an extra knob.
I am inclined to just setting the fixed delay to 10s for now and adding a sysfs knob later if someone asks for it.
Would that work for you?
If you don't have other objections to the v3 series, I'll just update it for v6.15 and post again a v4 with the 10s timeout...
Thanks for your input!
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
Quoted 24 lines · Jiri Bohac
Quoted 23 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko
Quoted 1 line · David Hildenbrand
On 30.05.25 10:28, Michal Hocko wrote:
[...]
Quoted 6 lines
Quoted 6 lines
All that being said I would go with an additional parameter to the kdump cma setup - e.g.
cma_sane_dmathat would skip waiting and use 10s otherwise. That would make the optimized behavior opt in, we do not need to support all sorts of timeouts and also learn if this is not sufficient.Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Makes sense?
Just so I understand correctly, you mean extending the "crashkernel=" option with a boolean parameter? If set, e.g., wait 1s, otherwise magic number 10?
crashkernel=1G,cma,cma_sane_dma # no wait on transition
But is no wait ok? I mean, any
O_DIRECTwith any device would at least take a bit, no?Of course, there is a short time between the crash and actually triggerying kdump.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
crashkernel=1G,cma # wait on transition with e.g. 10s timeout
In general, would work for me.
I don't like extending the crashkernel= syntax like this. It would make hooking into the generic parsing code in
parse_crashkernel()really ugly. The syntax is already convoluted as is and hard enough to explain in the documentation.
Would another boolean flag (on top of the other one you are adding) really make this significantly more ugly?
Quoted 4 lines
Also I don't see how adding a boolean knob is better than adding one that allows setting any arbitrary timeout. It has less flexibility and all the drawbacks of having an extra knob.
I guess Michals point is that specifying the higher-level problem and giving less flexibility mioght actually be less confusing for users.
Quoted 5 lines
I am inclined to just setting the fixed delay to 10s for now and adding a sysfs knob later if someone asks for it.
Would that work for you?
Sure. We could always add such a flag later if it's really a problem for someone.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 1 line · David Hildenbrand
On 30.05.25 11:34, Jiri Bohac wrote:
[...]
Quoted 7 lines
Quoted 4 lines
I am inclined to just setting the fixed delay to 10s for now and adding a sysfs knob later if someone asks for it.
Would that work for you?
Sure. We could always add such a flag later if it's really a problem for someone.
Yes, no objection with the most conservative approach first.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 12 lines · David Hildenbrand
Quoted 12 lines
Quoted 11 lines
Quoted 1 line
crashkernel=1G,cma,cma_sane_dma # no wait on transition
But is no wait ok? I mean, any
O_DIRECTwith any device would at least take a bit, no?Of course, there is a short time between the crash and actually triggerying kdump.
Quoted 1 line
crashkernel=1G,cma # wait on transition with e.g. 10s timeout
In general, would work for me.
Quoted 7 lines
Quoted 4 lines
I don't like extending the crashkernel= syntax like this. It would make hooking into the generic parsing code in
parse_crashkernel()really ugly. The syntax is already convoluted as is and hard enough to explain in the documentation.Would another boolean flag (on top of the other one you are adding) really make this significantly more ugly?
the current code does not split the parameter by commas and treat the part as boolean flags.
Both ",cma" and ",cma,cma_sane_dma" (and possibly ",cma_sane_dma,cma") would need to be added to suffix_tbl[] (carefully thinking about the order because one is a prefix of the other); then handled almost the same except setting the flag.
Also I think using the command line is way less flexible than sysfs. E.g. the userspace tool loading the crash kernel (kdump) may want to decide if the hardware is sane using its own whitelist/blacklist...
Quoted 7 lines
Quoted 4 lines
I am inclined to just setting the fixed delay to 10s for now and adding a sysfs knob later if someone asks for it.
Would that work for you?
Sure. We could always add such a flag later if it's really a problem for someone.
OK, thanks! Will post the v4 shortly.
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
Quoted 4 lines · David Hildenbrand
I think we just have to be careful to document it properly -- especially the shortcomings and that this feature might become a problem in the future. Movable user-space page tables getting placed on CMA memory would probably not be a problem if we don't care about ... user-space data either way.
Agreed; in the v3 series [1] I amended the documentation part [2] to explicitly mention that kernel movable allocations could be missing from the vmcore.
The risks associated with pending DMA are also mentioned.
Is there anything you're still missing from the v3 documentation?
Quoted 2 lines
The whole "Direct I/O takes max 1s" part is a bit shaky. Maybe it could be configurable how long to wait? 10s is certainly "safer".
I have nothing against making this configurable, or just setting the fixed/default delay to 10s. Which would you prefer? Would you prefer a command-line option, config option or a sysfs file?
Thanks!
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia
Quoted 27 lines · David Hildenbrand
Quoted 12 lines · Jiri Bohac
By reserving additional crashkernel memory from CMA, the main crashkernel reservation can be just large enough to fit the kernel and initrd image, minimizing the memory taken away from the production system. Most of the run-time memory for the crash kernel will be memory previously available to userspace in the production system. As this memory is no longer wasted, the reservation can be done with a generous margin, making kdump more reliable. Kernel memory that we need to preserve for dumping is never allocated from CMA. User data is typically not dumped by makedumpfile. When dumping of user data is intended this new CMA reservation cannot be used.
I'll note that your comment about "user space" is currently the case, but will likely not hold in the long run. The assumption you are making is that only user-space memory will be allocated from
MIGRATE_CMA, which is not necessarily the case. Any movable allocation will end up in there.Besides LRU folios (user space memory and the pagecache), we already support migration of some kernel allocations using the non-lru migration framework. Such allocations (which use
__GFP_MOVABLE, see__SetPageMovable()) currently only include
* memory balloon: pages we never want to dump either way
* zsmalloc (->zpool): only used by zswap (-> compressed LRU pages)
* z3fold (->zpool): only used by zswap (-> compressed LRU pages)
Quoted 6 lines
Just imagine if we support migration of other kernel allocations, such as user page tables. The dump would be missing important information.
Once that happens, it will become a lot harder to judge whether CMA can be used or not. At least, the kernel could bail out/warn for these kernel configs.
Thanks for ponting this out. I still don't see this as a roadblock for my primary usecase of the CMA reservation:
get at least some (less reliable and potentially
less useful) kdump where the user is not prepared to sacrifice the memory needed for the standard reservation and where the only other option is no kdump at all.
Still a lot can be analyzed with a vmcore that is missing those __GFP_MOVABLE pages. Even if/when some user page tables are missing.
I'll send a v3 with the documenatation part updated to better describe this.
Quoted 7 lines
Quoted 2 lines
The fourth patch adds a short delay before booting the kdump kernel, allowing pending DMA transfers to finish.
What does "short" mean? At least in theory, long-term pinning is forbidden for
MIGRATE_CMA, so we should not have such pages mapped into an iommu where DMA can happily keep going on for quite a while.
See patch 4/5 in the series:
I propose 1 second, which is a negligible time from the kdump POV but I assume it should be plenty enough for non-long-term pins in MIGRATE_CMA.
Quoted 7 lines
But that assumes that our old kernel is not buggy, and doesn't end up mapping these pages into an IOMMU where DMA will just continue. I recall that DRM might currently be a problem, described here [1].
If kdump starts not working as expected in case our old kernel is buggy, doesn't that partially destroy the purpose of kdump (-> debug bugs in the old kernel)?
Again, this is meant as a kind of "lightweight best effort kdump". If there is a bug that causes the crash _and_ a bug in a driver that hogs MIGRATE_CMA and maps it into IOMMU then this lightweight kdump may break. Then it's time to sacrifice more memory and use a normal crashkernel reservation.
It's not like any bug in the old kernel will break it. It's a very specific kind of bug that can potentially break it.
I see this whole thing as particularly useful for VMs. Unlike big physical machines, where taking away a couple hundred MBs of memory for kdump does not really hurt, a VM can ideally be given just enough memory for its particular task. This can often be less than 1 GB. Proper kdump reservation needs a couple hundred MBs, so a very large proportion of the VM memory. In case of a virtualization host running hundreds or thousands such VMs this means a huge waste of memory. And VMs often don't use too many drivers for real hardware, decreasing the risk of hitting a buggy driver like this.
Thanks,
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, Prague, Czechia