Fix kdump failures with crashkernel=high
Subsystems (2)
cover Fix kdump failures with crashkernel=high
Hi,
here is a patch-set to fix failed kdump kernel boots when the systems was booted with crashkernel=X,high. On those systems the kernel allocates only 72MiB of low-memory for DMA buffers, which showed to be too low on some systems.
The problem is that 64MiB of the low-memory is allocated by swiotlb, leaving 8MB for the page-allocator. But swiotlb tries to allocate DMA memory from the page-allocator first, which fails pretty fast in the boot sequence, causing warnings. This patch-set removes these warnings.
But even the 64MiB for swiotlb are eaten up on some systems, so that the default of low-memory allocated for the crash-kernel is increase from 72MB to 256MB (only changing the defaults, can still be overwritten by crashkernel=X,low).
This number comes from experiments on the affected systems, 128MiB low-memory was still not enough there, thus I set the value to 256MiB to fix the issues.
Any feedback appreciated.
Thanks,
Joerg
Joerg Roedel (3):
swiotlb: Warn on allocation failure in swiotlb_alloc_coherent x86, swiotlb: Try coherent allocations with __GFP_NOWARN x86, crash: Allocate enough low-mem when crashkernel=high
arch/x86/kernel/pci-swiotlb.c | 8 ++++++++ arch/x86/kernel/setup.c | 5 ++++- lib/swiotlb.c | 11 +++++++++-- 3 files changed, 21 insertions(+), 3 deletions(-)
Signature · 1 line
1.9.1
PATCH 1/3 swiotlb: Warn on allocation failure in swiotlb_alloc_coherent
From: Joerg Roedel <[email protected]>
Print a warning when all allocation tries have been failed and the function is about to return NULL. This prepares for calling the function with __GFP_NOWARN to suppress allocation failure warnings before all fall-backs have failed.
--- lib/swiotlb.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
lib/swiotlb.c +9 -2
@@ -655,7 +655,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,*/ phys_addr_t paddr =map_single (hwdev,0 , size,DMA_FROM_DEVICE );if (paddr ==SWIOTLB_MAP_ERROR ) -return NULL ; + goto err_warn; ret =phys_to_virt (paddr); dev_addr =phys_to_dma (hwdev, paddr);
@@ -669,7 +669,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,/* DMA_TO_DEVICE to avoid memcpy in unmap_single */ swiotlb_tbl_unmap_single (hwdev, paddr, size,DMA_TO_DEVICE ); -return NULL ; + goto err_warn; } }
@@ -677,6 +677,13 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,memset (ret,0 , size);return ret; + +err_warn: +pr_warn ("swiotlb: coherent allocation failed for device %s size=%zu\n" , +dev_name (hwdev), size); +dump_stack (); + +return NULL ; }EXPORT_SYMBOL (swiotlb_alloc_coherent);
Signature · 1 line
1.9.1
Re: [PATCH 1/3] swiotlb: Warn on allocation failure in swiotlb_alloc_coherent
Quoted 7 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
Print a warning when all allocation tries have been failed and the function is about to return NULL. This prepares for calling the function with
__GFP_NOWARNto suppress allocation failure warnings before all fall-backs have failed.
This can be quite noisy. Especially the dump-stack.
Perhaps have this trigger is the 'verbose' or 'debug' (new) parameter would be added to the 'swiotlb' one?
Quoted 45 lines
Signed-off-by Joerg Roedel <[email protected]>--- lib/swiotlb.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)lib/swiotlb.c +9
-2@@ -655,7 +655,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,*/ phys_addr_t paddr =map_single (hwdev,0 , size,DMA_FROM_DEVICE );if (paddr ==SWIOTLB_MAP_ERROR ) -return NULL ; + goto err_warn; ret =phys_to_virt (paddr); dev_addr =phys_to_dma (hwdev, paddr);@@ -669,7 +669,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,/* DMA_TO_DEVICE to avoid memcpy in unmap_single */ swiotlb_tbl_unmap_single (hwdev, paddr, size,DMA_TO_DEVICE ); -return NULL ; + goto err_warn; } }@@ -677,6 +677,13 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,memset (ret,0 , size);return ret; + +err_warn: +pr_warn ("swiotlb: coherent allocation failed for device %s size=%zu\n" , +dev_name (hwdev), size); +dump_stack (); + +return NULL ; }EXPORT_SYMBOL (swiotlb_alloc_coherent);Signature · 1 line
1.9.1
Re: [PATCH 1/3] swiotlb: Warn on allocation failure in swiotlb_alloc_coherent
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
Print a warning when all allocation tries have been failed and the function is about to return NULL. This prepares for calling the function with
__GFP_NOWARNto suppress allocation failure warnings before all fall-backs have failed.This can be quite noisy. Especially the dump-stack.
Well, this is as noisy as the dump_stack()s from the page-allocator when the first allocation try fails. The goal of the first two patches in this series is to only print a warning (with stack-trace) when alloc_coherent failed, and not when only an intermediate step failed that has a fall-back anyway (and might thus be no real problem).
Joerg
Re: [PATCH 1/3] swiotlb: Warn on allocation failure in swiotlb_alloc_coherent
Quoted 10 lines · Joerg Roedel
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
Print a warning when all allocation tries have been failed and the function is about to return NULL. This prepares for calling the function with
__GFP_NOWARNto suppress allocation failure warnings before all fall-backs have failed.
Quoted 4 lines
Quoted 1 line
This can be quite noisy. Especially the dump-stack.
Well, this is as noisy as the dump_stack()s from the page-allocator when the first allocation try fails. The goal of the first two patches in
Right, on the first allocation. Subsequent allocations won't be so noisy in the page-allocator (I think?). While this will be noisy on subsequent ones.
Quoted 7 lines
this series is to only print a warning (with stack-trace) when
alloc_coherentfailed, and not when only an intermediate step failed that has a fall-back anyway (and might thus be no real problem).Joerg
Re: [PATCH 1/3] swiotlb: Warn on allocation failure in swiotlb_alloc_coherent
Quoted 13 lines · Konrad Rzeszutek Wilk
Quoted 12 lines · Joerg Roedel
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
Print a warning when all allocation tries have been failed and the function is about to return NULL. This prepares for calling the function with
__GFP_NOWARNto suppress allocation failure warnings before all fall-backs have failed.This can be quite noisy. Especially the dump-stack.
Quoted 7 lines
Quoted 2 lines
Well, this is as noisy as the dump_stack()s from the page-allocator when the first allocation try fails. The goal of the first two patches in
Right, on the first allocation. Subsequent allocations won't be so noisy in the page-allocator (I think?).
From the code in
mm/page_alloc.c(functionwarn_alloc_failed) it doesn't
look like a one-time warning. The dmesg I have seen from a failing kernel also shows a lot of these messages.
So having the warning at the end of swiotlb_alloc_coherent won't be any more noisy than the (removed) warnings from the page allocator.
Joerg
Re: [PATCH 1/3] swiotlb: Warn on allocation failure in swiotlb_alloc_coherent
Quoted 20 lines · Joerg Roedel
Quoted 18 lines · Konrad Rzeszutek Wilk
Quoted 10 lines · Joerg Roedel
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
Print a warning when all allocation tries have been failed and the function is about to return NULL. This prepares for calling the function with
__GFP_NOWARNto suppress allocation failure warnings before all fall-backs have failed.Quoted 4 lines
Quoted 1 line
This can be quite noisy. Especially the dump-stack.
Well, this is as noisy as the dump_stack()s from the page-allocator when the first allocation try fails. The goal of the first two patches in
Right, on the first allocation. Subsequent allocations won't be so noisy in the page-allocator (I think?).
Quoted 6 lines
Quoted 1 line
From the code in
mm/page_alloc.c(functionwarn_alloc_failed) it doesn'tlook like a one-time warning. The dmesg I have seen from a failing kernel also shows a lot of these messages.
So having the warning at the end of
swiotlb_alloc_coherentwon't be any more noisy than the (removed) warnings from the page allocator.
OK, then Acked-by: Konrad Rzeszutek Wilk <[email protected]>
Quoted 4 lines
Joerg
PATCH 2/3 x86, swiotlb: Try coherent allocations with __GFP_NOWARN
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
When we boot a kdump kernel in high memory, there is by default only 72MB of low memory available. The swiotlb code takes 64MB of it (by default) so that there are only 8MB left to allocate from. On systems with many devices this causes page allocator warnings from dma_generic_alloc_coherent():
systemd-udevd: page allocation failure: order:0, mode:0x280d4
CPU: 0 PID: 197 Comm: systemd-udevd Tainted: G W 3.12.28-4-default #1 Hardware name: HP ProLiant DL980 G7, BIOS P66 07/30/2012 ffff8800781335e0 ffffffff8150b1db 00000000000280d4 ffffffff8113af90 0000000000000000 0000000000000000 ffff88007efdbb00 0000000100000000 0000000000000000 0000000000000000 0000000000000000 0000000000000001 Call Trace: [<ffffffff8100467d>] dump_trace+0x7d/0x2d0 [<ffffffff81004964>] show_stack_log_lvl+0x94/0x170 [<ffffffff81005d91>] show_stack+0x21/0x50 [<ffffffff8150b1db>] dump_stack+0x41/0x51 [<ffffffff8113af90>] warn_alloc_failed+0xf0/0x160 [<ffffffff8150763a>] __alloc_pages_slowpath+0x72f/0x796 [<ffffffff8113ee7a>] __alloc_pages_nodemask+0x1ea/0x210 [<ffffffff81008256>] dma_generic_alloc_coherent+0x96/0x140 [<ffffffff8103fccc>] x86_swiotlb_alloc_coherent+0x1c/0x50 [<ffffffffa048ae5b>] ttm_dma_pool_alloc_new_pages+0xab/0x320 [ttm] [<ffffffffa048bc6e>] ttm_dma_populate+0x3ce/0x640 [ttm] [<ffffffffa0486>] ttm_tt_bind+0x36/0x60 [ttm] [<ffffffffa0484faf>] ttm_bo_handle_move_mem+0x55f/0x5c0 [ttm] [<ffffffffa0485be5>] ttm_bo_move_buffer+0x105/0x130 [ttm] [<ffffffffa0485cd1>] ttm_bo_validate+0xc1/0x130 [ttm] [<ffffffffa0485f8b>] ttm_bo_init+0x24b/0x400 [ttm] [<ffffffffa054f8bc>] radeon_bo_create+0x16c/0x200 [radeon] [<ffffffffa0563c8e>] radeon_ring_init+0x11e/0x2b0 [radeon] [<ffffffffa056c143>] r100_cp_init+0x123/0x5b0 [radeon] [<ffffffffa056e8e4>] r100_startup+0x194/0x230 [radeon] [<ffffffffa056ece3>] r100_init+0x223/0x410 [radeon] [<ffffffffa053495f>] radeon_device_init+0x6af/0x830 [radeon] [<ffffffffa0536979>] radeon_driver_load_kms+0x89/0x180 [radeon] [<ffffffffa04eeb31>] drm_get_pci_dev+0x121/0x2f0 [drm] [<ffffffff812d3ec9>ocal_pci_probe+0x39/0x60 [<ffffffff812d51e9>] pci_device_probe+0xa9/0x120 [<ffffffff8139871d>] driver_probe_device+0x9d/0x3d0 [<ffffffff81398b1b>] __driver_attach+0x8b/0x90 [<ffffffff8139667b>] bus_for_each_dev+0x5b/0x90 [<ffffffff81397cd8>] bus_add_driver+0x1f8/0x2c0 [<ffffffff8139911b>] driver_register+0x5b/0xe0 [<ffffffff810002c2>] do_one_initcall+0xf2/0x1a0 [<ffffffff810c8837>] load_module+0x1207/0x1c70 [<ffffffff810c93f5>] SYSC_finit_module+0x75/0xa0 [<ffffffff81519329>] system_call_fastpath+0x16/0x1b [<00007fac533d2789>] 0x7fac533d2788
After these warnings the code enters a fall-back path and allocated directly from the swiotlb aperture in the end. So remove these warnings as this is not a fatal error.
--- arch/x86/kernel/pci-swiotlb.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kernel/pci-swiotlb.c +8 -0
@@ -20,6 +20,14 @@ void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, {void *vaddr; +/* +* When booting a kdump kernel in high memory these allocations are very +* likely to fail, as there are by default only 8MB of low memory to +* allocate from. So disable the warnings from the allocator when this +* happens. SWIOTLB also implements fall-backs for failed allocations. +*/ + flags |= __GFP_NOWARN; + vaddr =dma_generic_alloc_coherent (hwdev, size, dma_handle, flags, attrs);if (vaddr)
Signature · 1 line
1.9.1
Re: [PATCH 2/3] x86, swiotlb: Try coherent allocations with __GFP_NOWARN
Quoted 66 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
When we boot a kdump kernel in high memory, there is by default only 72MB of low memory available. The swiotlb code takes 64MB of it (by default) so that there are only 8MB left to allocate from. On systems with many devices this causes page allocator warnings from
dma_generic_alloc_coherent():systemd-udevd: page allocation failure: order:0, mode:0x280d4
CPU: 0 PID: 197 Comm: systemd-udevd Tainted: G W 3.12.28-4-default #1 Hardware name: HP ProLiant DL980 G7, BIOS P66 07/30/2012 ffff8800781335e0 ffffffff8150b1db 00000000000280d4 ffffffff8113af90 0000000000000000 0000000000000000 ffff88007efdbb00 0000000100000000 0000000000000000 0000000000000000 0000000000000000 0000000000000001 Call Trace: [<ffffffff8100467d>] dump_trace+0x7d/0x2d0 [<ffffffff81004964>] show_stack_log_lvl+0x94/0x170 [<ffffffff81005d91>] show_stack+0x21/0x50 [<ffffffff8150b1db>] dump_stack+0x41/0x51 [<ffffffff8113af90>] warn_alloc_failed+0xf0/0x160 [<ffffffff8150763a>] __alloc_pages_slowpath+0x72f/0x796 [<ffffffff8113ee7a>] __alloc_pages_nodemask+0x1ea/0x210 [<ffffffff81008256>] dma_generic_alloc_coherent+0x96/0x140 [<ffffffff8103fccc>] x86_swiotlb_alloc_coherent+0x1c/0x50 [<ffffffffa048ae5b>] ttm_dma_pool_alloc_new_pages+0xab/0x320 [ttm] [<ffffffffa048bc6e>] ttm_dma_populate+0x3ce/0x640 [ttm] [<ffffffffa0486>] ttm_tt_bind+0x36/0x60 [ttm] [<ffffffffa0484faf>] ttm_bo_handle_move_mem+0x55f/0x5c0 [ttm] [<ffffffffa0485be5>] ttm_bo_move_buffer+0x105/0x130 [ttm] [<ffffffffa0485cd1>] ttm_bo_validate+0xc1/0x130 [ttm] [<ffffffffa0485f8b>] ttm_bo_init+0x24b/0x400 [ttm] [<ffffffffa054f8bc>] radeon_bo_create+0x16c/0x200 [radeon] [<ffffffffa0563c8e>] radeon_ring_init+0x11e/0x2b0 [radeon] [<ffffffffa056c143>] r100_cp_init+0x123/0x5b0 [radeon] [<ffffffffa056e8e4>] r100_startup+0x194/0x230 [radeon] [<ffffffffa056ece3>] r100_init+0x223/0x410 [radeon] [<ffffffffa053495f>] radeon_device_init+0x6af/0x830 [radeon] [<ffffffffa0536979>] radeon_driver_load_kms+0x89/0x180 [radeon] [<ffffffffa04eeb31>] drm_get_pci_dev+0x121/0x2f0 [drm] [<ffffffff812d3ec9>ocal_pci_probe+0x39/0x60 [<ffffffff812d51e9>] pci_device_probe+0xa9/0x120 [<ffffffff8139871d>] driver_probe_device+0x9d/0x3d0 [<ffffffff81398b1b>] __driver_attach+0x8b/0x90 [<ffffffff8139667b>] bus_for_each_dev+0x5b/0x90 [<ffffffff81397cd8>] bus_add_driver+0x1f8/0x2c0 [<ffffffff8139911b>] driver_register+0x5b/0xe0 [<ffffffff810002c2>] do_one_initcall+0xf2/0x1a0 [<ffffffff810c8837>] load_module+0x1207/0x1c70 [<ffffffff810c93f5>] SYSC_finit_module+0x75/0xa0 [<ffffffff81519329>] system_call_fastpath+0x16/0x1b [<00007fac533d2789>] 0x7fac533d2788After these warnings the code enters a fall-back path and allocated directly from the swiotlb aperture in the end. So remove these warnings as this is not a fatal error.
Signed-off-by Joerg Roedel <[email protected]>--- arch/x86/kernel/pci-swiotlb.c | 8 ++++++++ 1 file changed, 8 insertions(+)arch/x86/kernel/pci-swiotlb.c +0
-0
arch/x86/kernel/pci-swiotlb.c void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, · line 29
Quoted 10 lines
arch/x86/kernel/pci-swiotlb.c +7
-0@@ -20,6 +20,14 @@ void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, {void *vaddr; +/* +* When booting a kdump kernel in high memory these allocations are very +* likely to fail, as there are by default only 8MB of low memory to +* allocate from. So disable the warnings from the allocator when this +* happens. SWIOTLB also implements fall-backs for failed allocations. +*/ + flags |= __GFP_NOWARN;
Should this perhaps then have 'if (kdump_kernel)' around it since the use-case seems to be kdump related?
Quoted 7 lines
arch/x86/kernel/pci-swiotlb.c +1
-0+ vaddr =dma_generic_alloc_coherent (hwdev, size, dma_handle, flags, attrs);if (vaddr)Signature · 1 line
1.9.1
Re: [PATCH 2/3] x86, swiotlb: Try coherent allocations with __GFP_NOWARN
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 8 lines · Joerg Roedel
arch/x86/kernel/pci-swiotlb.c +0
-0@@ -20,6 +20,14 @@ void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, {void *vaddr;
Quoted 10 lines
arch/x86/kernel/pci-swiotlb.c void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, · line 29
Quoted 7 lines
arch/x86/kernel/pci-swiotlb.c +7
-0+/* +* When booting a kdump kernel in high memory these allocations are very +* likely to fail, as there are by default only 8MB of low memory to +* allocate from. So disable the warnings from the allocator when this +* happens. SWIOTLB also implements fall-backs for failed allocations. +*/ + flags |= __GFP_NOWARN;Should this perhaps then have 'if (
kdump_kernel)' around it since the use-case seems to be kdump related?
Hmm, I don't think this is entirely kdump specific. It can also be triggered on a non-kdump kernel, it is just much more unlikely. But maybe I should change the comment to something like:
/*
* Don't print a warning when the first allocation attempt
* fails. The swiotlb_alloc_coherent() function will print a
* warning when the allocation of DMA memory ultimatly failed. */
This takes the kdump-specifics out of this change (in the end kdump-kernel loaded high is just a case where this failure is much more likely).
Joerg
Re: [PATCH 2/3] x86, swiotlb: Try coherent allocations with __GFP_NOWARN
Quoted 25 lines · Joerg Roedel
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 8 lines · Joerg Roedel
arch/x86/kernel/pci-swiotlb.c +0
-0@@ -20,6 +20,14 @@ void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, {void *vaddr;Quoted 10 lines
arch/x86/kernel/pci-swiotlb.c void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, · line 29
Quoted 7 lines
arch/x86/kernel/pci-swiotlb.c +7
-0+/* +* When booting a kdump kernel in high memory these allocations are very +* likely to fail, as there are by default only 8MB of low memory to +* allocate from. So disable the warnings from the allocator when this +* happens. SWIOTLB also implements fall-backs for failed allocations. +*/ + flags |= __GFP_NOWARN;Should this perhaps then have 'if (
kdump_kernel)' around it since the use-case seems to be kdump related?Hmm, I don't think this is entirely kdump specific. It can also be triggered on a non-kdump kernel, it is just much more unlikely. But maybe I should change the comment to something like:
Quoted 5 lines
/*
* Don't print a warning when the first allocation attempt
* fails. Theswiotlb_alloc_coherent()function will print a
* warning when the allocation of DMA memory ultimatly failed. */
Much better. Thank you.
Quoted 4 lines
This takes the kdump-specifics out of this change (in the end kdump-kernel loaded high is just a case where this failure is much more likely).
<nods>
Quoted 4 lines
Joerg
Re: [PATCH 2/3] x86, swiotlb: Try coherent allocations with __GFP_NOWARN
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 25 lines · Joerg Roedel
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 8 lines · Joerg Roedel
arch/x86/kernel/pci-swiotlb.c +0
-0@@ -20,6 +20,14 @@ void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, {void *vaddr;Quoted 10 lines
arch/x86/kernel/pci-swiotlb.c void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, · line 29
Quoted 7 lines
arch/x86/kernel/pci-swiotlb.c +7
-0+/* +* When booting a kdump kernel in high memory these allocations are very +* likely to fail, as there are by default only 8MB of low memory to +* allocate from. So disable the warnings from the allocator when this +* happens. SWIOTLB also implements fall-backs for failed allocations. +*/ + flags |= __GFP_NOWARN;Should this perhaps then have 'if (
kdump_kernel)' around it since the use-case seems to be kdump related?Hmm, I don't think this is entirely kdump specific. It can also be triggered on a non-kdump kernel, it is just much more unlikely. But maybe I should change the comment to something like:
Quoted 5 lines
/*
* Don't print a warning when the first allocation attempt
* fails. Theswiotlb_alloc_coherent()function will print a
* warning when the allocation of DMA memory ultimatly failed. */
Quoted 7 lines
Much better. Thank you.
Quoted 4 lines
This takes the kdump-specifics out of this change (in the end kdump-kernel loaded high is just a case where this failure is much more likely).
<nods>
Okay, thanks. I'll update the patch.
Joerg
PATCH 3/3 x86, crash: Allocate enough low-mem when crashkernel=high
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
When the crashkernel is loaded above 4GiB in memory the first kernel only allocates 72MiB of low-memory for the DMA requirements of the second kernel. On systems with many devices this is not enough and causes device driver initialization errors and failed crash dumps. Set this default value to 256MiB to make sure there is enough memory available for DMA.
--- arch/x86/kernel/setup.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/setup.c +4 -1
@@ -536,8 +536,11 @@ static void __init reserve_crashkernel_low(void)* swiotlb overflow buffer: now is hardcoded to 32k. * We round it to 8M for other buffers that * may need to stay low too. +* Also make sure we allocate enough extra memory +* low memory so that we don't run out of DMA +* buffers for 32bit devices. */ - low_size =swiotlb_size_or_default () + (8UL <<20 ); + low_size =max (swiotlb_size_or_default () + (8UL <<20 ),256UL <<20 ); auto_set = true; }else {/* passed with crashkernel=0,low ? */
Signature · 1 line
1.9.1
Quoted 13 lines · Joerg Roedel
Hi,
here is a patch-set to fix failed kdump kernel boots when the systems was booted with crashkernel=X,high. On those systems the kernel allocates only 72MiB of low-memory for DMA buffers, which showed to be too low on some systems.
The problem is that 64MiB of the low-memory is allocated by swiotlb, leaving 8MB for the page-allocator. But swiotlb tries to allocate DMA memory from the page-allocator first, which fails pretty fast in the boot sequence, causing warnings. This patch-set removes these warnings.
Quoted 4 lines
But even the 64MiB for swiotlb are eaten up on some systems, so that the default of low-memory allocated for the crash-kernel is increase from 72MB to 256MB (only changing the defaults, can still be overwritten by crashkernel=X,low).
Hi Joerg,
The default low memory is calculated in swiotlb_size_or_default(), and this relies on IO_TLB_DEFAULT_SIZE which is default value for swiotlb. If this doesn't work for your case in kdump kernel, does the default value IO_TLB_DEFAULT_SIZE work for swiotlb in 1st kenrel? If not, user knows it and should know it will fail for kdump kernel either with default vaule, user can specify that by crashkernel=x,low which is why it's introduced.
It may not be a good idea to increase the default low value from 72M to 256M. After all the case you are encountering is a special case, special case need be treated specially, namely specify it in crashkernel=x,low clearly.
Thanks
Baoquan
Quoted 21 lines
This number comes from experiments on the affected systems, 128MiB low-memory was still not enough there, thus I set the value to 256MiB to fix the issues.
Any feedback appreciated.
Thanks,
Joerg
Joerg Roedel (3):
swiotlb: Warn on allocation failure inswiotlb_alloc_coherentx86, swiotlb: Try coherent allocations with__GFP_NOWARNx86, crash: Allocate enough low-mem when crashkernel=higharch/x86/kernel/pci-swiotlb.c | 8 ++++++++ arch/x86/kernel/setup.c | 5 ++++- lib/swiotlb.c | 11 +++++++++-- 3 files changed, 21 insertions(+), 3 deletions(-)
Quoted 7 lines · Baoquan He
The default low memory is calculated in
swiotlb_size_or_default(), and this relies onIO_TLB_DEFAULT_SIZEwhich is default value for swiotlb. If this doesn't work for your case in kdump kernel, does the default valueIO_TLB_DEFAULT_SIZEwork for swiotlb in 1st kenrel? If not, user knows it and should know it will fail for kdump kernel either with default vaule, user can specify that by crashkernel=x,low which is why it's introduced.
In the first kernel the defaults work fine because it has plenty of low-memory (below 4GB) to allocate from. But in the kdump kernel there are only 72MB by default, which showed to not be enough to reliably boot with certain hardware in the machine.
Quoted 4 lines
It may not be a good idea to increase the default low value from 72M to 256M. After all the case you are encountering is a special case, special case need be treated specially, namely specify it in crashkernel=x,low clearly.
I think the kernel should set sane defaults if possible. But to not increase memory usage for kdump too much, how about subtracting the amount of low-memory allocated for kdump from the high-mem amount? This would not increase the overall memory usage.
Joerg
Quoted 8 lines
Hi,
here is a patch-set to fix failed kdump kernel boots when the systems was booted with crashkernel=X,high. On those systems the kernel allocates only 72MiB of low-memory for DMA buffers, which showed to be too low on some systems.
From your experience It seems like swiotlb isn't working well with
crashkernel=X,high alone. What about using crashkernel=X,low with crashkernel=X,high? Is there any reason you have to use crashkernel=X,high alone?
Quoted 11 lines
The problem is that 64MiB of the low-memory is allocated by swiotlb, leaving 8MB for the page-allocator. But swiotlb tries to allocate DMA memory from the page-allocator first, which fails pretty fast in the boot sequence, causing warnings. This patch-set removes these warnings.
But even the 64MiB for swiotlb are eaten up on some systems, so that the default of low-memory allocated for the crash-kernel is increase from 72MB to 256MB (only changing the defaults, can still be overwritten by crashkernel=X,low).
crashkernel=X,high shouldn't automatically reserve 72M low at the first place. Now it's going insane if you increase it to 256M by default.
Thanks
WANG Chao
Quoted 30 lines
This number comes from experiments on the affected systems, 128MiB low-memory was still not enough there, thus I set the value to 256MiB to fix the issues.
Any feedback appreciated.
Thanks,
Joerg
Joerg Roedel (3):
swiotlb: Warn on allocation failure inswiotlb_alloc_coherentx86, swiotlb: Try coherent allocations with__GFP_NOWARNx86, crash: Allocate enough low-mem when crashkernel=higharch/x86/kernel/pci-swiotlb.c | 8 ++++++++ arch/x86/kernel/setup.c | 5 ++++- lib/swiotlb.c | 11 +++++++++-- 3 files changed, 21 insertions(+), 3 deletions(-)Signature · 7 lines
1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Hi,
Quoted 4 lines · WANG Chao
From your experience It seems like swiotlb isn't working well with crashkernel=X,high alone. What about using crashkernel=X,low with crashkernel=X,high? Is there any reason you have to use crashkernel=X,high alone?
Sure, when I specify an additional crashkernel=X,low then I can get things to work this way too. But my patch-set is about changing the default, since the failure was seen on common server systems with the defaults.
Quoted 2 lines
crashkernel=X,high shouldn't automatically reserve 72M low at the first place. Now it's going insane if you increase it to 256M by default.
How should a kernel without some low memory (which has only memory above 4G available) handle any 32bit DMA devices? There would be no way to allocate DMA-able memory for those devices.
And as I said, if people prefer it I can change the patch-set so that the amount of low-memory allocated is subtracted from the amount of high-memory. This way the overall memory usage for kdump would stay the same while changing the defaults to work on more systems.
Joerg
Hi,
Quoted 8 lines · Joerg Roedel
Hi,
Quoted 4 lines · WANG Chao
From your experience It seems like swiotlb isn't working well with crashkernel=X,high alone. What about using crashkernel=X,low with crashkernel=X,high? Is there any reason you have to use crashkernel=X,high alone?
Quoted 4 lines
Sure, when I specify an additional crashkernel=X,low then I can get things to work this way too. But my patch-set is about changing the default, since the failure was seen on common server systems with the defaults.
crashkernel=X doesn't work for you?
Quoted 7 lines
Quoted 3 lines
crashkernel=X,high shouldn't automatically reserve 72M low at the first place. Now it's going insane if you increase it to 256M by default.
How should a kernel without some low memory (which has only memory above 4G available) handle any 32bit DMA devices? There would be no way to allocate DMA-able memory for those devices.
I mean crashkernel=X,high shouldn't automatically reserve low. crashkernel=X,high should always be used together with crashkernel=X,low.
If one is not satisfied with the combination of two parameters, crashkernel=X should be able to look for a suitable area below 4G. But right now, crashkernel=X only deals under 896M.
I've sent a patch for this in the past:
[PATCH] x86, kdump: crashkernel=X try to reserve below 896M first, then try below 4G, then MAXMEM
- https://lkml.org/lkml/2013/10/14/183
X86 people don't like this idea so I didn't update the patch even there's minor nit to clean up.
Quoted 5 lines
And as I said, if people prefer it I can change the patch-set so that the amount of low-memory allocated is subtracted from the amount of high-memory. This way the overall memory usage for kdump would stay the same while changing the defaults to work on more systems.
Say you do this, crashkernel=X,high would be reserve (X-n) high and (n) low? IMHO I think it's better crashkernel=X,high means reserve X high and high only.
Thanks
WANG Chao