Fix kdump failures with crashkernel=high
Subsystems (4)
v3 cover Fix kdump failures with crashkernel=high
v2->v3
* Rebased to v4.1-rc6
* Updated changelogs as requested and added outcome of previous discussions
* Added more Acked-bys
v1->v2:
* Updated comments based on feedback from Konrad
* Added Acked-bys
* Rebased to v3.19-rc3
Hi,
here is a patch-set to fix failed kdump kernel boots when the systems was booted with crashkernel=X,high. On those systems the kernel allocates only 72MiB of low-memory for DMA buffers, which showed to be too low on some systems.
The problem is that 64MiB of the low-memory is allocated by swiotlb, leaving 8MB for the page-allocator. But swiotlb tries to allocate DMA memory from the page-allocator first, which fails pretty fast in the boot sequence, causing warnings. This patch-set removes these warnings.
But even the 64MiB for swiotlb are eaten up on some systems, so that the default of low-memory allocated for the crash-kernel is increase from 72MB to 256MB (only changing the defaults, can still be overwritten by crashkernel=X,low).
This number comes from experiments on the affected systems, 128MiB low-memory was still not enough there, thus I set the value to 256MiB to fix the issues.
Any feedback appreciated.
Thanks,
Joerg
Joerg Roedel (3):
swiotlb: Warn on allocation failure in swiotlb_alloc_coherent x86, swiotlb: Try coherent allocations with __GFP_NOWARN x86, crash: Allocate enough low-mem when crashkernel=high
arch/x86/kernel/pci-swiotlb.c | 7 +++++++ arch/x86/kernel/setup.c | 5 ++++- lib/swiotlb.c | 11 +++++++++-- 3 files changed, 20 insertions(+), 3 deletions(-)
Signature · 1 line
1.9.1
PATCH v3 1/3 swiotlb: Warn on allocation failure in swiotlb_alloc_coherent
From: Joerg Roedel <[email protected]>
Print a warning when all allocation tries have been failed and the function is about to return NULL. This prepares for calling the function with __GFP_NOWARN to suppress allocation failure warnings before all fall-backs have failed.
--- lib/swiotlb.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
lib/swiotlb.c +9 -2
@@ -655,7 +655,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,*/ phys_addr_t paddr =map_single (hwdev,0 , size,DMA_FROM_DEVICE );if (paddr ==SWIOTLB_MAP_ERROR ) -return NULL ; + goto err_warn; ret =phys_to_virt (paddr); dev_addr =phys_to_dma (hwdev, paddr);
@@ -669,7 +669,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,/* DMA_TO_DEVICE to avoid memcpy in unmap_single */ swiotlb_tbl_unmap_single (hwdev, paddr, size,DMA_TO_DEVICE ); -return NULL ; + goto err_warn; } }
@@ -677,6 +677,13 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,memset (ret,0 , size);return ret; + +err_warn: +pr_warn ("swiotlb: coherent allocation failed for device %s size=%zu\n" , +dev_name (hwdev), size); +dump_stack (); + +return NULL ; }EXPORT_SYMBOL (swiotlb_alloc_coherent);
Signature · 1 line
1.9.1
tip-bot for Joerg Roedel
merge bot[tip:x86/kdump] swiotlb: Warn on allocation failure in swiotlb_alloc_coherent()
swiotlb: Warn on allocation failure in swiotlb_alloc_coherent()
Commit-ID: 94cc81f9a8f995923e35e2db936741dd62d18350 Gitweb: http://git.kernel.org/tip/94cc81f9a8f995923e35e2db936741dd62d18350 Author: Joerg Roedel <[email protected]> AuthorDate: Wed, 10 Jun 2015 17:49:40 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 11 Jun 2015 08:28:38 +0200
swiotlb: Warn on allocation failure in swiotlb_alloc_coherent()
Print a warning when all allocation tries have been failed and the function is about to return NULL.
This prepares for calling the function with __GFP_NOWARN to suppress allocation failure warnings before all fall-backs have failed - which we'll do to improve kdump behavior.
--- lib/swiotlb.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
lib/swiotlb.c +9 -2
@@ -656,7 +656,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,*/ phys_addr_t paddr =map_single (hwdev,0 , size,DMA_FROM_DEVICE );if (paddr ==SWIOTLB_MAP_ERROR ) -return NULL ; + goto err_warn; ret =phys_to_virt (paddr); dev_addr =phys_to_dma (hwdev, paddr);
@@ -670,7 +670,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,/* DMA_TO_DEVICE to avoid memcpy in unmap_single */ swiotlb_tbl_unmap_single (hwdev, paddr, size,DMA_TO_DEVICE ); -return NULL ; + goto err_warn; } }
@@ -678,6 +678,13 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,memset (ret,0 , size);return ret; + +err_warn: +pr_warn ("swiotlb: coherent allocation failed for device %s size=%zu\n" , +dev_name (hwdev), size); +dump_stack (); + +return NULL ; }EXPORT_SYMBOL (swiotlb_alloc_coherent);
PATCH v3 2/3 x86, swiotlb: Try coherent allocations with __GFP_NOWARN
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
When we boot a kdump kernel in high memory, there is by default only 72MB of low memory available. The swiotlb code takes 64MB of it (by default) so that there are only 8MB left to allocate from. On systems with many devices this causes page allocator warnings from dma_generic_alloc_coherent():
systemd-udevd: page allocation failure: order:0, mode:0x280d4
CPU: 0 PID: 197 Comm: systemd-udevd Tainted: G W 3.12.28-4-default #1 Hardware name: HP ProLiant DL980 G7, BIOS P66 07/30/2012 ffff8800781335e0 ffffffff8150b1db 00000000000280d4 ffffffff8113af90 0000000000000000 0000000000000000 ffff88007efdbb00 0000000100000000 0000000000000000 0000000000000000 0000000000000000 0000000000000001 Call Trace: dump_trace+0x7d/0x2d0 show_stack_log_lvl+0x94/0x170 show_stack+0x21/0x50 dump_stack+0x41/0x51 warn_alloc_failed+0xf0/0x160 __alloc_pages_slowpath+0x72f/0x796 __alloc_pages_nodemask+0x1ea/0x210 dma_generic_alloc_coherent+0x96/0x140 x86_swiotlb_alloc_coherent+0x1c/0x50 ttm_dma_pool_alloc_new_pages+0xab/0x320 [ttm] ttm_dma_populate+0x3ce/0x640 [ttm] ttm_tt_bind+0x36/0x60 [ttm] ttm_bo_handle_move_mem+0x55f/0x5c0 [ttm] ttm_bo_move_buffer+0x105/0x130 [ttm] ttm_bo_validate+0xc1/0x130 [ttm] ttm_bo_init+0x24b/0x400 [ttm] radeon_bo_create+0x16c/0x200 [radeon] radeon_ring_init+0x11e/0x2b0 [radeon] r100_cp_init+0x123/0x5b0 [radeon] r100_startup+0x194/0x230 [radeon] r100_init+0x223/0x410 [radeon] radeon_device_init+0x6af/0x830 [radeon] radeon_driver_load_kms+0x89/0x180 [radeon] drm_get_pci_dev+0x121/0x2f0 [drm] local_pci_probe+0x39/0x60 pci_device_probe+0xa9/0x120 driver_probe_device+0x9d/0x3d0 __driver_attach+0x8b/0x90 bus_for_each_dev+0x5b/0x90 bus_add_driver+0x1f8/0x2c0 driver_register+0x5b/0xe0 do_one_initcall+0xf2/0x1a0 load_module+0x1207/0x1c70 SYSC_finit_module+0x75/0xa0 system_call_fastpath+0x16/0x1b 0x7fac533d2788
After these warnings the code enters a fall-back path and allocated directly from the swiotlb aperture in the end. So remove these warnings as this is not a fatal error.
--- arch/x86/kernel/pci-swiotlb.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/kernel/pci-swiotlb.c +7 -0
@@ -20,6 +20,13 @@ void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, {void *vaddr; +/* +* Don't print a warning when the first allocation attempt +* fails. The swiotlb_alloc_coherent() function will print a +* warning when the allocation of DMA memory ultimatly failed. +*/ + flags |= __GFP_NOWARN; + vaddr =dma_generic_alloc_coherent (hwdev, size, dma_handle, flags, attrs);if (vaddr)
Signature · 1 line
1.9.1
tip-bot for Joerg Roedel
merge bot[tip:x86/kdump] x86/swiotlb: Try coherent allocations with __GFP_NOWARN
x86/swiotlb: Try coherent allocations with __GFP_NOWARN
Commit-ID: 186dfc9d69b96a38ec6ec654127dba4432184494 Gitweb: http://git.kernel.org/tip/186dfc9d69b96a38ec6ec654127dba4432184494 Author: Joerg Roedel <[email protected]> AuthorDate: Wed, 10 Jun 2015 17:49:41 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 11 Jun 2015 08:28:38 +0200
x86/swiotlb: Try coherent allocations with __GFP_NOWARN
When we boot a kdump kernel in high memory, there is by default only 72MB of low memory available. The swiotlb code takes 64MB of it (by default) so that there are only 8MB left to allocate from. On systems with many devices this causes page allocator warnings from dma_generic_alloc_coherent():
systemd-udevd: page allocation failure: order:0, mode:0x280d4
CPU: 0 PID: 197 Comm: systemd-udevd Tainted: G W
3.12.28-4-default #1 Hardware name: HP ProLiant DL980 G7, BIOS
P66 07/30/2012 ffff8800781335e0 ffffffff8150b1db 00000000000280d4 ffffffff8113af90
0000000000000000 0000000000000000 ffff88007efdbb00 0000000100000000
0000000000000000 0000000000000000 0000000000000000 0000000000000001
Call Trace:
dump_trace+0x7d/0x2d0
show_stack_log_lvl+0x94/0x170
show_stack+0x21/0x50
dump_stack+0x41/0x51
warn_alloc_failed+0xf0/0x160
__alloc_pages_slowpath+0x72f/0x796
__alloc_pages_nodemask+0x1ea/0x210
dma_generic_alloc_coherent+0x96/0x140
x86_swiotlb_alloc_coherent+0x1c/0x50
ttm_dma_pool_alloc_new_pages+0xab/0x320 [ttm]
ttm_dma_populate+0x3ce/0x640 [ttm]
ttm_tt_bind+0x36/0x60 [ttm]
ttm_bo_handle_move_mem+0x55f/0x5c0 [ttm]
ttm_bo_move_buffer+0x105/0x130 [ttm]
ttm_bo_validate+0xc1/0x130 [ttm]
ttm_bo_init+0x24b/0x400 [ttm]
radeon_bo_create+0x16c/0x200 [radeon]
radeon_ring_init+0x11e/0x2b0 [radeon]
r100_cp_init+0x123/0x5b0 [radeon]
r100_startup+0x194/0x230 [radeon]
r100_init+0x223/0x410 [radeon]
radeon_device_init+0x6af/0x830 [radeon]
radeon_driver_load_kms+0x89/0x180 [radeon]
drm_get_pci_dev+0x121/0x2f0 [drm]
local_pci_probe+0x39/0x60
pci_device_probe+0xa9/0x120
driver_probe_device+0x9d/0x3d0
__driver_attach+0x8b/0x90
bus_for_each_dev+0x5b/0x90
bus_add_driver+0x1f8/0x2c0
driver_register+0x5b/0xe0
do_one_initcall+0xf2/0x1a0
load_module+0x1207/0x1c70
SYSC_finit_module+0x75/0xa0
system_call_fastpath+0x16/0x1b
0x7fac533d2788After these warnings the code enters a fall-back path and allocated directly from the swiotlb aperture in the end. So remove these warnings as this is not a fatal error.
--- arch/x86/kernel/pci-swiotlb.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/kernel/pci-swiotlb.c +7 -0
@@ -20,6 +20,13 @@ void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, {void *vaddr; +/* +* Don't print a warning when the first allocation attempt fails. +* swiotlb_alloc_coherent() will print a warning when the DMA +* memory allocation ultimately failed. +*/ + flags |= __GFP_NOWARN; + vaddr =dma_generic_alloc_coherent (hwdev, size, dma_handle, flags, attrs);if (vaddr)
PATCH v3 3/3 x86, crash: Allocate enough low-mem when crashkernel=high
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
When the crashkernel is loaded above 4GiB in memory the first kernel only allocates 72MiB of low-memory for the DMA requirements of the second kernel. On systems with many devices this is not enough and causes device driver initialization errors and failed crash dumps. Testing by SUSE and Redhat have shown that 256MiB is a good default value for now and the discussion has agreed on this value as well. So set this default value to 256MiB to make sure there is enough memory available for DMA.
--- arch/x86/kernel/setup.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/setup.c +4 -1
@@ -535,8 +535,11 @@ static void __init reserve_crashkernel_low(void)* swiotlb overflow buffer: now is hardcoded to 32k. * We round it to 8M for other buffers that * may need to stay low too. +* Also make sure we allocate enough extra memory +* low memory so that we don't run out of DMA +* buffers for 32bit devices. */ - low_size =swiotlb_size_or_default () + (8UL <<20 ); + low_size =max (swiotlb_size_or_default () + (8UL <<20 ),256UL <<20 ); auto_set = true; }else {/* passed with crashkernel=0,low ? */
Signature · 1 line
1.9.1
tip-bot for Joerg Roedel
merge bot[tip:x86/kdump] x86/crash: Allocate enough low memory when crashkernel=high
x86/crash: Allocate enough low memory when crashkernel=high
Commit-ID: 94fb9334182284e8e7e4bcb9125c25dc33af19d4 Gitweb: http://git.kernel.org/tip/94fb9334182284e8e7e4bcb9125c25dc33af19d4 Author: Joerg Roedel <[email protected]> AuthorDate: Wed, 10 Jun 2015 17:49:42 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 11 Jun 2015 08:28:39 +0200
x86/crash: Allocate enough low memory when crashkernel=high
When the crash kernel is loaded above 4GiB in memory, the first kernel allocates only 72MiB of low-memory for the DMA requirements of the second kernel. On systems with many devices this is not enough and causes device driver initialization errors and failed crash dumps. Testing by SUSE and Redhat has shown that 256MiB is a good default value for now and the discussion has lead to this value as well. So set this default value to 256MiB to make sure there is enough memory available for DMA.
--- arch/x86/kernel/setup.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/kernel/setup.c +7 -5
@@ -531,12 +531,14 @@ static void __init reserve_crashkernel_low(void)if (ret !=0 ) {/* * two parts from lib/swiotlb.c: -* swiotlb size: user specified with swiotlb= or default. -* swiotlb overflow buffer: now is hardcoded to 32k. -* We round it to 8M for other buffers that -* may need to stay low too. +* -swiotlb size: user-specified with swiotlb= or default. +* +* -swiotlb overflow buffer: now hardcoded to 32k. We round it +* to 8M for other buffers that may need to stay low too. Also +* make sure we allocate enough extra low memory so that we +* don't run out of DMA buffers for 32-bit devices. */ - low_size =swiotlb_size_or_default () + (8UL <<20 ); + low_size =max (swiotlb_size_or_default () + (8UL <<20 ),256UL <<20 ); auto_set = true; }else {/* passed with crashkernel=0,low ? */
Borislav Petkov
maintainerQuoted 46 lines · Joerg Roedel
v2->v3* Rebased to v4.1-rc6
* Updated changelogs as requested and added outcome of previous discussions
* Added more Acked-bys
v1->v2:* Updated comments based on feedback from Konrad
* Added Acked-bys
* Rebased to v3.19-rc3Hi,
here is a patch-set to fix failed kdump kernel boots when the systems was booted with crashkernel=X,high. On those systems the kernel allocates only 72MiB of low-memory for DMA buffers, which showed to be too low on some systems.
The problem is that 64MiB of the low-memory is allocated by swiotlb, leaving 8MB for the page-allocator. But swiotlb tries to allocate DMA memory from the page-allocator first, which fails pretty fast in the boot sequence, causing warnings. This patch-set removes these warnings.
But even the 64MiB for swiotlb are eaten up on some systems, so that the default of low-memory allocated for the crash-kernel is increase from 72MB to 256MB (only changing the defaults, can still be overwritten by crashkernel=X,low).
This number comes from experiments on the affected systems, 128MiB low-memory was still not enough there, thus I set the value to 256MiB to fix the issues.
Any feedback appreciated.
Thanks,
Joerg
Joerg Roedel (3):
swiotlb: Warn on allocation failure inswiotlb_alloc_coherentx86, swiotlb: Try coherent allocations with__GFP_NOWARNx86, crash: Allocate enough low-mem when crashkernel=high
Quoted 4 lines
arch/x86/kernel/pci-swiotlb.c | 7 +++++++ arch/x86/kernel/setup.c | 5 ++++- lib/swiotlb.c | 11 +++++++++-- 3 files changed, 20 insertions(+), 3 deletions(-)
Looks simple enough, all applied, thanks.
hpa, Ingo, any concerns?
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Quoted 5 lines · Borislav Petkov
Quoted 5 lines
Joerg Roedel (3):
swiotlb: Warn on allocation failure inswiotlb_alloc_coherentx86, swiotlb: Try coherent allocations with__GFP_NOWARNx86, crash: Allocate enough low-mem when crashkernel=high
Quoted 8 lines
Quoted 4 lines
arch/x86/kernel/pci-swiotlb.c | 7 +++++++ arch/x86/kernel/setup.c | 5 ++++- lib/swiotlb.c | 11 +++++++++-- 3 files changed, 20 insertions(+), 3 deletions(-)Looks simple enough, all applied, thanks.
hpa, Ingo, any concerns?
Modulo a couple of typos it all looks sane.
Thanks,
Ingo
Borislav Petkov
maintainerQuoted 1 line · Ingo Molnar
Modulo a couple of typos it all looks sane.
Ah yes, I caught a few while applying. I'll send you the final result tomorrow.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--