iommu/iova: silence warnings under memory pressure
Subsystems (2)
PATCH 1/1 iommu/iova: silence warnings under memory pressure
When running heavy memory pressure workloads, this 5+ old system is throwing endless warnings below because disk IO is too slow to recover from swapping. Since the volume from alloc_iova_fast() could be large, once it calls printk(), it will trigger disk IO (writing to the log files) and pending softirqs which could cause a loop and no progress from memory reclaim for days.
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
slab_out_of_memory: 66 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
cache: iommu_iova, object size: 40, buffer size: 448, default order:
0, min order: 0
node 0: slabs: 1822, objs: 16398, free: 0
node 1: slabs: 2051, objs: 18459, free: 31
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
cache: iommu_iova, object size: 40, buffer size: 448, default order:
0, min order: 0
node 0: slabs: 1822, objs: 16398, free: 0
node 1: slabs: 2051, objs: 18459, free: 31
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
cache: iommu_iova, object size: 40, buffer size: 448, default order:
0, min order: 0
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
cache: skbuff_head_cache, object size: 208, buffer size: 640, default
order: 0, min order: 0
cache: skbuff_head_cache, object size: 208, buffer size: 640, default
order: 0, min order: 0
cache: skbuff_head_cache, object size: 208, buffer size: 640, default
order: 0, min order: 0
cache: skbuff_head_cache, object size: 208, buffer size: 640, default
order: 0, min order: 0
node 0: slabs: 697, objs: 4182, free: 0
node 0: slabs: 697, objs: 4182, free: 0
node 0: slabs: 697, objs: 4182, free: 0
node 0: slabs: 697, objs: 4182, free: 0
node 1: slabs: 381, objs: 2286, free: 27
node 1: slabs: 381, objs: 2286, free: 27
node 1: slabs: 381, objs: 2286, free: 27
node 1: slabs: 381, objs: 2286, free: 27
node 0: slabs: 1822, objs: 16398, free: 0
cache: skbuff_head_cache, object size: 208, buffer size: 640, default
order: 0, min order: 0
node 1: slabs: 2051, objs: 18459, free: 31
node 0: slabs: 697, objs: 4182, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
node 1: slabs: 381, objs: 2286, free: 27
cache: skbuff_head_cache, object size: 208, buffer size: 640, default
order: 0, min order: 0
node 0: slabs: 697, objs: 4182, free: 0
node 1: slabs: 381, objs: 2286, free: 27
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
warn_alloc: 96 callbacks suppressed
kworker/11:1H: page allocation failure: order:0,
mode:0xa20(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0-1
CPU: 11 PID: 1642 Comm: kworker/11:1H Tainted: G B Hardware name: HP ProLiant XL420 Gen9/ProLiant XL420 Gen9, BIOS U19 12/27/2015 Workqueue: kblockd blk_mq_run_work_fn Call Trace: dump_stack+0xa0/0xea warn_alloc.cold.94+0x8a/0x12d __alloc_pages_slowpath+0x1750/0x1870 __alloc_pages_nodemask+0x58a/0x710 alloc_pages_current+0x9c/0x110 alloc_slab_page+0xc9/0x760 allocate_slab+0x48f/0x5d0 new_slab+0x46/0x70 ___slab_alloc+0x4ab/0x7b0 __slab_alloc+0x43/0x70 kmem_cache_alloc+0x2dd/0x450 SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC) alloc_iova+0x33/0x210 cache: skbuff_head_cache, object size: 208, buffer size: 640, default order: 0, min order: 0 node 0: slabs: 697, objs: 4182, free: 0 alloc_iova_fast+0x62/0x3d1 node 1: slabs: 381, objs: 2286, free: 27 intel_alloc_iova+0xce/0xe0 intel_map_sg+0xed/0x410 scsi_dma_map+0xd7/0x160 scsi_queue_rq+0xbf7/0x1310 blk_mq_dispatch_rq_list+0x4d9/0xbc0 blk_mq_sched_dispatch_requests+0x24a/0x300 __blk_mq_run_hw_queue+0x156/0x230 blk_mq_run_work_fn+0x3b/0x40 process_one_work+0x579/0xb90 worker_thread+0x63/0x5b0 kthread+0x1e6/0x210 ret_from_fork+0x3a/0x50 Mem-Info: active_anon:2422723 inactive_anon:361971 isolated_anon:34403 active_file:2285 inactive_file:1838 isolated_file:0 unevictable:0 dirty:1 writeback:5 unstable:0 slab_reclaimable:13972 slab_unreclaimable:453879 mapped:2380 shmem:154 pagetables:6948 bounce:0 free:19133 free_pcp:7363 free_cma:0
--- drivers/iommu/iova.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/iova.c +1 -1
@@ -233,7 +233,7 @@ static int __alloc_and_insert_iova_range(struct iova_domain *iovad,struct iova *alloc_iova_mem (void ) { -return kmem_cache_alloc (iova_cache,GFP_ATOMIC ); +return kmem_cache_alloc (iova_cache,GFP_ATOMIC | __GFP_NOWARN); }EXPORT_SYMBOL (alloc_iova_mem);
Signature · 1 line
1.8.3.1
Quoted 6 lines · Qian Cai
When running heavy memory pressure workloads, this 5+ old system is throwing endless warnings below because disk IO is too slow to recover from swapping. Since the volume from
alloc_iova_fast()could be large, once it callsprintk(), it will trigger disk IO (writing to the log files) and pending softirqs which could cause a loop and no progress from memory reclaim for days.
[]
drivers/iommu/iova.c
Quoted 1 line
drivers/iommu/iova.c +0
-0
[]
drivers/iommu/iova.c static int __alloc_and_insert_iova_range(struct iova_domain *iovad, · line 238
Quoted 8 lines
drivers/iommu/iova.c +1
-1@@ -233,7 +233,7 @@ static int __alloc_and_insert_iova_range(struct iova_domain *iovad,struct iova *alloc_iova_mem (void ) { -return kmem_cache_alloc (iova_cache,GFP_ATOMIC ); +return kmem_cache_alloc (iova_cache,GFP_ATOMIC | __GFP_NOWARN); }EXPORT_SYMBOL (alloc_iova_mem);
Is notification ever useful?
If so, maybe something like:
struct iova *alloc_iova_mem(void)
{
void *mem = kmem_cache_alloc(iova_cache, GFP_ATOMIC | __GFP_NOWARN)_
WARN_RATELIMIT(!mem, "%s: unable to alloc cache\n", __func__);
return mem;
}or maybe use printk_deferred or prink_deferred_once
?
Quoted 34 lines · Joe Perches
Quoted 6 lines · Qian Cai
When running heavy memory pressure workloads, this 5+ old system is throwing endless warnings below because disk IO is too slow to recover from swapping. Since the volume from
alloc_iova_fast()could be large, once it callsprintk(), it will trigger disk IO (writing to the log files) and pending softirqs which could cause a loop and no progress from memory reclaim for days.[]
drivers/iommu/iova.c
Quoted 1 line
drivers/iommu/iova.c +0
-0[]
drivers/iommu/iova.c static int __alloc_and_insert_iova_range(struct iova_domain *iovad, · line 238
Quoted 8 lines
drivers/iommu/iova.c +1
-1@@ -233,7 +233,7 @@ static int __alloc_and_insert_iova_range(struct iova_domain *iovad,struct iova *alloc_iova_mem (void ) { -return kmem_cache_alloc (iova_cache,GFP_ATOMIC ); +return kmem_cache_alloc (iova_cache,GFP_ATOMIC | __GFP_NOWARN); }EXPORT_SYMBOL (alloc_iova_mem);Is notification ever useful?
If so, maybe something like:
struct iova
*alloc_iova_mem(void){ void *mem = kmem_cache_alloc(iova_cache, GFP_ATOMIC | __GFP_NOWARN)_ WARN_RATELIMIT(!mem, "%s: unable to alloc cache\n", __func__); return mem; }
Quoted 4 lines
or maybe use
printk_deferredorprink_deferred_once?
Forgot to mentioned that errors are also reported in hpsa driver which is fine.
hpsa 0000:03:00.0: DMAR: Allocating 1-page iova failed
but warn_alloc() is way too expense for this old server.