Skip to contentSkip to conversation

dma-debug: fix soft lockup when a lot of debug are enabled

PATCH In review Anders Roxell 2 participants 3 messages Dec 11, 2018 – Dec 12, 2018 DMA mapping helpers

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dma-debug: fix soft lockup when a lot of debug are enabled

When running a kernel in qemu with enough debugging options (slub-debug, ftrace, kcov, kasan, ubsan, ...) enabled together, that results in a slow initcall. So a 'watchdog: BUG: soft lockup' happens:

[   44.105619] Call trace:
[   44.106709]  __slab_alloc+0x70/0x88
[   44.107757]  kmem_cache_alloc_trace+0x13c/0x590
[   44.108764]  prealloc_memory+0xb8/0x240
[   44.109768]  dma_debug_init+0x174/0x1f0
[   44.110725]  do_one_initcall+0x430/0x8b0
[   44.111719]  do_initcall_level+0x548/0x5b4
[   44.112672]  do_initcalls+0x28/0x4c
[   44.113628]  do_basic_setup+0x34/0x3c
[   44.114559]  kernel_init_freeable+0x194/0x24c
[   44.115570]  kernel_init+0x24/0x18c
[   44.116498]  ret_from_fork+0x10/0x18

Rework to call cond_resched(), in the function prealloc_memory()'s for-loop.

---
 kernel/dma/debug.c | 1 +
 1 file changed, 1 insertion(+)
kernel/dma/debug.c +1 -0
index 231ca4628062..2abdc265aec4 100644
@@ -767,6 +767,7 @@ static int prealloc_memory(u32 num_entries)
 	int i;
 
 	for (i = 0; i < num_entries; ++i) {
+		cond_resched();
 		entry = kzalloc(sizeof(*entry), GFP_KERNEL);
 		if (!entry)
 			goto out_err;
Signature · 1 line
2.19.2
reviewer
#1 lore raw

Hi Anders,

Quoted 17 lines · Anders Roxell

When running a kernel in qemu with enough debugging options (slub-debug, ftrace, kcov, kasan, ubsan, ...) enabled together, that results in a slow initcall. So a 'watchdog: BUG: soft lockup' happens:

[   44.105619] Call trace:
[   44.106709]  __slab_alloc+0x70/0x88
[   44.107757]  kmem_cache_alloc_trace+0x13c/0x590
[   44.108764]  prealloc_memory+0xb8/0x240
[   44.109768]  dma_debug_init+0x174/0x1f0
[   44.110725]  do_one_initcall+0x430/0x8b0
[   44.111719]  do_initcall_level+0x548/0x5b4
[   44.112672]  do_initcalls+0x28/0x4c
[   44.113628]  do_basic_setup+0x34/0x3c
[   44.114559]  kernel_init_freeable+0x194/0x24c
[   44.115570]  kernel_init+0x24/0x18c
[   44.116498]  ret_from_fork+0x10/0x18
Quoted 2 lines

Rework to call cond_resched(), in the function prealloc_memory()'s for-loop.

Funnily enough I've just done a bit of work in this area which should make allocation a lot more efficient - would you mind giving this series a spin on the same setup to see how much it helps the performance?

https://lore.kernel.org/lkml/[email protected]/

Thanks,
Robin.

Quoted 19 lines
---
  kernel/dma/debug.c | 1 +
  1 file changed, 1 insertion(+)
kernel/dma/debug.c +1 -0
index 231ca4628062..2abdc265aec4 100644
@@ -767,6 +767,7 @@ static int prealloc_memory(u32 num_entries)
  	int i;
 
  	for (i = 0; i < num_entries; ++i) {
+		cond_resched();
  		entry = kzalloc(sizeof(*entry), GFP_KERNEL);
  		if (!entry)
  			goto out_err;
#2 lore raw
Quoted 2 lines · Robin Murphy

Hi Anders,

Hi Robin,

Quoted 19 lines · Anders Roxell
Quoted 17 lines

When running a kernel in qemu with enough debugging options (slub-debug, ftrace, kcov, kasan, ubsan, ...) enabled together, that results in a slow initcall. So a 'watchdog: BUG: soft lockup' happens:

[   44.105619] Call trace:
[   44.106709]  __slab_alloc+0x70/0x88
[   44.107757]  kmem_cache_alloc_trace+0x13c/0x590
[   44.108764]  prealloc_memory+0xb8/0x240
[   44.109768]  dma_debug_init+0x174/0x1f0
[   44.110725]  do_one_initcall+0x430/0x8b0
[   44.111719]  do_initcall_level+0x548/0x5b4
[   44.112672]  do_initcalls+0x28/0x4c
[   44.113628]  do_basic_setup+0x34/0x3c
[   44.114559]  kernel_init_freeable+0x194/0x24c
[   44.115570]  kernel_init+0x24/0x18c
[   44.116498]  ret_from_fork+0x10/0x18
Quoted 8 lines
Quoted 2 lines

Rework to call cond_resched(), in the function prealloc_memory()'s for-loop.

Funnily enough I've just done a bit of work in this area which should make allocation a lot more efficient - would you mind giving this series a spin on the same setup to see how much it helps the performance?

https://lore.kernel.org/lkml/[email protected]/

I tried it on todays next tag and couldn't reproduce the soft lockup.

Cheers,
Anders

Quoted 23 lines

Thanks,
Robin.

Quoted 19 lines
---
  kernel/dma/debug.c | 1 +
  1 file changed, 1 insertion(+)
kernel/dma/debug.c +1 -0
index 231ca4628062..2abdc265aec4 100644
@@ -767,6 +767,7 @@ static int prealloc_memory(u32 num_entries)
      int i;
 
      for (i = 0; i < num_entries; ++i) {
+             cond_resched();
              entry = kzalloc(sizeof(*entry), GFP_KERNEL);
              if (!entry)
                      goto out_err;