dmatest regression in 3.10-rc1
Hello,
I've been observing a regression in the dmatest module with 3.10-rc1. It manifests as either:
- a spurious timeout on one or more of the channel threads
- a complete kernel lockup (loss of console)
- a panic (see below, noting that the callback [dmatest_callback] is dereferencing a NULL pointer)
If I revert 77101ce578bb ("dmatest: cancel thread immediately when asked for") then things are rosy again, but I'm not sure if this is hiding another problem.
I'm using 8 iterations, 8 threads per channel and a 128k buffer size, although I've reproduced it with 1 iteration, 1 thread, default buffer size too.
Any ideas? Some of the changes to dmatest.c since 3.9 have some fishy looking synchronisation variables without barriers/atomic accesses, but I'm reproducing this on a software model so I wouldn't expect too much memory re-ordering.
Any help appreciated,
Will
--->8
[ 18.911813] dmatest: thread dma0chan0-copy1 exited with status 0 [ 18.911916] Unable to handle kernel NULL pointer dereference at virtual address 00000000 [ 18.911974] pgd = c0003000 [ 18.912020] [00000000] *pgd=80000080004003, *pmd=00000000 [ 18.912097] Internal error: Oops: 207 [#1] PREEMPT SMP ARM [ 18.912144] Modules linked in: dmatest(-) [ 18.912230] CPU: 0 PID: 651 Comm: dma2chan0-copy3 Not tainted 3.10.0-rc1+ #15 [ 18.912300] task: dfac8880 ti: df37a000 task.ti: df37a000 [ 18.912378] PC is at _raw_spin_lock_irqsave+0x24/0x60 [ 18.912448] LR is at __wake_up+0x20/0x50 [ 18.912522] pc : [<c032ec60>] lr : [<c0044054>] psr: a0000193 [ 18.912522] sp : df37bb78 ip : 00000000 fp : df37bb9c [ 18.912616] r10: df37a000 r9 : 00000000 r8 : 00000000 [ 18.912679] r7 : 00000003 r6 : 00000000 r5 : df82c000 r4 : 00000000 [ 18.912749] r3 : df37a000 r2 : 00000000 r1 : 40000103 r0 : a0000113 [ 18.912818] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel [ 18.912893] Control: 30c5387d Table: 9f97ec80 DAC: fffffffd [ 18.912956] Process dma2chan0-copy3 (pid: 651, stack limit = 0xdf37a240) [ 18.913020] Stack: (0xdf37bb78 to 0xdf37c000) [ 18.913093] bb60: d28de712 00000001 [ 18.913207] bb80: 0000006e df836878 df82c000 00000000 df82c040 df82c000 df37bbc4 c01e92d8 [ 18.913322] bba0: 00000001 c0ce3a40 00000000 80000193 dfac88b8 00002e7b df82c04c 20000113 [ 18.913438] bbc0: c0ce2e9c df836878 df836878 00000080 00000000 70b93fb1 8f46af61 df82c000 [ 18.913551] bbe0: df82c004 00000000 c0556508 c0585d00 00000000 df37a000 00000006 c0025304 [ 18.913665] bc00: 00000001 00000018 c055c09c df37a000 c055c080 40000102 00000018 c00256e0 [ 18.913782] bc20: df80c950 df0850c0 e0802000 00000000 ffff90d1 00200040 df37bcec df37a000 [ 18.913895] bc40: 00000060 00000000 e0802000 dfa00cc0 00000000 df0d5c00 df37bcec c0025ae8 [ 18.914013] bc60: c0556e48 c000eb38 e080200c c0562820 df37bc90 c0008560 c00444b0 c032efe4 [ 18.914130] bc80: 60000113 ffffffff df37bcc4 c000de80 c0ce3a40 dfac8880 df37a000 0000221b [ 18.914244] bca0: 00000000 c0ce3a40 dfacc840 00000002 dfa00cc0 00000000 df0d5c00 df37bcec [ 18.914362] bcc0: c0557a40 df37bcd8 c00444b0 c032efe4 60000113 ffffffff c0ce3a40 df37a000 [ 18.914478] bce0: c0557a40 dfac8880 df37bd8c c032db74 00000001 c002bd68 c0586a54 c0586c54 [ 18.914593] bd00: df37bd00 df37bd00 6475a018 00000004 00000004 c032e458 c0557a40 c0557a40 [ 18.914706] bd20: 00000000 00000008 c055c08c c0025754 c05731c0 df81c840 c0ce6680 00000000 [ 18.914821] bd40: ffff90cd 00200040 c01d4a84 df37a000 0000001b 00000000 e0802000 00007317 [ 18.914936] bd60: c0556e48 0000001b 00000000 df37a000 80000113 ffffffff df37bdd4 c000dea0 [ 18.915052] bd80: df37a000 0001beae df37bd9c c032e458 bf001fe0 bf001e70 00000000 c000deb8 [ 18.915164] bda0: df15a200 dfb4ca40 0001beae 00020000 0001c8d5 00000000 0000000b 0000000c [ 18.915279] bdc0: 00007317 deea0000 0001beae 00000000 0001c8d4 df37bde8 bf001fe0 bf001e70 [ 18.915393] bde0: 80000113 ffffffff 00020000 df15a200 00000000 00014b97 0000517d 00007317 [ 18.915505] be00: dfb4ca40 0001beae 00000000 bf001fe0 0001beae 00000000 00000000 00000001 [ 18.915618] be20: 00000080 00020000 0001beae bf002dfc dfb4c980 00000001 bf00262c 00000000 [ 18.915731] be40: 00000000 00000001 00000080 ffffffff 000000c0 0000517d df37bf64 bf001cdc [ 18.915843] be60: 00014b97 00007317 dfb4ca40 00000001 0001beae 00000000 00000000 bf000e78 [ 18.915954] be80: 9eea0000 00000000 df374000 00000002 9ee8517d 00000000 00000000 00000000 [ 18.916066] bea0: df37401c 00000008 df37bf20 00000010 dfac8ab8 00000010 00000000 00000008 [ 18.916181] bec0: df37be80 df82e014 df37be98 0000517d df37be80 df37be90 00000001 dfb4c980 [ 18.916294] bee0: 00014b97 00000001 00000001 c0564504 c0591d00 00000000 00007317 df97e340 [ 18.916408] bf00: 0000517d bf002dfc dfa0ac40 00000003 df37bf44 75b7c14f 00000001 df37bf20 [ 18.916521] bf20: 00040004 df37bf24 df37bf24 00000007 00000007 00000000 df37bf38 df37bf38 [ 18.916634] bf40: df37bf64 df161e98 00000000 df97e340 bf000b0c 00000000 00000000 00000000 [ 18.916746] bf60: 00000000 c003c1a4 dfdfdfcf 00000000 dfdfdfcf df97e340 00000000 00000000 [ 18.916861] bf80: df37bf80 df37bf80 00000000 00000000 df37bf90 df37bf90 df37bfac df161e98 [ 18.916971] bfa0: c003c100 00000000 00000000 c000e318 00000000 00000000 00000000 00000000 [ 18.917078] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 18.917189] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 dfdfdfcf cfdfdfdf [ 18.917318] [<c032ec60>] (_raw_spin_lock_irqsave+0x24/0x60) from [<c0044054>] (__wake_up+0x20/0x50) [ 18.917453] [<c0044054>] (__wake_up+0x20/0x50) from [<c01e92d8>] (pl330_tasklet+0x4c8/0x580) [ 18.917593] [<c01e92d8>] (pl330_tasklet+0x4c8/0x580) from [<c0025304>] (tasklet_action+0x74/0x100) [ 18.917734] [<c0025304>] (tasklet_action+0x74/0x100) from [<c00256e0>] (__do_softirq+0xe0/0x1b8) [ 18.917858] [<c00256e0>] (__do_softirq+0xe0/0x1b8) from [<c0025ae8>] (irq_exit+0x90/0xc8) [ 18.917972] [<c0025ae8>] (irq_exit+0x90/0xc8) from [<c000eb38>] (handle_IRQ+0x3c/0x94) [ 18.918089] [<c000eb38>] (handle_IRQ+0x3c/0x94) from [<c0008560>] (gic_handle_irq+0x28/0x5c) [ 18.918204] [<c0008560>] (gic_handle_irq+0x28/0x5c) from [<c000de80>] (__irq_svc+0x40/0x70) [ 18.918273] Exception stack(0xdf37bc90 to 0xdf37bcd8) [ 18.918357] bc80: c0ce3a40 dfac8880 df37a000 0000221b [ 18.918470] bca0: 00000000 c0ce3a40 dfacc840 00000002 dfa00cc0 00000000 df0d5c00 df37bcec [ 18.918572] bcc0: c0557a40 df37bcd8 c00444b0 c032efe4 60000113 ffffffff [ 18.918697] [<c000de80>] (__irq_svc+0x40/0x70) from [<c032efe4>] (_raw_spin_unlock_irq+0x1c/0x44) [ 18.918831] [<c032efe4>] (_raw_spin_unlock_irq+0x1c/0x44) from [<c00444b0>] (finish_task_switch+0x54/0xdc) [ 18.918965] [<c00444b0>] (finish_task_switch+0x54/0xdc) from [<c032db74>] (__schedule+0x208/0x66c) [ 18.919097] [<c032db74>] (__schedule+0x208/0x66c) from [<c032e458>] (preempt_schedule_irq+0x40/0x64) [ 18.919116] dma2chan0-copy7: #1: No errors with src_off=0x5000 dst_off=0x5028 len=0x8d77 (0) [ 18.919292] [<c032e458>] (preempt_schedule_irq+0x40/0x64) from [<c000deb8>] (svc_preempt+0x8/0x18) [ 18.919431] [<c000deb8>] (svc_preempt+0x8/0x18) from [<bf001e70>] (dmatest_verify.isra.9+0x14/0xb4 [dmatest]) [ 18.919499] unwind: Index not found bf001e70 [ 18.919582] Code: e3c3303f e5931004 e2811001 e5831004 (e1923f9f) [ 18.919644] ---[ end trace f24842049530d274 ]--- [ 18.919699] Kernel panic - not syncing: Fatal exception in interrupt [ 18.919754] CPU4: stopping [ 18.919822] CPU: 4 PID: 655 Comm: dma2chan0-copy7 Tainted: G D 3.10.0-rc1+ #15 [ 18.919949] [<c0013b50>] (unwind_backtrace+0x0/0xf8) from [<c0011434>] (show_stack+0x10/0x14) [ 18.920073] [<c0011434>] (show_stack+0x10/0x14) from [<c0012918>] (handle_IPI+0xf8/0x11c) [ 18.920183] [<c0012918>] (handle_IPI+0xf8/0x11c) from [<c000858c>] (gic_handle_irq+0x54/0x5c) [ 18.920299] [<c000858c>] (gic_handle_irq+0x54/0x5c) from [<c000de80>] (__irq_svc+0x40/0x70) [ 18.920369] Exception stack(0xdef23e18 to 0xdef23e60) [ 18.920442] 3e00: dfb4ce40 000106d3 [ 18.920555] 3e20: defa0000 000015d2 00020000 00012292 00008a83 00000080 ffffffff 000000c0 [ 18.920671] 3e40: 00008a83 def23f64 ffffff8e def23e60 bf000fb4 bf000fe8 80000113 ffffffff [ 18.920808] [<c000de80>] (__irq_svc+0x40/0x70) from [<bf000fe8>] (dmatest_func+0x4dc/0x12ec [dmatest]) [ 18.920944] [<bf000fe8>] (dmatest_func+0x4dc/0x12ec [dmatest]) from [<c003c1a4>] (kthread+0xa4/0xb0) [ 18.921066] [<c003c1a4>] (kthread+0xa4/0xb0) from [<c000e318>] (ret_from_fork+0x14/0x3c)
Quoted 11 lines · Will Deacon
I've been observing a regression in the dmatest module with 3.10-rc1. It manifests as either:
- a spurious timeout on one or more of the channel threads
- a complete kernel lockup (loss of console)
- a panic (see below, noting that the callback [dmatest_callback] is dereferencing a NULL pointer)If I revert
77101ce578bb("dmatest: cancel thread immediately when asked for") then things are rosy again, but I'm not sure if this is hiding another problem.
Right, so I think I understand what's causing this, but I'll leave it to Andriy to suggest a fix. The problem comes about because the dmatest module is now driven from debugfs, making it possible to unload the module whilst a test run is in progress. In this case:
- The DMA threads will return from wait_event_freezable_timeout(...) due to kthread_should_stop() returning true, and subsequently report failure because done.done is false.
- The DMA engines may not be idle, so the asynchronous callback can be invoked after we've started cleaning up, explaining the NULL dereference I'm seeing.
The solutions are either fixing the module exit code to cope with concurrent DMA transfers or to revert 77101ce578bb and not allow the channel threads to return mid-transfer.
Will
Quoted 22 lines · Will Deacon
Quoted 11 lines · Will Deacon
I've been observing a regression in the dmatest module with 3.10-rc1. It manifests as either:
- a spurious timeout on one or more of the channel threads
- a complete kernel lockup (loss of console)
- a panic (see below, noting that the callback [dmatest_callback] is dereferencing a NULL pointer)If I revert
77101ce578bb("dmatest: cancel thread immediately when asked for") then things are rosy again, but I'm not sure if this is hiding another problem.Right, so I think I understand what's causing this, but I'll leave it to Andriy to suggest a fix. The problem comes about because the dmatest module is now driven from debugfs, making it possible to unload the module whilst a test run is in progress. In this case:
- The DMA threads will return from
wait_event_freezable_timeout(...)due tokthread_should_stop()returning true, and subsequently report failure because done.done is false.
Quoted 7 lines
- The DMA engines may not be idle, so the asynchronous callback can be invoked after we've started cleaning up, explaining the NULL dereference I'm seeing.
The solutions are either fixing the module exit code to cope with concurrent DMA transfers or to revert
77101ce578bband not allow the channel threads to return mid-transfer.
We need to properly abort the channels on removal. This is already handled in the code but the kthread_stop is called after the transactions are aborted. It should be the other way round. Can you try with below patch
---
drivers/dma/dmatest.c +3 -3
@@ -822,6 +822,9 @@ static void dmatest_cleanup_channel(struct dmatest_chan *dtc)struct dmatest_thread *_thread;int ret; +/* terminate all transfers on specified channels */ +dmaengine_terminate_all (dtc->chan); +list_for_each_entry_safe (thread, _thread, &dtc->threads, node) { ret =kthread_stop (thread->task); pr_debug("dmatest: thread %s exited with status %d\n",
@@ -830,9 +833,6 @@ static void dmatest_cleanup_channel(struct dmatest_chan *dtc)kfree (thread); } -/* terminate all transfers on specified channels */ -dmaengine_terminate_all (dtc->chan); -kfree (dtc); }
--
~Vinod
Hi Vinod,
Thanks for the reply.
Quoted 14 lines · Vinod Koul
Quoted 13 lines · Will Deacon
Right, so I think I understand what's causing this, but I'll leave it to Andriy to suggest a fix. The problem comes about because the dmatest module is now driven from debugfs, making it possible to unload the module whilst a test run is in progress. In this case:
- The DMA threads will return from
wait_event_freezable_timeout(...)due tokthread_should_stop()returning true, and subsequently report failure because done.done is false.- The DMA engines may not be idle, so the asynchronous callback can be invoked after we've started cleaning up, explaining the NULL dereference I'm seeing.
Quoted 6 lines
Quoted 3 lines
The solutions are either fixing the module exit code to cope with concurrent DMA transfers or to revert
77101ce578bband not allow the channel threads to return mid-transfer.We need to properly abort the channels on removal. This is already handled in the code but the
kthread_stopis called after the transactions are aborted. It should be the other way round. Can you try with below patch
Unfortunately, I can trigger the exact same panic with this patch applied.
Isn't there a race between terminating the dmaengine transfers (dmaengine_terminate_all) and killing the test threads (kthread_stop) where a new transfer could be kicked off by dmatest_func?
Will
Quoted 4 lines · Will Deacon
Quoted 3 lines · Vinod Koul
Quoted 2 lines · Will Deacon
Right, so I think I understand what's causing this, but I'll leave it to Andriy to suggest a fix.
Thanks for the report.
Unfortunately neither me nor other guys in the team could not reproduce mentioned issue on our hardware.
Nevertheless I will try to allocate time and suggest possible fix if any comes to my mind. So, your testing and analysis is appreciated.
Signature · 2 lines
Andy Shevchenko <[email protected]> Intel Finland Oy
Hi Andy,
Quoted 6 lines · Andy Shevchenko
Quoted 4 lines · Will Deacon
Quoted 3 lines · Vinod Koul
Quoted 2 lines · Will Deacon
Right, so I think I understand what's causing this, but I'll leave it to Andriy to suggest a fix.
Quoted 6 lines
Thanks for the report.
Unfortunately neither me nor other guys in the team could not reproduce mentioned issue on our hardware.
Nevertheless I will try to allocate time and suggest possible fix if any comes to my mind. So, your testing and analysis is appreciated.
Sure. They way I trigger it is to unload the module whilst the DMA is ongoing (I'm using a software model, so I can make the DMA nice and slow -- I guess you could try using some large buffers).
My script (I just run from busybox) is:
#!/bin/sh
mount proc -t proc /proc
mount sysfs -t sysfs /sys
mount debugfs -t debugfs /sys/kernel/debug
echo 9 > /proc/sysrq-trigger
modprobe dmatest
echo 8 > /sys/kernel/debug/dmatest/iterations echo 8 > /sys/kernel/debug/dmatest/threads_per_chan echo 131072 > /sys/kernel/debug/dmatest/test_buf_size echo 1 > /sys/kernel/debug/dmatest/run modprobe -r dmatest
Will
Quoted 3 lines · Will Deacon
Sure. They way I trigger it is to unload the module whilst the DMA is ongoing (I'm using a software model, so I can make the DMA nice and slow -- I guess you could try using some large buffers).
Thank you for the script. It looks like I managed to reproduce it with help of your script. I'm about to send the patch. I will appreciate to collect a Tested-by tags in case it solves the issue.
Signature · 2 lines
Andy Shevchenko <[email protected]> Intel Finland Oy
PATCH 1/1 dmatest: abort transfers immediately when asked for
In reply to Will DeaconWhen thread is going to be stopped we have to unconditionally terminate all ongoing transfers. Otherwise it would be possible that callback function will be called on the next interrupt and will try to access to already freed structures.
The patch introduces specific error message for this, though it doesn't increase the counter of the failed tests.
--- drivers/dma/dmatest.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/dma/dmatest.c +11 -0
@@ -92,6 +92,7 @@ enum dmatest_error_type {DMATEST_ET_MAP_DST ,DMATEST_ET_PREP ,DMATEST_ET_SUBMIT , +DMATEST_ET_ABORT ,DMATEST_ET_TIMEOUT ,DMATEST_ET_DMA_ERROR , DMATEST_ET_DMA_IN_PROGRESS,
@@ -366,6 +367,7 @@ static char *thread_result_get(const char *name, [DMATEST_ET_MAP_DST] = "dst mapping error", [DMATEST_ET_PREP] = "prep error", [DMATEST_ET_SUBMIT] = "submit error", + [DMATEST_ET_ABORT] = "transfer aborted", [DMATEST_ET_TIMEOUT] = "test timed out", [DMATEST_ET_DMA_ERROR ] = "got completion callback (DMA_ERROR)",
@@ -720,6 +722,15 @@ static int dmatest_func(void *data) done.done ||kthread_should_stop (), msecs_to_jiffies(params->timeout)); +/* terminate all transfers on specified channel if needed */ +if (kthread_should_stop ()) { +dmaengine_terminate_all (chan); +thread_result_add (info, result,DMATEST_ET_ABORT , + total_tests, src_off, dst_off, + len,0 ); +break ; + } + status =dma_async_is_tx_complete (chan, cookie,NULL ,NULL );if (!done.done) {
Signature · 1 line
1.8.2.rc0.22.gb3600c3
Re: [PATCH] dmatest: abort transfers immediately when asked for
Hi Andy,
Quoted 10 lines · Andy Shevchenko
When thread is going to be stopped we have to unconditionally terminate all ongoing transfers. Otherwise it would be possible that callback function will be called on the next interrupt and will try to access to already freed structures.
The patch introduces specific error message for this, though it doesn't increase the counter of the failed tests.
Signed-off-by Andy Shevchenko <[email protected]>Reported-by Will Deacon <[email protected]>
Thanks for persevering with this! Although this patch definitely fixes the panic I was seeing, I now observe buffer verification failures in subsequent test runs after an aborted run:
[ 43.888424] dma0chan0-copy0: #1: transfer aborted with src_off=0x81f0 dst_off=0x9f4 len=0xdf29 (0) [ 43.888526] dma0chan0-copy0: terminating after 1 tests, 0 failures (status 0) [ 43.888635] dmatest: thread dma0chan0-copy0 exited with status 0
<reload module and start new tests>
[ 77.341405] dma0chan0-copy0: #1: errors with src_off=0x8e29 dst_off=0x11dc8 len=0x2b93 (370) [ 77.341505] dma0chan0-copy0: terminating after 1 tests, 1 failures (status 0)
Note that the non-aborted transfers (I have 4 dma controllers) seem fine.
Looking at the results file doesn't show anything obvious, although I do see some strange entries:
[...] dstbuf not copied! [0xffffffff] Expected ff, got ff dstbuf not copied! [0xffffffff] Expected ff, got ff dstbuf not copied! [0xffffffff] Expected 00, got 00 dstbuf not copied! [0xffffffff] Expected 00, got 00 dstbuf not copied! [0xffffffff] Expected ff, got ff [...]
The more alarming ones look like:
[...] dstbuf not copied! [0x147e9] Expected d5, got 36 dstbuf not copied! [0x147ea] Expected d4, got 35 dstbuf not copied! [0x147eb] Expected d3, got 34 dstbuf not copied! [0x147ec] Expected d2, got 33 dstbuf not copied! [0x147ed] Expected d1, got 32 dstbuf not copied! [0x147ee] Expected d0, got 31 [...]
Of course, this could be a driver bug with the PL330 that is being exposed by this aborting code.
Will
Re: [PATCH] dmatest: abort transfers immediately when asked for
Quoted 11 lines · Will Deacon
Hi Andy,
Quoted 8 lines · Andy Shevchenko
When thread is going to be stopped we have to unconditionally terminate all ongoing transfers. Otherwise it would be possible that callback function will be called on the next interrupt and will try to access to already freed structures.
The patch introduces specific error message for this, though it doesn't increase the counter of the failed tests.
Quoted 6 lines
Quoted 2 lines
Signed-off-by Andy Shevchenko <[email protected]>Reported-by Will Deacon <[email protected]>Thanks for persevering with this! Although this patch definitely fixes the panic I was seeing, I now observe buffer verification failures in subsequent test runs after an aborted run:
I think the description to the commit adfa543e "dmatest: don't use set_freezable_with_signal()" may shed light on this.
The background (if I got it correctly) is in race with done flag. So, we got a callback call from the DMA engine, but we don't know which transfer triggers it.
I might be wrong. This is just an assumption.
Have you ever see such behaviour on pre v3.10-rc1 kernels? (I mean with old dmatest module)
--
With Best Regards,
Andy Shevchenko
Re: [PATCH] dmatest: abort transfers immediately when asked for
Quoted 19 lines · Andy Shevchenko
Quoted 11 lines · Will Deacon
Hi Andy,
Quoted 8 lines · Andy Shevchenko
When thread is going to be stopped we have to unconditionally terminate all ongoing transfers. Otherwise it would be possible that callback function will be called on the next interrupt and will try to access to already freed structures.
The patch introduces specific error message for this, though it doesn't increase the counter of the failed tests.
Quoted 6 lines
Quoted 2 lines
Signed-off-by Andy Shevchenko <[email protected]>Reported-by Will Deacon <[email protected]>Thanks for persevering with this! Although this patch definitely fixes the panic I was seeing, I now observe buffer verification failures in subsequent test runs after an aborted run:
Quoted 7 lines
I think the description to the commit
adfa543e"dmatest: don't useset_freezable_with_signal()" may shed light on this.The background (if I got it correctly) is in race with done flag. So, we got a callback call from the DMA engine, but we don't know which transfer triggers it.
I might be wrong. This is just an assumption.
I've not managed to work out exactly what's going on, but it's certainly something like that. In fact, I just managed to trigger a case where all but one of the transfers is aborted, whilst the remaining one fails. Looking at the code, I can't see how that situation comes about, since the threads are protected with the info mutex and kthread_stop is synchronous.
Quoted 2 lines
Have you ever see such behaviour on pre v3.10-rc1 kernels? (I mean with old dmatest module)
No, dmatest from 3.9 is completely reliable in my experience.
Will
Re: [PATCH] dmatest: abort transfers immediately when asked for
Quoted 19 lines · Will Deacon
Quoted 11 lines · Andy Shevchenko
Quoted 2 lines · Will Deacon
I now observe buffer verification failures in subsequent test runs after an aborted run:
I think the description to the commit
adfa543e"dmatest: don't useset_freezable_with_signal()" may shed light on this.The background (if I got it correctly) is in race with done flag. So, we got a callback call from the DMA engine, but we don't know which transfer triggers it.
I might be wrong. This is just an assumption.I've not managed to work out exactly what's going on, but it's certainly something like that. In fact, I just managed to trigger a case where all but one of the transfers is aborted, whilst the remaining one fails. Looking at the code, I can't see how that situation comes about, since the threads are protected with the info mutex and
kthread_stopis synchronous.
Quoted 4 lines
Quoted 2 lines
Have you ever see such behaviour on pre v3.10-rc1 kernels? (I mean with old dmatest module)
No, dmatest from 3.9 is completely reliable in my experience.
Yeah, I supposed that was a rhetorical question.
So, have I understood correctly that if you revert the 77101ce5 ("cancel thread ...")
everything is working fine / as before?
--
With Best Regards,
Andy Shevchenko
Re: [PATCH] dmatest: abort transfers immediately when asked for
In reply to Andy ShevchenkoQuoted 22 lines · Andy Shevchenko
Quoted 11 lines · Will Deacon
Hi Andy,
Quoted 8 lines · Andy Shevchenko
When thread is going to be stopped we have to unconditionally terminate all ongoing transfers. Otherwise it would be possible that callback function will be called on the next interrupt and will try to access to already freed structures.
The patch introduces specific error message for this, though it doesn't increase the counter of the failed tests.
Quoted 6 lines
Quoted 2 lines
Signed-off-by Andy Shevchenko <[email protected]>Reported-by Will Deacon <[email protected]>Thanks for persevering with this! Although this patch definitely fixes the panic I was seeing, I now observe buffer verification failures in subsequent test runs after an aborted run:
I think the description to the commit
adfa543e"dmatest: don't useset_freezable_with_signal()" may shed light on this.
Quoted 7 lines
The background (if I got it correctly) is in race with done flag. So, we got a callback call from the DMA engine, but we don't know which transfer triggers it.
I might be wrong. This is just an assumption.Have you ever see such behaviour on pre v3.10-rc1 kernels? (I mean with old dmatest module)
Terminate shouldnt cause the issue with buffer verfication, can you try this on dw_dmac, do you see similar failures on verfication?
--
~Vinod
Re: [PATCH] dmatest: abort transfers immediately when asked for
Quoted 23 lines · Vinod Koul
Quoted 22 lines · Andy Shevchenko
Quoted 6 lines · Will Deacon
Quoted 5 lines · Andy Shevchenko
When thread is going to be stopped we have to unconditionally terminate all ongoing transfers. Otherwise it would be possible that callback function will be called on the next interrupt and will try to access to already freed structures.
Quoted 6 lines
Quoted 3 lines
The patch introduces specific error message for this, though it doesn't increase the counter of the failed tests.
Thanks for persevering with this! Although this patch definitely fixes the panic I was seeing, I now observe buffer verification failures in subsequent test runs after an aborted run:
I think the description to the commit
adfa543e"dmatest: don't useset_freezable_with_signal()" may shed light on this.The background (if I got it correctly) is in race with done flag. So, we got a callback call from the DMA engine, but we don't know which transfer triggers it.
I might be wrong. This is just an assumption.
Quoted 4 lines
Quoted 2 lines
Have you ever see such behaviour on pre v3.10-rc1 kernels? (I mean with old dmatest module)
Terminate shouldnt cause the issue with buffer verfication, can you try this on
dw_dmac, do you see similar failures on verfication?
I saw the similar errors on dw_dmac on Intel Medfield device. Anyway, I checked another approach with Will.
For now I will send a quick fix that prevents tester to abort an ongoing transfer.
In future we could implement a robust logic when transfers can be interrupted at any time.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH] dmatest: abort transfers immediately when asked for
Quoted 24 lines · Andy Shevchenko
Quoted 23 lines · Vinod Koul
Quoted 22 lines · Andy Shevchenko
Quoted 6 lines · Will Deacon
Quoted 5 lines · Andy Shevchenko
When thread is going to be stopped we have to unconditionally terminate all ongoing transfers. Otherwise it would be possible that callback function will be called on the next interrupt and will try to access to already freed structures.
Quoted 6 lines
Quoted 3 lines
The patch introduces specific error message for this, though it doesn't increase the counter of the failed tests.
Thanks for persevering with this! Although this patch definitely fixes the panic I was seeing, I now observe buffer verification failures in subsequent test runs after an aborted run:
I think the description to the commit
adfa543e"dmatest: don't useset_freezable_with_signal()" may shed light on this.The background (if I got it correctly) is in race with done flag. So, we got a callback call from the DMA engine, but we don't know which transfer triggers it.
I might be wrong. This is just an assumption.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Have you ever see such behaviour on pre v3.10-rc1 kernels? (I mean with old dmatest module)
Terminate shouldnt cause the issue with buffer verfication, can you try this on
dw_dmac, do you see similar failures on verfication?I saw the similar errors on
dw_dmacon Intel Medfield device.
Ah, so may not be a driver issue.
Quoted 3 lines
Anyway, I checked another approach with Will.
For now I will send a quick fix that prevents tester to abort an ongoing transfer.
okay so taht should prevent regression, let me check on my setup if I find anything
--
~Vinod
Quoted 6 lines
In future we could implement a robust logic when transfers can be interrupted at any time.
Signature · 2 lines
With Best Regards, Andy Shevchenko