Problems with Freescale DMA driver
Hello LKML,
I'm developing on a Freescale MPC8349EA MDS evaluation board. I'm currently testing the DMA controller. It uses the CONFIG_FSL_DMA driver.
All tests were done with a recent git pull.
I built the CONFIG_DMATEST DMA test client, and it runs perfectly in the absence of other DMA activity. Enabling CONFIG_NET_DMA and running the DMA test client while network traffic is running causes failures such as the following:
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
dma0chan0-test0: dstbuf[0x9c0] mismatch! Expected c2, got df
dma0chan0-test0: dstbuf[0x9c1] mismatch! Expected c1, got df
dma0chan0-test0: dstbuf[0x9c2] mismatch! Expected c0, got df
dma0chan0-test0: dstbuf[0x9c3] not copied! Expected df, got 3c
dma0chan0-test0: dstbuf[0x9c4] not copied! Expected de, got 3b
dma0chan0-test0: dstbuf[0x9c5] not copied! Expected dd, got 3a
dma0chan0-test0: dstbuf[0x9c6] not copied! Expected dc, got 39
dma0chan0-test0: dstbuf[0x9c7] not copied! Expected db, got 38
dma0chan0-test0: dstbuf[0x9c8] not copied! Expected da, got 37
dma0chan0-test0: dstbuf[0x9c9] not copied! Expected d9, got 36
dma0chan0-test0: dstbuf[0x9ca] not copied! Expected d8, got 35
dma0chan0-test0: dstbuf[0x9d8] mismatch! Expected ca, got c2
dma0chan0-test0: dstbuf[0x9d9] mismatch! Expected c9, got c1
dma0chan0-test0: dstbuf[0x9da] mismatch! Expected c8, got c0
dma0chan0-test0: dstbuf[0x9db] mismatch! Expected c7, got df
dma0chan0-test0: dstbuf[0x9dc] mismatch! Expected c6, got de
dma0chan0-test0: dstbuf[0x9dd] mismatch! Expected c5, got dd
dma0chan0-test0: dstbuf[0x9de] mismatch! Expected c4, got dc
dma0chan0-test0: dstbuf[0x9df] mismatch! Expected c3, got db
dma0chan0-test0: dstbuf[0x9e0] mismatch! Expected c2, got da
dma0chan0-test0: dstbuf[0x9e1] mismatch! Expected c1, got d9
dma0chan0-test0: dstbuf[0x9e2] mismatch! Expected c0, got d8
dma0chan0-test0: dstbuf[0x9e3] mismatch! Expected df, got d7
dma0chan0-test0: dstbuf[0x9e4] mismatch! Expected de, got d6
dma0chan0-test0: dstbuf[0x9e5] mismatch! Expected dd, got d5
dma0chan0-test0: dstbuf[0x9e6] mismatch! Expected dc, got d4
dma0chan0-test0: dstbuf[0x9e7] mismatch! Expected db, got d3
dma0chan0-test0: dstbuf[0x9e8] mismatch! Expected da, got d2
dma0chan0-test0: dstbuf[0x9e9] mismatch! Expected d9, got d1
dma0chan0-test0: dstbuf[0x9ea] mismatch! Expected d8, got d0
dma0chan0-test0: dstbuf[0x9eb] mismatch! Expected d7, got cf
dma0chan0-test0: dstbuf[0x9ec] mismatch! Expected d6, got ce
dma0chan0-test0: 7011 errors suppressed
dma0chan0-test0: #12268: 7043 errors with src_off=0x1b6b dst_off=0x96e len=0x1be2
Is this to be expected? I thought the whole point of the DMAEngine API was to enable sharing of DMA channels.
Another issue I noticed was running a network transfer from another computer to the board. I get the following in dmesg, varying numbers of times. This is with no other DMA activity running, dmatest is not loaded. NET_DMA should be the only thing using the controller:
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
On the run where the above errors were printed, /sys/class/dma/dma0chan0/memcpy_count increased by ~36000. Certainly the DMA halt error doesn't happen for every single dma transaction.
Is this normal behavior, or is something broken in the driver?
I've CC'd the maintainers responsible for the FSL DMA driver, as well as the generic DMA offload system.
Please CC me on any responses, I'm not subscribed to LKML.
I'm happy to help test any changes, or add information if it is needed, please just ask.
Thanks,
Ira
[adding Timur to the cc]
Quoted 82 lines · Ira Snyder
Hello LKML,
I'm developing on a Freescale MPC8349EA MDS evaluation board. I'm currently testing the DMA controller. It uses the
CONFIG_FSL_DMAdriver.All tests were done with a recent git pull.
I built the
CONFIG_DMATESTDMA test client, and it runs perfectly in the absence of other DMA activity. EnablingCONFIG_NET_DMAand running the DMA test client while network traffic is running causes failures such as the following:of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
dma0chan0-test0: dstbuf[0x9c0] mismatch! Expected c2, got df
dma0chan0-test0: dstbuf[0x9c1] mismatch! Expected c1, got df
dma0chan0-test0: dstbuf[0x9c2] mismatch! Expected c0, got df
dma0chan0-test0: dstbuf[0x9c3] not copied! Expected df, got 3c
dma0chan0-test0: dstbuf[0x9c4] not copied! Expected de, got 3b
dma0chan0-test0: dstbuf[0x9c5] not copied! Expected dd, got 3a
dma0chan0-test0: dstbuf[0x9c6] not copied! Expected dc, got 39
dma0chan0-test0: dstbuf[0x9c7] not copied! Expected db, got 38
dma0chan0-test0: dstbuf[0x9c8] not copied! Expected da, got 37
dma0chan0-test0: dstbuf[0x9c9] not copied! Expected d9, got 36
dma0chan0-test0: dstbuf[0x9ca] not copied! Expected d8, got 35
dma0chan0-test0: dstbuf[0x9d8] mismatch! Expected ca, got c2
dma0chan0-test0: dstbuf[0x9d9] mismatch! Expected c9, got c1
dma0chan0-test0: dstbuf[0x9da] mismatch! Expected c8, got c0
dma0chan0-test0: dstbuf[0x9db] mismatch! Expected c7, got df
dma0chan0-test0: dstbuf[0x9dc] mismatch! Expected c6, got de
dma0chan0-test0: dstbuf[0x9dd] mismatch! Expected c5, got dd
dma0chan0-test0: dstbuf[0x9de] mismatch! Expected c4, got dc
dma0chan0-test0: dstbuf[0x9df] mismatch! Expected c3, got db
dma0chan0-test0: dstbuf[0x9e0] mismatch! Expected c2, got da
dma0chan0-test0: dstbuf[0x9e1] mismatch! Expected c1, got d9
dma0chan0-test0: dstbuf[0x9e2] mismatch! Expected c0, got d8
dma0chan0-test0: dstbuf[0x9e3] mismatch! Expected df, got d7
dma0chan0-test0: dstbuf[0x9e4] mismatch! Expected de, got d6
dma0chan0-test0: dstbuf[0x9e5] mismatch! Expected dd, got d5
dma0chan0-test0: dstbuf[0x9e6] mismatch! Expected dc, got d4
dma0chan0-test0: dstbuf[0x9e7] mismatch! Expected db, got d3
dma0chan0-test0: dstbuf[0x9e8] mismatch! Expected da, got d2
dma0chan0-test0: dstbuf[0x9e9] mismatch! Expected d9, got d1
dma0chan0-test0: dstbuf[0x9ea] mismatch! Expected d8, got d0
dma0chan0-test0: dstbuf[0x9eb] mismatch! Expected d7, got cf
dma0chan0-test0: dstbuf[0x9ec] mismatch! Expected d6, got ce
dma0chan0-test0: 7011 errors suppressed
dma0chan0-test0: #12268: 7043 errors with src_off=0x1b6b dst_off=0x96e len=0x1be2Is this to be expected? I thought the whole point of the DMAEngine API was to enable sharing of DMA channels.
Another issue I noticed was running a network transfer from another computer to the board. I get the following in dmesg, varying numbers of times. This is with no other DMA activity running, dmatest is not loaded.
NET_DMAshould be the only thing using the controller:of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout! of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
On the run where the above errors were printed,
/sys/class/dma/dma0chan0/memcpy_countincreased by ~36000. Certainly the DMA halt error doesn't happen for every single dma transaction.Is this normal behavior, or is something broken in the driver?
I've CC'd the maintainers responsible for the FSL DMA driver, as well as the generic DMA offload system.
Please CC me on any responses, I'm not subscribed to LKML.
I'm happy to help test any changes, or add information if it is needed, please just ask.
Thanks,
Ira
Quoted 9 lines · Dan Williams
[adding Timur to the cc]
Quoted 6 lines · Ira Snyder
Hello LKML,
I'm developing on a Freescale MPC8349EA MDS evaluation board. I'm currently testing the DMA controller. It uses the
CONFIG_FSL_DMAdriver.All tests were done with a recent git pull.
I just posted two patches lkml that affect the FSL DMA driver:
[PATCH v2] dmatest: properly handle duplicate DMA channels [PATCH] fsl-dma: allow Freescale Elo DMA driver to be compiled as a module
Could you apply them both and re-run the tests?
I will also try to duplicate this here on an 8360.
Signature · 2 lines
Timur Tabi Linux kernel developer at Freescale
Quoted 9 lines · Timur Tabi
Quoted 8 lines · Dan Williams
[adding Timur to the cc]
Quoted 5 lines · Ira Snyder
Hello LKML,
I'm developing on a Freescale MPC8349EA MDS evaluation board. I'm currently testing the DMA controller. It uses the
CONFIG_FSL_DMAdriver.
Quoted 9 lines
Quoted 1 line
Quoted 1 line
All tests were done with a recent git pull.
I just posted two patches lkml that affect the FSL DMA driver:
[PATCH v2] dmatest: properly handle duplicate DMA channels [PATCH] fsl-dma: allow Freescale Elo DMA driver to be compiled as a module
Could you apply them both and re-run the tests?
I have tested today's git pull with your patches. I had to add the cell-index property to each DMA channel in the dts for my board. I just added them as 0, 1, 2, and 3 respectively.
When your patch makes it in, you should probably add the cell-index to all of the dts files that are missing them.
I was still able to cause the "DMA Halt Timeout" error message (on channel 0, just like in the previous report).
To cause this, I built a kernel with CONFIG_DMATEST=y and CONFIG_NET_DMA=y. Then, I ran the following concurrently:
netperf -H localhost -l 600
netperf -H 192.168.17.65 -l 600
While they were running, I got the "DMA Halt Timeout" message 6 times. Here is the output (all messages were the same):of_platform e0008100.dma-channe: DMA halt timeout!
The following output leads me to believe that NET_DMA is using channel 0 only:
cat /sys/class/dma/*/memcpy_count
1278897
198585
198681
198591
I was not able to cause the DMA controller to copy bad data. Perhaps the kernel just had a bug at the time. It was still relatively early on in the development cycle.
Quoted 2 lines
I will also try to duplicate this here on an 8360.
Thanks.
In a related note, I've been somewhat following the discussion on LKML about creating an API for requesting a single DMA channel. This would be great in a driver I've written for PCI communication over a backplane. (I have a test board running in PCI Agent mode. It creates a virtual ethernet interface that passes data over the PCI bus.)
Also, are there any plans to support the external start feature on the 83XX parts? (According to my datasheet, it is supported.) I will be using this feature in a driver I will be starting fairly soon.
Thanks,
Ira
Quoted 2 lines · Ira Snyder
When your patch makes it in, you should probably add the cell-index to all of the dts files that are missing them.
Ah, I see that Kumar added a bunch of DMA nodes back in June and forgot the cell-index properties.
Quoted 3 lines
I was not able to cause the DMA controller to copy bad data. Perhaps the kernel just had a bug at the time. It was still relatively early on in the development cycle.
dmatest runs all the time, so it's possible that it's eating up DMA resources. Try running your tests without CONFIG_DMATEST.
Quoted 5 lines
In a related note, I've been somewhat following the discussion on LKML about creating an API for requesting a single DMA channel. This would be great in a driver I've written for PCI communication over a backplane. (I have a test board running in PCI Agent mode. It creates a virtual ethernet interface that passes data over the PCI bus.)
PCI devices should not be using generic DMA resources, so I don't see how this would apply.
Quoted 3 lines
Also, are there any plans to support the external start feature on the 83XX parts? (According to my datasheet, it is supported.) I will be using this feature in a driver I will be starting fairly soon.
External start is very device-specific, so I don't see how a generic DMA driver, which is intended only for memory-to-memory DMA operations, can coordinate with an external master.
Signature · 2 lines
Timur Tabi Linux kernel developer at Freescale
Quoted 8 lines · Timur Tabi
Quoted 2 lines · Ira Snyder
When your patch makes it in, you should probably add the cell-index to all of the dts files that are missing them.
Ah, I see that Kumar added a bunch of DMA nodes back in June and forgot the cell-index properties.
Quoted 7 lines
Quoted 3 lines
I was not able to cause the DMA controller to copy bad data. Perhaps the kernel just had a bug at the time. It was still relatively early on in the development cycle.
dmatest runs all the time, so it's possible that it's eating up DMA resources. Try running your tests without
CONFIG_DMATEST.
I am unable to cause any problems without dmatest running. You are probably correct, it is just using lots of resources.
Quoted 9 lines
Quoted 5 lines
In a related note, I've been somewhat following the discussion on LKML about creating an API for requesting a single DMA channel. This would be great in a driver I've written for PCI communication over a backplane. (I have a test board running in PCI Agent mode. It creates a virtual ethernet interface that passes data over the PCI bus.)
PCI devices should not be using generic DMA resources, so I don't see how this would apply.
Let me try to explain what I'm doing. If you have any ideas of a better way to do this, I'm completely open to suggestions.
I have the following:
1) A normal x86 desktop computer, running Linux
2) An MPC8349EMDS eval board, running in PCI agent/slave mode. It is plugged into a PCI slot in the desktop. It also runs Linux.
Naturally, I want to transfer lots of data between them. They are already connected by the PCI bus, so transferring data across that seems logical. Using the network infrastructure in the kernel seemed like a good solution, since it handles multiplexing/demultiplexing of traffic nicely. I chose to use an ethernet device because there is some documentation for them in LDD3.
So I wrote a device driver for both computers which creates an ethernet interface. The actual data transfer must happen over the PCI bus, which just shows up as memory in the MPC8349's address space.
Since using the CPU to do data transfer is very slow, I need to use the DMA controller to transfer the data. I used the DMAEngine API because it existed, and so I didn't have to program the DMA controller registers manually. The MPC8349 handles all of the data transfer using its DMA controller.
I hope that makes sense. I can post the code if anyone would like to see it. It is not ready for mainline inclusion yet, but it is in a fairly good state.
Quoted 8 lines
Quoted 3 lines
Also, are there any plans to support the external start feature on the 83XX parts? (According to my datasheet, it is supported.) I will be using this feature in a driver I will be starting fairly soon.
External start is very device-specific, so I don't see how a generic DMA driver, which is intended only for memory-to-memory DMA operations, can coordinate with an external master.
Ok. I was just wondering, since drivers/dma/fsl_dma.c has some code that appears to handle external master. It is 85XX specific though. There are no users in the kernel tree showing example uses.
I will be using it in the same setup as above. The board I am working on is based on the MPC8349EMDS board, but with 5 FPGA's on it for data processing. The FPGA's will do some calculations, then use the external start to trigger a DMA to copy the data from the FPGA memory to the MPC8349's DDR, where it is buffered up for later processing.
Sorry if this has gone off topic.
Thanks,
Ira
(Removing various CC:'s since this thread is changing topic)
Quoted 3 lines · Ira Snyder
I hope that makes sense. I can post the code if anyone would like to see it. It is not ready for mainline inclusion yet, but it is in a fairly good state.
Well, it's a bit goofy, but it does make sense. Since you don't have an actual single-function PCI device, you need to use the MPC8349's DMA controller.
I don't see where you need the ability to reserve a DMA channel, though. You're using fsldma.c as its intended.
Quoted 3 lines
Ok. I was just wondering, since
drivers/dma/fsl_dma.chas some code that appears to handle external master. It is 85XX specific though. There are no users in the kernel tree showing example uses.
Indeed it does. It looks like a feature that was thrown in but never given much thought or testing. It appears that the current code just lets you turn on the external master pause bit in the MR register, but there's nothing to make sure that it actually works in a meaningful way. If there's a queue of transfers for a given channel, I don't see where the fsldma avoids stalled queues.
Signature · 2 lines
Timur Tabi Linux kernel developer at Freescale
Quoted 8 lines · Timur Tabi
(Removing various CC:'s since this thread is changing topic)
Quoted 3 lines · Ira Snyder
I hope that makes sense. I can post the code if anyone would like to see it. It is not ready for mainline inclusion yet, but it is in a fairly good state.
Quoted 6 lines
Well, it's a bit goofy, but it does make sense. Since you don't have an actual single-function PCI device, you need to use the MPC8349's DMA controller.
I don't see where you need the ability to reserve a DMA channel, though. You're using
fsldma.cas its intended.
Ok. I don't really care which channel I get, just that I get one. I won't be adding or removing DMA devices from the system, so it shouldn't be a problem.
Quoted 10 lines
Quoted 3 lines
Ok. I was just wondering, since
drivers/dma/fsl_dma.chas some code that appears to handle external master. It is 85XX specific though. There are no users in the kernel tree showing example uses.Indeed it does. It looks like a feature that was thrown in but never given much thought or testing. It appears that the current code just lets you turn on the external master pause bit in the MR register, but there's nothing to make sure that it actually works in a meaningful way. If there's a queue of transfers for a given channel, I don't see where the fsldma avoids stalled queues.
That was also my impression. I'll probably start by programming the registers directly, and hopefully come up with a nice interface. When I figure it out, I'll send patches.
In this case, I probably would like to reserve a DMA channel for my exclusive use. That way I won't run into problems by programming the DMA by hand.
Thanks for all the input.
Ira
Quoted 3 lines · Ira Snyder
In this case, I probably would like to reserve a DMA channel for my exclusive use. That way I won't run into problems by programming the DMA by hand.
If you write your own DMA driver, the current method for reserving a DMA channel is to change the compatible property for that channel in the device tree to something other than "fsl,elo-dma-channel" (or "fsl,eloplus-dma-channel").
Signature · 2 lines
Timur Tabi Linux kernel developer at Freescale