Skip to contentSkip to conversation

dmaengine: Add Synopsys eDMA IP driver (version 0)

RFC Superseded Gustavo Pimentel 2 participants 4 messages Dec 17, 2018 – Jan 7, 2019 PCI

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover dmaengine: Add Synopsys eDMA IP driver (version 0)

Add Synopsys eDMA IP driver (version 0) to Linux kernel. This IP is generally distributed with Synopsys PCIe EndPoint IP (depends of the use and licensing agreement), which supports:
- legacy and unroll modes
- 16 independent and concurrent channels (8 write + 8 read)
- supports linked list (scatter-gather) transfer
- each linked list descriptor can transfer from 1 byte to 4 Gbytes
- PCIe EndPoint glue-logic

Gustavo Pimentel (6):
dmaengine: Add Synopsys eDMA IP core driver
dmaengine: Add Synopsys eDMA IP version 0 support
dmaengine: Add Synopsys eDMA IP version 0 debugfs support
PCI: Add Synopsys endpoint EDDA Device id
dmaengine: Add Synopsys eDMA IP PCIe glue-logic
MAINTAINERS: Add Synopsys eDMA IP driver maintainer

 MAINTAINERS                              |   7 +
 drivers/dma/Kconfig                      |   2 +
 drivers/dma/Makefile                     |   1 +
 drivers/dma/dw-edma/Kconfig              |  18 +
 drivers/dma/dw-edma/Makefile             |   7 +
 drivers/dma/dw-edma/dw-edma-core.c       | 845 +++++++++++++++++++++++++++++++
 drivers/dma/dw-edma/dw-edma-core.h       | 147 ++++++
 drivers/dma/dw-edma/dw-edma-pcie.c       | 258 ++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c    | 361 +++++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.h    |  26 +
 drivers/dma/dw-edma/dw-edma-v0-debugfs.c | 357 +++++++++++++
 drivers/dma/dw-edma/dw-edma-v0-debugfs.h |  24 +
 drivers/dma/dw-edma/dw-edma-v0-regs.h    | 145 ++++++
 drivers/misc/pci_endpoint_test.c         |   2 +-
 include/linux/dma/edma.h                 |  43 ++
 include/linux/pci_ids.h                  |   1 +
 16 files changed, 2243 insertions(+), 1 deletion(-)
 create mode 100644 drivers/dma/dw-edma/Kconfig
 create mode 100644 drivers/dma/dw-edma/Makefile
 create mode 100644 drivers/dma/dw-edma/dw-edma-core.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-core.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-pcie.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-core.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-core.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-debugfs.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-debugfs.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-regs.h
 create mode 100644 include/linux/dma/edma.h
Cc Vinod Koul <[email protected]>
Cc Andy Shevchenko <[email protected]>
Cc Russell King <[email protected]>
Cc Eugeniy Paltsev <[email protected]>
Cc Lorenzo Pieralisi <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc Kishon Vijay Abraham I <[email protected]>
Cc Niklas Cassel <[email protected]>
Cc Joao Pinto <[email protected]>
Cc Jose Abreu <[email protected]>
Cc Luis Oliveira <[email protected]>
Cc Vitor Soares <[email protected]>
Cc Nelson Costa <[email protected]>
Cc Pedro Sousa <[email protected]>
Signature · 1 line
2.7.4
#1 lore raw
Quoted 9 lines · Gustavo Pimentel

Add Synopsys eDMA IP driver (version 0) to Linux kernel. This IP is generally distributed with Synopsys PCIe EndPoint IP (depends of the use and licensing agreement), which supports:
- legacy and unroll modes
- 16 independent and concurrent channels (8 write + 8 read)
- supports linked list (scatter-gather) transfer
- each linked list descriptor can transfer from 1 byte to 4 Gbytes
- PCIe EndPoint glue-logic

Quoted 7 lines

Gustavo Pimentel (6):
dmaengine: Add Synopsys eDMA IP core driver
dmaengine: Add Synopsys eDMA IP version 0 support
dmaengine: Add Synopsys eDMA IP version 0 debugfs support
PCI: Add Synopsys endpoint EDDA Device id
dmaengine: Add Synopsys eDMA IP PCIe glue-logic
MAINTAINERS: Add Synopsys eDMA IP driver maintainer

Hello Gustavo,

Nice to see support for the embedded DMA controller used in the DWC PCIe controller.

Some kind of documentation in Documentation/ would be nice.

The eDMA can be used both while the PCIe controller is in EP and RC mode, right?
I guess that this support is only to use the eDMA for RC mode?

Do you have any benchmarks to share? (Perhaps with different buffer sizes). Not sure what we will see, lower CPU utilization but more IRQs? Have you thought about doing IRQ coalescing?

Usually we need dmas and dma-names in device tree, however, maybe that isn't needed here, since it is an embedded DMA controller. Don't we even need/want a DT property in the PCIe node, that says if we want to enable the eDMA or not?

Usually a PCIe EP driver will call pci_ioremap_bar() and then use readl()/writel() directly.
But since we now need to setup a DMA transfer, and wait for an IRQ, how does this work?

I have a board that has a CC_DMA_ENABLE set, so if you give me some detailed instructions how to test, I might be able to provide you with a Tested-by.

Kind regards,
Niklas

Quoted 49 lines
 MAINTAINERS                              |   7 +
 drivers/dma/Kconfig                      |   2 +
 drivers/dma/Makefile                     |   1 +
 drivers/dma/dw-edma/Kconfig              |  18 +
 drivers/dma/dw-edma/Makefile             |   7 +
 drivers/dma/dw-edma/dw-edma-core.c       | 845 +++++++++++++++++++++++++++++++
 drivers/dma/dw-edma/dw-edma-core.h       | 147 ++++++
 drivers/dma/dw-edma/dw-edma-pcie.c       | 258 ++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c    | 361 +++++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.h    |  26 +
 drivers/dma/dw-edma/dw-edma-v0-debugfs.c | 357 +++++++++++++
 drivers/dma/dw-edma/dw-edma-v0-debugfs.h |  24 +
 drivers/dma/dw-edma/dw-edma-v0-regs.h    | 145 ++++++
 drivers/misc/pci_endpoint_test.c         |   2 +-
 include/linux/dma/edma.h                 |  43 ++
 include/linux/pci_ids.h                  |   1 +
 16 files changed, 2243 insertions(+), 1 deletion(-)
 create mode 100644 drivers/dma/dw-edma/Kconfig
 create mode 100644 drivers/dma/dw-edma/Makefile
 create mode 100644 drivers/dma/dw-edma/dw-edma-core.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-core.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-pcie.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-core.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-core.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-debugfs.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-debugfs.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-regs.h
 create mode 100644 include/linux/dma/edma.h
Cc Vinod Koul <[email protected]>
Cc Andy Shevchenko <[email protected]>
Cc Russell King <[email protected]>
Cc Eugeniy Paltsev <[email protected]>
Cc Lorenzo Pieralisi <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc Kishon Vijay Abraham I <[email protected]>
Cc Niklas Cassel <[email protected]>
Cc Joao Pinto <[email protected]>
Cc Jose Abreu <[email protected]>
Cc Luis Oliveira <[email protected]>
Cc Vitor Soares <[email protected]>
Cc Nelson Costa <[email protected]>
Cc Pedro Sousa <[email protected]>
Signature · 1 line
2.7.4
#2 lore raw

Hi,

Quoted 18 lines · Niklas Cassel
Quoted 9 lines · Gustavo Pimentel

Add Synopsys eDMA IP driver (version 0) to Linux kernel. This IP is generally distributed with Synopsys PCIe EndPoint IP (depends of the use and licensing agreement), which supports:
- legacy and unroll modes
- 16 independent and concurrent channels (8 write + 8 read)
- supports linked list (scatter-gather) transfer
- each linked list descriptor can transfer from 1 byte to 4 Gbytes
- PCIe EndPoint glue-logic

Quoted 7 lines

Gustavo Pimentel (6):
dmaengine: Add Synopsys eDMA IP core driver
dmaengine: Add Synopsys eDMA IP version 0 support
dmaengine: Add Synopsys eDMA IP version 0 debugfs support
PCI: Add Synopsys endpoint EDDA Device id
dmaengine: Add Synopsys eDMA IP PCIe glue-logic
MAINTAINERS: Add Synopsys eDMA IP driver maintainer

Quoted 4 lines

Hello Gustavo,

Nice to see support for the embedded DMA controller used in the DWC PCIe controller.

Yes, I included you because you were from the beginning one of those who requested this feature. :)

Quoted 2 lines

Some kind of documentation in Documentation/ would be nice.

Yes, I thought so too, but under of DOCUMENTATION folder doesn't have anything related with DMA drivers implementations either on topic of DMA or PCI.

I thought maybe putting on the dw-edma folder but that doesn't follow the standard documentation flow.

Quoted 4 lines

The eDMA can be used both while the PCIe controller is in EP and RC mode, right?

For now I can not give this certainty. My setup only have the eDMA on the EP side and has the eDMA mapped on PCI BAR, which allows me to configure it through PCI.

Quoted 1 line

I guess that this support is only to use the eDMA for RC mode?

For now this driver runs on the RC side (host/CPU) and configures the eDMA EP. I've some plans to support eDMA configuration but on the EP side. I'm guessing that also typical use case.

Quoted 2 lines

Do you have any benchmarks to share? (Perhaps with different buffer sizes).

I'm currently working on the dw-edma-test driver (similar to dmatest driver) that supports DEV_TO_MEM and MEM_TO_DEV directions instead of MEM_TO_MEM (dmatest), and adapted to WRITE and READ channels with multiple threads. After finishing this driver I'll able to do some benchmarks and comparisons. :)

Given your enthusiasm, when I'm finished we could do some brainstorm and define some test cases / benchmarks, what do you think?

Quoted 1 line

Not sure what we will see, lower CPU utilization but more IRQs?

Probably yes. Let me explain you

With this implementation, you can transfer several chunks (a chunk is linked list of burst elements).

Each burst basically defines the source, destination addresses and size to transfer (from 1 Byte to 4 GB).

This linked list will be written on EP's RAM (mapped on a PCI BAR) which the HW block will consume it. The linked list size is dependent of this EP's RAM size, i.e. 8 KB will allow 340 elements (bursts).

On my currently setup I'm testing the driver implementation like this:
- 4 chunks
- each chunk is composed by 340 elements (bursts)
- each burst have a transfer size of 100 bytes (can be up to 4GB and the size
can be also different for each burst)

Chunk #0

       |--> Burst #0 --> ... --> Burst #339 -> IRQ (Reload new linked list)
Chunk #1
       |--> Burst #0 --> ... --> Burst #339 -> IRQ (Reload new linked list)
Chunk #2
       |--> Burst #0 --> ... --> Burst #339 -> IRQ (Reload new linked list)
Chunk #3
       |--> Burst #0 --> ... --> Burst #34 -> IRQ (End transfer)

In total I'm writing ((4 * 340) + 35) * 100 = 139500 Bytes <=> 136 KB

Quoted 1 line

Have you thought about doing IRQ coalescing?

No. I'm only focus to implement the driver for now. After the implementation and the benchmarks, maybe we could think about it. :)

Quoted 5 lines

Usually we need dmas and dma-names in device tree, however, maybe that isn't needed here, since it is an embedded DMA controller. Don't we even need/want a DT property in the PCIe node, that says if we want to enable the eDMA or not?

That will be necessary in the future when we have eDMA in RC or when we implement the driver to control the eDMA on the EP side.

Quoted 5 lines

Usually a PCIe EP driver will call pci_ioremap_bar() and then use readl()/writel() directly.
But since we now need to setup a DMA transfer, and wait for an IRQ, how does this work?

I gave a brief explanation in a previous comment in this email (see info about the chunks and bursts)

I'm dividing this patch series like this:

- eDMA core + eDMA core v0 driver (implementing the interface with DMAengine
controller APIs and the interface with the eDMA HW block)

- eDMA PCIe glue-logic driver (attaches to EP and provides memory access to
eDMA core driver), for now is attach to Synopsys Vendor and Device ID and has a platform data specific to our EP.

- eDMA Test driver (is a sample driver which aims to test the eDMA, but also
provides a example how an customer driver would use the eDMA driver, through the DMAengine client APIs) (under developing, I hope to make it available before xmas :))

Sounds logic?

Quoted 4 lines

I have a board that has a CC_DMA_ENABLE set, so if you give me some detailed instructions how to test, I might be able to provide you with a Tested-by.

Ok, let me just finish the eDMA test and append it to the patch series. I'm happy to help you with this. :)

Thanks!

Gustavo

Quoted 55 lines

Kind regards,
Niklas

Quoted 49 lines
 MAINTAINERS                              |   7 +
 drivers/dma/Kconfig                      |   2 +
 drivers/dma/Makefile                     |   1 +
 drivers/dma/dw-edma/Kconfig              |  18 +
 drivers/dma/dw-edma/Makefile             |   7 +
 drivers/dma/dw-edma/dw-edma-core.c       | 845 +++++++++++++++++++++++++++++++
 drivers/dma/dw-edma/dw-edma-core.h       | 147 ++++++
 drivers/dma/dw-edma/dw-edma-pcie.c       | 258 ++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c    | 361 +++++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.h    |  26 +
 drivers/dma/dw-edma/dw-edma-v0-debugfs.c | 357 +++++++++++++
 drivers/dma/dw-edma/dw-edma-v0-debugfs.h |  24 +
 drivers/dma/dw-edma/dw-edma-v0-regs.h    | 145 ++++++
 drivers/misc/pci_endpoint_test.c         |   2 +-
 include/linux/dma/edma.h                 |  43 ++
 include/linux/pci_ids.h                  |   1 +
 16 files changed, 2243 insertions(+), 1 deletion(-)
 create mode 100644 drivers/dma/dw-edma/Kconfig
 create mode 100644 drivers/dma/dw-edma/Makefile
 create mode 100644 drivers/dma/dw-edma/dw-edma-core.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-core.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-pcie.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-core.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-core.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-debugfs.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-debugfs.h
 create mode 100644 drivers/dma/dw-edma/dw-edma-v0-regs.h
 create mode 100644 include/linux/dma/edma.h
Cc Vinod Koul <[email protected]>
Cc Andy Shevchenko <[email protected]>
Cc Russell King <[email protected]>
Cc Eugeniy Paltsev <[email protected]>
Cc Lorenzo Pieralisi <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc Kishon Vijay Abraham I <[email protected]>
Cc Niklas Cassel <[email protected]>
Cc Joao Pinto <[email protected]>
Cc Jose Abreu <[email protected]>
Cc Luis Oliveira <[email protected]>
Cc Vitor Soares <[email protected]>
Cc Nelson Costa <[email protected]>
Cc Pedro Sousa <[email protected]>
Signature · 1 line
2.7.4
#3 lore raw
Quoted 21 lines · Gustavo Pimentel

Hi,

Quoted 18 lines · Niklas Cassel
Quoted 16 lines · Gustavo Pimentel

Add Synopsys eDMA IP driver (version 0) to Linux kernel. This IP is generally distributed with Synopsys PCIe EndPoint IP (depends of the use and licensing agreement), which supports:
- legacy and unroll modes
- 16 independent and concurrent channels (8 write + 8 read)
- supports linked list (scatter-gather) transfer
- each linked list descriptor can transfer from 1 byte to 4 Gbytes
- PCIe EndPoint glue-logic

Gustavo Pimentel (6):
dmaengine: Add Synopsys eDMA IP core driver
dmaengine: Add Synopsys eDMA IP version 0 support
dmaengine: Add Synopsys eDMA IP version 0 debugfs support
PCI: Add Synopsys endpoint EDDA Device id
dmaengine: Add Synopsys eDMA IP PCIe glue-logic
MAINTAINERS: Add Synopsys eDMA IP driver maintainer

Quoted 7 lines
Quoted 4 lines

Hello Gustavo,

Nice to see support for the embedded DMA controller used in the DWC PCIe controller.

Yes, I included you because you were from the beginning one of those who requested this feature. :)

Hello Gustavo,

sorry for the delayed response, I took an extra long christmas vacation :)

Unfortunately, it was my previous employer (Axis) who was interested in this. Right now I am simply looking at this patch series out of my own curiosity :)

(snip)

Quoted 6 lines
Quoted 2 lines

The eDMA can be used both while the PCIe controller is in EP and RC mode, right?

For now I can not give this certainty. My setup only have the eDMA on the EP side and has the eDMA mapped on PCI BAR, which allows me to configure it through PCI.

Ok, I looked at the databook, and it does mention support for both modes.

Perhaps the cover letter could mention more clearly that this is just for eDMA on the EP side. (Since the IP seems to support eDMA for both RC and EP side.)

(snip)

Quoted 9 lines
Quoted 1 line

Do you have any benchmarks to share? (Perhaps with different buffer sizes).

I'm currently working on the dw-edma-test driver (similar to dmatest driver) that supports DEV_TO_MEM and MEM_TO_DEV directions instead of MEM_TO_MEM (dmatest), and adapted to WRITE and READ channels with multiple threads. After finishing this driver I'll able to do some benchmarks and comparisons. :)

Given your enthusiasm, when I'm finished we could do some brainstorm and define some test cases / benchmarks, what do you think?

I might have the enthusiasm, but I don't have the expertise to come up with some relevant use-cases :p

Quoted 30 lines
Quoted 2 lines

Not sure what we will see, lower CPU utilization but more IRQs?

Probably yes. Let me explain you

With this implementation, you can transfer several chunks (a chunk is linked list of burst elements).

Each burst basically defines the source, destination addresses and size to transfer (from 1 Byte to 4 GB).

This linked list will be written on EP's RAM (mapped on a PCI BAR) which the HW block will consume it. The linked list size is dependent of this EP's RAM size, i.e. 8 KB will allow 340 elements (bursts).

On my currently setup I'm testing the driver implementation like this:
- 4 chunks
- each chunk is composed by 340 elements (bursts)
- each burst have a transfer size of 100 bytes (can be up to 4GB and the size
can be also different for each burst)

Chunk #0

       |--> Burst #0 --> ... --> Burst #339 -> IRQ (Reload new linked list)
Chunk #1
       |--> Burst #0 --> ... --> Burst #339 -> IRQ (Reload new linked list)
Chunk #2
       |--> Burst #0 --> ... --> Burst #339 -> IRQ (Reload new linked list)
Chunk #3
       |--> Burst #0 --> ... --> Burst #34 -> IRQ (End transfer)
Quoted 1 line

In total I'm writing ((4 * 340) + 35) * 100 = 139500 Bytes <=> 136 KB

This is a nice explaination, and it probably belongs in the cover letter, and/or commit message, and/or somewhere in Documentation/, and/or as a comment in the source code.

(snip)

Quoted 9 lines

I'm dividing this patch series like this:

- eDMA core + eDMA core v0 driver (implementing the interface with DMAengine
controller APIs and the interface with the eDMA HW block)

- eDMA PCIe glue-logic driver (attaches to EP and provides memory access to
eDMA core driver), for now is attach to Synopsys Vendor and Device ID and has a platform data specific to our EP.

Quoted 6 lines

- eDMA Test driver (is a sample driver which aims to test the eDMA, but also
provides a example how an customer driver would use the eDMA driver, through the DMAengine client APIs) (under developing, I hope to make it available before xmas :))

Sounds logic?

This definitely belongs in the cover letter :)

Quoted 8 lines
Quoted 5 lines

I have a board that has a CC_DMA_ENABLE set, so if you give me some detailed instructions how to test, I might be able to provide you with a Tested-by.

Ok, let me just finish the eDMA test and append it to the patch series. I'm happy to help you with this. :)

The hardware I currently have access to (Qualcomm hardware), hasn't implemented the software support for running the PCIe controller in endpoint mode yet.
This means that I won't be able to help you with testing this series. Sorry for that. (I wanted to help out here, promise :p).

Hopefully there is someone else who has this IP configured with CC_DMA_ENABLE + has EP support implemented, who can help you out :)

Kind regards,
Niklas