staging: fsl-mc: add DPAA2 overview readme
PATCH v2 1/1 staging: fsl-mc: add DPAA2 overview readme
add README file providing an overview of the DPAA2 architecture and how it is integrated in Linux
---
-v2: added changelog text
drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++ drivers/staging/fsl-mc/TODO | 4 - 2 files changed, 364 insertions(+), 4 deletions(-) create mode 100644 drivers/staging/fsl-mc/README.txt
drivers/staging/fsl-mc/README.txt new file +364 -0
@@ -0,0 +1,364 @@ +Copyright (C) 2015 Freescale Semiconductor Inc. + +DPAA2 (Data Path Acceleration Architecture Gen2) +------------------------------------------------ + +This document provides an overview of the Freescale DPAA2 architecture +and how it is integrated into the Linux kernel. + +Contents summary + -DPAA2 overview + -Overview of DPAA2 objects + -DPAA2 Linux driver architecture overview + -bus driver + -dprc driver + -allocator + -dpio driver + -Ethernet + -mac + +DPAA2 Overview +-------------- + +DPAA2 is a hardware architecture designed for high-speeed network +packet processing. DPAA2 consists of sophisticated mechanisms for +processing Ethernet packets, queue management, buffer management, +autonomous L2 switching, virtual Ethernet bridging, and accelerator +(e.g. crypto) sharing. + +A DPAA2 hardware component called the Management Complex (or MC) manages the +DPAA2 hardware resources. The MC provides an object-based abstraction for +software drivers to use the DPAA2 hardware. + +The MC uses DPAA2 hardware resources such as queues, buffer pools, and +network ports to create functional objects/devices such as network +interfaces, an L2 switch, or accelerator instances. + +The MC provides memory-mapped I/O command interfaces (MC portals) +which DPAA2 software drivers use to operate on DPAA2 objects: + + +--------------------------------------+ + | OS | + | DPAA2 drivers | + | | | + +-----------------------------|--------+ + | + | (create,discover,connect + | config,use,destroy) + | + DPAA2 | + +------------------------| mc portal |-+ + | | | + | +- - - - - - - - - - - - -V- - -+ | + | | | | + | | Management Complex (MC) | | + | | | | + | +- - - - - - - - - - - - - - - -+ | + | | + | Hardware Hardware | + | Resources Objects | + | --------- ------- | + | -queues -DPRC | + | -buffer pools -DPMCP | + | -Eth MACs/ports -DPIO | + | -network interface -DPNI | + | profiles -DPMAC | + | -queue portals -DPBP | + | -MC portals ... | + | ... | + | | + +--------------------------------------+ + +The MC mediates operations such as create, discover, +connect, configuration, and destroy. Fast-path operations +on data, such as packet transmit/receive, are not mediated by +the MC and are done directly using memory mapped regions in +DPIO objects. + +Overview of DPAA2 Objects +------------------------- +The section provides a brief overview of some key objects +in the DPAA2 hardware. A simple scenario is described illustrating +the objects involved in creating a network interfaces. + +-DPRC (Datapath Resource Container) + + A DPRC is an container object that holds all the other + types of DPAA2 objects. In the example diagram below there + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) + in the container. + + +---------------------------------------------------------+ + | DPRC | + | | + | +-------+ +-------+ +-------+ +-------+ +-------+ | + | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | + | +-------+ +-------+ +-------+ +---+---+ +---+---+ | + | | DPMCP | | DPIO | | + | +-------+ +-------+ | + | | DPMCP | | + | +-------+ | + | | + +---------------------------------------------------------+ + + From the point of view of an OS, a DPRC is bus-like. Like + a plug-and-play bus, such as PCI, DPRC commands can be used to + enumerate the contents of the DPRC, discover the hardware + objects present (including mappable regions and interrupts). + + dprc.1 (bus) + | + +--+--------+-------+-------+-------+ + | | | | | + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 + dpmcp.2 dpio.2 + dpmcp.3 + + Hardware objects can be created and destroyed dynamically, providing + the ability to hot plug/unplug objects in and out of the DPRC. + + A DPRC has a mappable mmio region (an MC portal) that can be used + to send MC commands. It has an interrupt for status events (like + hotplug). + + All objects in a container share the same hardware "isolation context". + This means that with respect to an IOMMU the isolation granularity + is at the DPRC (container) level, not at the individual object + level. + + DPRCs can be defined statically and populated with objects + via a config file passed to the MC when firmware starts + it. There is also a Linux user space tool called "restool" + that can be used to create/destroy containers and objects + dynamically. + +-DPAA2 Objects for an Ethernet Network Interface + + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX + queuing mechanisms, configuration mechanisms, buffer management, + physical ports, and interrupts. DPAA2 uses a more granular approach + utilizing multiple hardware objects. Each object has specialized + functions, and are used together by software to provide Ethernet network + interface functionality. This approach provides efficient use of finite + hardware resources, flexibility, and performance advantages. + + The diagram below shows the objects needed for a simple + network interface configuration on a system with 2 CPUs. + + +---+---+ +---+---+ + CPU0 CPU1 + +---+---+ +---+---+ + | | + +---+---+ +---+---+ + DPIO DPIO + +---+---+ +---+---+ + \ / + \ / + \ / + +---+---+ + DPNI --- DPBP,DPMCP + +---+---+ + | + | + +---+---+ + DPMAC + +---+---+ + | + port/PHY + + Below the objects are described. For each object a brief description + is provided along with a summary of the kinds of operations the object + supports and a summary of key resources of the object (mmio regions + and irqs). + + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a + hardware device that connects to an Ethernet PHY and allows + physical transmission and reception of Ethernet frames. + -mmio regions: none + -irqs: dpni link change + -commands: set link up/down, link config, get stats, + irq config, enable, reset + + -DPNI (Datapath Network Interface): contains TX/RX queues, + network interface configuration, and rx buffer pool configuration + mechanisms. + -mmio regions: none + -irqs: link state + -commands: port config, offload config, queue config, + parse/classify config, irq config, enable, reset + + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue + packets and do hardware buffer pool management operations. For + optimum performance there is typically DPIO per CPU. This allows + each CPU to perform simultaneous enqueue/dequeue operations. + -mmio regions: queue operations, buffer mgmt + -irqs: data availability, congestion notification, buffer + pool depletion + -commands: irq config, enable, reset + + -DPBP (Datapath Buffer Pool): represents a hardware buffer + pool. + -mmio regions: none + -irqs: none + -commands: enable, reset + + -DPMCP (Datapath MC Portal): provides an MC command portal. + Used by drivers to send commands to the MC to manage + objects. + -mmio regions: MC command portal + -irqs: command completion + -commands: irq config, enable, reset + + Object Connections + ------------------ + Some objects have explicit relationships that must + be configured: + + -DPNI <--> DPMAC + -DPNI <--> DPNI + -DPNI <--> L2-switch-port + A DPNI must be connected to something such as a DPMAC, + another DPNI, or L2 switch port. The DPNI connection + is made via a DPRC command. + + +-------+ +-------+ + | DPNI | | DPMAC | + +---+---+ +---+---+ + | | + +==========+ + + -DPNI <--> DPBP + A network interface requires a 'buffer pool' (DPBP + object) which provides a list of pointers to memory + where received Ethernet data is to be copied. The + Ethernet driver configures the DPBPs associated with + the network interface. + + Interrupts + ---------- + All interrupts generated by DPAA2 objects are message + interrupts. At the hardware level message interrupts + generated by devices will normally have 3 components-- + 1) a non-spoofable 'device-id' expressed on the hardware + bus, 2) an address, 3) a data value. + + In the case of DPAA2 devices/objects, all objects in the + same container/DPRC share the same 'device-id'. + For ARM-based SoC this is the same as the stream ID. + + +DPAA2 Linux Driver Overview +--------------------------- + +This section provides an overview of the Linux kernel drivers for +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" +drivers and 2) functional object drivers (such as Ethernet). + +As described previously, a DPRC is a container that holds the other +types of DPAA2 objects. It is functionally similar to a plug-and-play +bus controller. + +Each object in the DPRC is a Linux "device" and is bound to a driver. +The diagram below shows the Linux drivers involved in a networking +scenario and the objects bound to each driver. A brief description +of each driver follows. + + +------------+ + | OS Network | + | Stack | + +------------+ +------------+ + | Allocator |. . . . . . . | Ethernet | + |(dpmcp,dpbp)| | (dpni) | + +-.----------+ +---+---+----+ + . . ^ | + . . <data avail, | |<enqueue, + . . tx confirm> | | dequeue> + +-------------+ . | | + | DPRC driver | . +---+---V----+ +---------+ + | (dprc) | . . . . . .| DPIO driver| | MAC | + +----------+--+ | (dpio) | | (dpmac) | + | +------+-----+ +-----+---+ + |<dev add/remove> | | + | | | + +----+--------------+ | +--+---+ + | mc-bus driver | | | PHY | + | | | |driver| + | /fsl-mc@80c000000 | | +--+---+ + +-------------------+ | | + | | + ================================ HARDWARE =========|=================|====== + DPIO | + | | + DPNI---DPBP | + | | + DPMAC | + | | + PHY ---------------+ + ===================================================|======================== + +A brief description of each driver is provided below. + + mc-bus driver + ------------- + The mc-bus driver is a platform driver and is probed from an + "/fsl-mc@xxxx" node in the device tree passed in by boot firmware. + It is responsible for bootstrapping the DPAA2 kernel infrastructure. + Key functions include: + -registering a new bus type named "fsl-mc" with the kernel, + and implementing bus call-backs (e.g. match/uevent/dev_groups) + -implemeting APIs for DPAA2 driver registration and for device + add/remove + -creates an MSI irq domain + -do a device add of the 'root' DPRC device, which is needed + to bootstrap things + + DPRC driver + ----------- + The dprc-driver is bound DPRC objects and does runtime management + of a bus instance. It performs the initial bus scan of the DPRC + and handles interrupts for container events such as hot plug. + + Allocator + ---------- + Certain objects such as DPMCP and DPBP are generic and fungible, + and are intended to be used by other drivers. For example, + the DPAA2 Ethernet driver needs: + -DPMCPs to send MC commands, to configure network interfaces + -DPBPs for network buffer pools + + The allocator driver registers for these allocatable object types + and those objects are bound to the allocator when the bus is probed. + The allocator maintains a pool of objects that are available for + allocation by other DPAA2 drivers. + + DPIO driver + ----------- + The DPIO driver is bound to DPIO objects and provides services that allow + other drivers such as the Ethernet driver to receive and transmit data. + Key services include: + -data availability notifications + -hardware queuing operations (enqueue and dequeue of data) + -hardware buffer pool management + + There is typically one DPIO object per physical CPU for optimum + performance, allowing each CPU to simultaneously enqueue + and dequeue data. + + The DPIO driver operates on behalf of all DPAA2 drivers + active in the kernel-- Ethernet, crypto, compression, + etc. + + Ethernet + -------- + The Ethernet driver is bound to a DPNI and implements the kernel + interfaces needed to connect the DPAA2 network interface to + the network stack. + + Each DPNI corresponds to a Linux network interface. + + MAC driver + ---------- + An Ethernet PHY is an off-chip, board specific component and is managed + by the appropriate PHY driver via an mdio bus. The MAC driver + plays a role of being a proxy between the PHY driver and the + MC. It does this proxy via the MC commands to a DPMAC object.
drivers/staging/fsl-mc/TODO +0 -4
@@ -1,7 +1,3 @@ -* Add README file (with ASCII art) describing relationships between - DPAA2 objects and how combine them to make a NIC, an LS2 switch, etc. - Also, define all acronyms used. - * Decide if multiple root fsl-mc buses will be supported per Linux instance, and if so add support for this.
Signature · 1 line
2.3.3
Quoted 141 lines · Stuart Yoder
add README file providing an overview of the DPAA2 architecture and how it is integrated in Linux
Signed-off-by Stuart Yoder <[email protected]>---
(quoted diff) +0
-1-v2: added changelog text drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++ drivers/staging/fsl-mc/TODO | 4 - 2 files changed, 364 insertions(+), 4 deletions(-) create mode 100644 drivers/staging/fsl-mc/README.txtdrivers/staging/fsl-mc/README.txt new file +123
-0@@ -0,0 +1,364 @@ +Copyright (C) 2015 Freescale Semiconductor Inc. + +DPAA2 (Data Path Acceleration Architecture Gen2) +------------------------------------------------ + +This document provides an overview of the Freescale DPAA2 architecture +and how it is integrated into the Linux kernel. + +Contents summary + -DPAA2 overview + -Overview of DPAA2 objects + -DPAA2 Linux driver architecture overview + -bus driver + -dprc driver + -allocator + -dpio driver + -Ethernet + -mac + +DPAA2 Overview +-------------- + +DPAA2 is a hardware architecture designed for high-speeed network +packet processing. DPAA2 consists of sophisticated mechanisms for +processing Ethernet packets, queue management, buffer management, +autonomous L2 switching, virtual Ethernet bridging, and accelerator +(e.g. crypto) sharing. + +A DPAA2 hardware component called the Management Complex (or MC) manages the +DPAA2 hardware resources. The MC provides an object-based abstraction for +software drivers to use the DPAA2 hardware. + +The MC uses DPAA2 hardware resources such as queues, buffer pools, and +network ports to create functional objects/devices such as network +interfaces, an L2 switch, or accelerator instances. + +The MC provides memory-mapped I/O command interfaces (MC portals) +which DPAA2 software drivers use to operate on DPAA2 objects: + + +--------------------------------------+ + | OS | + | DPAA2 drivers | + | | | + +-----------------------------|--------+ + | + | (create,discover,connect + | config,use,destroy) + | + DPAA2 | + +------------------------| mc portal |-+ + | | | + | +- - - - - - - - - - - - -V- - -+ | + | | | | + | | Management Complex (MC) | | + | | | | + | +- - - - - - - - - - - - - - - -+ | + | | + | Hardware Hardware | + | Resources Objects | + | --------- ------- | + | -queues -DPRC | + | -buffer pools -DPMCP | + | -Eth MACs/ports -DPIO | + | -network interface -DPNI | + | profiles -DPMAC | + | -queue portals -DPBP | + | -MC portals ... | + | ... | + | | + +--------------------------------------+ + +The MC mediates operations such as create, discover, +connect, configuration, and destroy. Fast-path operations +on data, such as packet transmit/receive, are not mediated by +the MC and are done directly using memory mapped regions in +DPIO objects. + +Overview of DPAA2 Objects +------------------------- +The section provides a brief overview of some key objects +in the DPAA2 hardware. A simple scenario is described illustrating +the objects involved in creating a network interfaces. + +-DPRC (Datapath Resource Container) + + A DPRC is an container object that holds all the other + types of DPAA2 objects. In the example diagram below there + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) + in the container. + + +---------------------------------------------------------+ + | DPRC | + | | + | +-------+ +-------+ +-------+ +-------+ +-------+ | + | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | + | +-------+ +-------+ +-------+ +---+---+ +---+---+ | + | | DPMCP | | DPIO | | + | +-------+ +-------+ | + | | DPMCP | | + | +-------+ | + | | + +---------------------------------------------------------+ + + From the point of view of an OS, a DPRC is bus-like. Like + a plug-and-play bus, such as PCI, DPRC commands can be used to + enumerate the contents of the DPRC, discover the hardware + objects present (including mappable regions and interrupts). + + dprc.1 (bus) + | + +--+--------+-------+-------+-------+ + | | | | | + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 + dpmcp.2 dpio.2 + dpmcp.3 + + Hardware objects can be created and destroyed dynamically, providing + the ability to hot plug/unplug objects in and out of the DPRC. + + A DPRC has a mappable mmio region (an MC portal) that can be used + to send MC commands. It has an interrupt for status events (like + hotplug). +
drivers/staging/fsl-mc/README.txt line 133
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ All objects in a container share the same hardware "isolation context". + This means that with respect to an IOMMU the isolation granularity + is at the DPRC (container) level, not at the individual object + level. + + DPRCs can be defined statically and populated with objects + via a config file passed to the MC when firmware starts + it. There is also a Linux user space tool called "restool" + that can be used to create/destroy containers and objects + dynamically.
Is this tool publicly available yet? Also, I find the naming unfortunate for a tool that potentially will get included in general purpose distributions. Naming it "dpaa2-restool" for example would make it much clearer what its purpose is and would give you a nice namespace to add more tools to later.
Quoted 48 lines
drivers/staging/fsl-mc/README.txt new file +48
-0+ +-DPAA2 Objects for an Ethernet Network Interface + + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX + queuing mechanisms, configuration mechanisms, buffer management, + physical ports, and interrupts. DPAA2 uses a more granular approach + utilizing multiple hardware objects. Each object has specialized + functions, and are used together by software to provide Ethernet network + interface functionality. This approach provides efficient use of finite + hardware resources, flexibility, and performance advantages. + + The diagram below shows the objects needed for a simple + network interface configuration on a system with 2 CPUs. + + +---+---+ +---+---+ + CPU0 CPU1 + +---+---+ +---+---+ + | | + +---+---+ +---+---+ + DPIO DPIO + +---+---+ +---+---+ + \ / + \ / + \ / + +---+---+ + DPNI --- DPBP,DPMCP + +---+---+ + | + | + +---+---+ + DPMAC + +---+---+ + | + port/PHY + + Below the objects are described. For each object a brief description + is provided along with a summary of the kinds of operations the object + supports and a summary of key resources of the object (mmio regions + and irqs). + + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a + hardware device that connects to an Ethernet PHY and allows + physical transmission and reception of Ethernet frames. + -mmio regions: none + -irqs: dpni link change + -commands: set link up/down, link config, get stats, + irq config, enable, reset +
drivers/staging/fsl-mc/README.txt line 191
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ -DPNI (Datapath Network Interface): contains TX/RX queues, + network interface configuration, and rx buffer pool configuration + mechanisms. + -mmio regions: none + -irqs: link state + -commands: port config, offload config, queue config, + parse/classify config, irq config, enable, reset + + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue + packets and do hardware buffer pool management operations. For
I think you may want to explain the difference between TX/RX queues and "interfaces to enqueue and dequeue packets" ;).
drivers/staging/fsl-mc/README.txt line 192
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ optimum performance there is typically DPIO per CPU. This allows
typically [one] DPIO?
Quoted 102 lines
drivers/staging/fsl-mc/README.txt new file +102
-0+ each CPU to perform simultaneous enqueue/dequeue operations. + -mmio regions: queue operations, buffer mgmt + -irqs: data availability, congestion notification, buffer + pool depletion + -commands: irq config, enable, reset + + -DPBP (Datapath Buffer Pool): represents a hardware buffer + pool. + -mmio regions: none + -irqs: none + -commands: enable, reset + + -DPMCP (Datapath MC Portal): provides an MC command portal. + Used by drivers to send commands to the MC to manage + objects. + -mmio regions: MC command portal + -irqs: command completion + -commands: irq config, enable, reset + + Object Connections + ------------------ + Some objects have explicit relationships that must + be configured: + + -DPNI <--> DPMAC + -DPNI <--> DPNI + -DPNI <--> L2-switch-port + A DPNI must be connected to something such as a DPMAC, + another DPNI, or L2 switch port. The DPNI connection + is made via a DPRC command. + + +-------+ +-------+ + | DPNI | | DPMAC | + +---+---+ +---+---+ + | | + +==========+ + + -DPNI <--> DPBP + A network interface requires a 'buffer pool' (DPBP + object) which provides a list of pointers to memory + where received Ethernet data is to be copied. The + Ethernet driver configures the DPBPs associated with + the network interface. + + Interrupts + ---------- + All interrupts generated by DPAA2 objects are message + interrupts. At the hardware level message interrupts + generated by devices will normally have 3 components-- + 1) a non-spoofable 'device-id' expressed on the hardware + bus, 2) an address, 3) a data value. + + In the case of DPAA2 devices/objects, all objects in the + same container/DPRC share the same 'device-id'. + For ARM-based SoC this is the same as the stream ID. + + +DPAA2 Linux Driver Overview +--------------------------- + +This section provides an overview of the Linux kernel drivers for +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" +drivers and 2) functional object drivers (such as Ethernet). + +As described previously, a DPRC is a container that holds the other +types of DPAA2 objects. It is functionally similar to a plug-and-play +bus controller. + +Each object in the DPRC is a Linux "device" and is bound to a driver. +The diagram below shows the Linux drivers involved in a networking +scenario and the objects bound to each driver. A brief description +of each driver follows. + + +------------+ + | OS Network | + | Stack | + +------------+ +------------+ + | Allocator |. . . . . . . | Ethernet | + |(dpmcp,dpbp)| | (dpni) | + +-.----------+ +---+---+----+ + . . ^ | + . . <data avail, | |<enqueue, + . . tx confirm> | | dequeue> + +-------------+ . | | + | DPRC driver | . +---+---V----+ +---------+ + | (dprc) | . . . . . .| DPIO driver| | MAC | + +----------+--+ | (dpio) | | (dpmac) | + | +------+-----+ +-----+---+ + |<dev add/remove> | | + | | | + +----+--------------+ | +--+---+ + | mc-bus driver | | | PHY | + | | | |driver| + | /fsl-mc@80c000000 | | +--+---+ + +-------------------+ | | + | | + ================================ HARDWARE =========|=================|====== + DPIO | + | | + DPNI---DPBP | + | | + DPMAC |
drivers/staging/fsl-mc/README.txt line 304
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ | | + PHY ---------------+ + ===================================================|======================== + +A brief description of each driver is provided below. + + mc-bus driver + ------------- + The mc-bus driver is a platform driver and is probed from an + "/fsl-mc@xxxx" node in the device tree passed in by boot firmware.
Probably better to describe the actual binding here which is based on compatible.
drivers/staging/fsl-mc/README.txt line 309
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ It is responsible for bootstrapping the DPAA2 kernel infrastructure. + Key functions include: + -registering a new bus type named "fsl-mc" with the kernel, + and implementing bus call-backs (e.g. match/uevent/dev_groups) + -implemeting APIs for DPAA2 driver registration and for device
implemeting? ;)
drivers/staging/fsl-mc/README.txt line 313
Quoted 4 lines
drivers/staging/fsl-mc/README.txt new file +4
-0+ add/remove + -creates an MSI irq domain + -do a device add of the 'root' DPRC device, which is needed + to bootstrap things
I think you can find a better way to describe exposure of the root container than "to bootstrap things".
drivers/staging/fsl-mc/README.txt line 317
Quoted 4 lines
drivers/staging/fsl-mc/README.txt new file +4
-0+ + DPRC driver + ----------- + The dprc-driver is bound DPRC objects and does runtime management
bound [to] DPRC
Quoted 22 lines
drivers/staging/fsl-mc/README.txt new file +22
-0+ of a bus instance. It performs the initial bus scan of the DPRC + and handles interrupts for container events such as hot plug. + + Allocator + ---------- + Certain objects such as DPMCP and DPBP are generic and fungible, + and are intended to be used by other drivers. For example, + the DPAA2 Ethernet driver needs: + -DPMCPs to send MC commands, to configure network interfaces + -DPBPs for network buffer pools + + The allocator driver registers for these allocatable object types + and those objects are bound to the allocator when the bus is probed. + The allocator maintains a pool of objects that are available for + allocation by other DPAA2 drivers. + + DPIO driver + ----------- + The DPIO driver is bound to DPIO objects and provides services that allow + other drivers such as the Ethernet driver to receive and transmit data. + Key services include: + -data availability notifications
drivers/staging/fsl-mc/README.txt line 349
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ -hardware queuing operations (enqueue and dequeue of data) + -hardware buffer pool management + + There is typically one DPIO object per physical CPU for optimum + performance, allowing each CPU to simultaneously enqueue + and dequeue data. + + The DPIO driver operates on behalf of all DPAA2 drivers + active in the kernel-- Ethernet, crypto, compression, + etc.
I'm not quite sure what this means? Where do I MMIO into when I want to transmit a packet?
Alex
Quoted 28 lines
drivers/staging/fsl-mc/README.txt new file +15
-0+ + Ethernet + -------- + The Ethernet driver is bound to a DPNI and implements the kernel + interfaces needed to connect the DPAA2 network interface to + the network stack. + + Each DPNI corresponds to a Linux network interface. + + MAC driver + ---------- + An Ethernet PHY is an off-chip, board specific component and is managed + by the appropriate PHY driver via an mdio bus. The MAC driver + plays a role of being a proxy between the PHY driver and the + MC. It does this proxy via the MC commands to a DPMAC object.drivers/staging/fsl-mc/TODO +0
-4@@ -1,7 +1,3 @@ -* Add README file (with ASCII art) describing relationships between - DPAA2 objects and how combine them to make a NIC, an LS2 switch, etc. - Also, define all acronyms used. - * Decide if multiple root fsl-mc buses will be supported per Linux instance, and if so add support for this.
Quoted 157 lines
-----Original Message-----
From: Alexander Graf [mailto:[email protected]]
Sent: Sunday, August 09, 2015 9:25 AM
To: Yoder Stuart-B08248; [email protected]; Rivera Jose-B46482; katz Itai-RM05202
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v2] staging: fsl-mc: add DPAA2 overview readmeQuoted 145 lines · Stuart Yoder
add README file providing an overview of the DPAA2 architecture and how it is integrated in Linux
Signed-off-by Stuart Yoder <[email protected]>---
(quoted diff) +0
-1-v2: added changelog text drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++ drivers/staging/fsl-mc/TODO | 4 - 2 files changed, 364 insertions(+), 4 deletions(-) create mode 100644 drivers/staging/fsl-mc/README.txtdrivers/staging/fsl-mc/README.txt new file +127
-0@@ -0,0 +1,364 @@ +Copyright (C) 2015 Freescale Semiconductor Inc. + +DPAA2 (Data Path Acceleration Architecture Gen2) +------------------------------------------------ + +This document provides an overview of the Freescale DPAA2 architecture +and how it is integrated into the Linux kernel. + +Contents summary + -DPAA2 overview + -Overview of DPAA2 objects + -DPAA2 Linux driver architecture overview + -bus driver + -dprc driver + -allocator + -dpio driver + -Ethernet + -mac + +DPAA2 Overview +-------------- + +DPAA2 is a hardware architecture designed for high-speeed network +packet processing. DPAA2 consists of sophisticated mechanisms for +processing Ethernet packets, queue management, buffer management, +autonomous L2 switching, virtual Ethernet bridging, and accelerator +(e.g. crypto) sharing. + +A DPAA2 hardware component called the Management Complex (or MC) manages the +DPAA2 hardware resources. The MC provides an object-based abstraction for +software drivers to use the DPAA2 hardware. + +The MC uses DPAA2 hardware resources such as queues, buffer pools, and +network ports to create functional objects/devices such as network +interfaces, an L2 switch, or accelerator instances. + +The MC provides memory-mapped I/O command interfaces (MC portals) +which DPAA2 software drivers use to operate on DPAA2 objects: + + +--------------------------------------+ + | OS | + | DPAA2 drivers | + | | | + +-----------------------------|--------+ + | + | (create,discover,connect + | config,use,destroy) + | + DPAA2 | + +------------------------| mc portal |-+ + | | | + | +- - - - - - - - - - - - -V- - -+ | + | | | | + | | Management Complex (MC) | | + | | | | + | +- - - - - - - - - - - - - - - -+ | + | | + | Hardware Hardware | + | Resources Objects | + | --------- ------- | + | -queues -DPRC | + | -buffer pools -DPMCP | + | -Eth MACs/ports -DPIO | + | -network interface -DPNI | + | profiles -DPMAC | + | -queue portals -DPBP | + | -MC portals ... | + | ... | + | | + +--------------------------------------+ + +The MC mediates operations such as create, discover, +connect, configuration, and destroy. Fast-path operations +on data, such as packet transmit/receive, are not mediated by +the MC and are done directly using memory mapped regions in +DPIO objects. + +Overview of DPAA2 Objects +------------------------- +The section provides a brief overview of some key objects +in the DPAA2 hardware. A simple scenario is described illustrating +the objects involved in creating a network interfaces. + +-DPRC (Datapath Resource Container) + + A DPRC is an container object that holds all the other + types of DPAA2 objects. In the example diagram below there + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) + in the container. + + +---------------------------------------------------------+ + | DPRC | + | | + | +-------+ +-------+ +-------+ +-------+ +-------+ | + | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | + | +-------+ +-------+ +-------+ +---+---+ +---+---+ | + | | DPMCP | | DPIO | | + | +-------+ +-------+ | + | | DPMCP | | + | +-------+ | + | | + +---------------------------------------------------------+ + + From the point of view of an OS, a DPRC is bus-like. Like + a plug-and-play bus, such as PCI, DPRC commands can be used to + enumerate the contents of the DPRC, discover the hardware + objects present (including mappable regions and interrupts). + + dprc.1 (bus) + | + +--+--------+-------+-------+-------+ + | | | | | + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 + dpmcp.2 dpio.2 + dpmcp.3 + + Hardware objects can be created and destroyed dynamically, providing + the ability to hot plug/unplug objects in and out of the DPRC. + + A DPRC has a mappable mmio region (an MC portal) that can be used + to send MC commands. It has an interrupt for status events (like + hotplug). + + All objects in a container share the same hardware "isolation context". + This means that with respect to an IOMMU the isolation granularity + is at the DPRC (container) level, not at the individual object + level.
Quoted 8 lines
drivers/staging/fsl-mc/README.txt line 133
Quoted 6 lines
drivers/staging/fsl-mc/README.txt new file +6
-0+ + DPRCs can be defined statically and populated with objects + via a config file passed to the MC when firmware starts + it. There is also a Linux user space tool called "restool" + that can be used to create/destroy containers and objects + dynamically.Is this tool publicly available yet?
Not yet.
Quoted 5 lines
Also, I find the naming unfortunate
for a tool that potentially will get included in general purpose distributions. Naming it "dpaa2-restool" for example would make it much clearer what its purpose is and would give you a nice namespace to add more tools to later.
Probably a good idea.
Quoted 53 lines
Quoted 53 lines
drivers/staging/fsl-mc/README.txt new file +53
-0+ +-DPAA2 Objects for an Ethernet Network Interface + + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX + queuing mechanisms, configuration mechanisms, buffer management, + physical ports, and interrupts. DPAA2 uses a more granular approach + utilizing multiple hardware objects. Each object has specialized + functions, and are used together by software to provide Ethernet network + interface functionality. This approach provides efficient use of finite + hardware resources, flexibility, and performance advantages. + + The diagram below shows the objects needed for a simple + network interface configuration on a system with 2 CPUs. + + +---+---+ +---+---+ + CPU0 CPU1 + +---+---+ +---+---+ + | | + +---+---+ +---+---+ + DPIO DPIO + +---+---+ +---+---+ + \ / + \ / + \ / + +---+---+ + DPNI --- DPBP,DPMCP + +---+---+ + | + | + +---+---+ + DPMAC + +---+---+ + | + port/PHY + + Below the objects are described. For each object a brief description + is provided along with a summary of the kinds of operations the object + supports and a summary of key resources of the object (mmio regions + and irqs). + + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a + hardware device that connects to an Ethernet PHY and allows + physical transmission and reception of Ethernet frames. + -mmio regions: none + -irqs: dpni link change + -commands: set link up/down, link config, get stats, + irq config, enable, reset + + -DPNI (Datapath Network Interface): contains TX/RX queues, + network interface configuration, and rx buffer pool configuration + mechanisms. + -mmio regions: none + -irqs: link state
Quoted 8 lines
drivers/staging/fsl-mc/README.txt line 191
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ -commands: port config, offload config, queue config, + parse/classify config, irq config, enable, reset + + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue + packets and do hardware buffer pool management operations. ForI think you may want to explain the difference between TX/RX queues and "interfaces to enqueue and dequeue packets" ;).
So, the queues are literally just the queues themselves (identified by queue #). They are accessible via the DPIO mmio region. So to enqueue something you write a descriptor to the DPIO mmio region, which includes the target queue #.
So the architecture separates the interface to access the queues from the queues themselves. DPIOs are intended to be shared among all DPAA2 drivers in the kernel that interact with queues.
Since a CPU can only do one thing at a time, you just need 1 DPIO
queuing interface per CPU for optimal performance. But, there are
thousands of queues that could exists in the system.
Will expand the description to clarify this.
Quoted 3 lines
drivers/staging/fsl-mc/README.txt line 192
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ optimum performance there is typically DPIO per CPU. This allowstypically [one] DPIO?
Yes, as mentioned above, one DPIO per CPU is optimal. But, you could have just 1 DPIO period and 8 cpus sharing it.
Quoted 107 lines
Quoted 107 lines
drivers/staging/fsl-mc/README.txt new file +107
-0+ each CPU to perform simultaneous enqueue/dequeue operations. + -mmio regions: queue operations, buffer mgmt + -irqs: data availability, congestion notification, buffer + pool depletion + -commands: irq config, enable, reset + + -DPBP (Datapath Buffer Pool): represents a hardware buffer + pool. + -mmio regions: none + -irqs: none + -commands: enable, reset + + -DPMCP (Datapath MC Portal): provides an MC command portal. + Used by drivers to send commands to the MC to manage + objects. + -mmio regions: MC command portal + -irqs: command completion + -commands: irq config, enable, reset + + Object Connections + ------------------ + Some objects have explicit relationships that must + be configured: + + -DPNI <--> DPMAC + -DPNI <--> DPNI + -DPNI <--> L2-switch-port + A DPNI must be connected to something such as a DPMAC, + another DPNI, or L2 switch port. The DPNI connection + is made via a DPRC command. + + +-------+ +-------+ + | DPNI | | DPMAC | + +---+---+ +---+---+ + | | + +==========+ + + -DPNI <--> DPBP + A network interface requires a 'buffer pool' (DPBP + object) which provides a list of pointers to memory + where received Ethernet data is to be copied. The + Ethernet driver configures the DPBPs associated with + the network interface. + + Interrupts + ---------- + All interrupts generated by DPAA2 objects are message + interrupts. At the hardware level message interrupts + generated by devices will normally have 3 components-- + 1) a non-spoofable 'device-id' expressed on the hardware + bus, 2) an address, 3) a data value. + + In the case of DPAA2 devices/objects, all objects in the + same container/DPRC share the same 'device-id'. + For ARM-based SoC this is the same as the stream ID. + + +DPAA2 Linux Driver Overview +--------------------------- + +This section provides an overview of the Linux kernel drivers for +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" +drivers and 2) functional object drivers (such as Ethernet). + +As described previously, a DPRC is a container that holds the other +types of DPAA2 objects. It is functionally similar to a plug-and-play +bus controller. + +Each object in the DPRC is a Linux "device" and is bound to a driver. +The diagram below shows the Linux drivers involved in a networking +scenario and the objects bound to each driver. A brief description +of each driver follows. + + +------------+ + | OS Network | + | Stack | + +------------+ +------------+ + | Allocator |. . . . . . . | Ethernet | + |(dpmcp,dpbp)| | (dpni) | + +-.----------+ +---+---+----+ + . . ^ | + . . <data avail, | |<enqueue, + . . tx confirm> | | dequeue> + +-------------+ . | | + | DPRC driver | . +---+---V----+ +---------+ + | (dprc) | . . . . . .| DPIO driver| | MAC | + +----------+--+ | (dpio) | | (dpmac) | + | +------+-----+ +-----+---+ + |<dev add/remove> | | + | | | + +----+--------------+ | +--+---+ + | mc-bus driver | | | PHY | + | | | |driver| + | /fsl-mc@80c000000 | | +--+---+ + +-------------------+ | | + | | + ================================ HARDWARE =========|=================|====== + DPIO | + | | + DPNI---DPBP | + | | + DPMAC | + | | + PHY ---------------+ + ===================================================|======================== + +A brief description of each driver is provided below.
Quoted 8 lines
drivers/staging/fsl-mc/README.txt line 304
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ + mc-bus driver + ------------- + The mc-bus driver is a platform driver and is probed from an + "/fsl-mc@xxxx" node in the device tree passed in by boot firmware.Probably better to describe the actual binding here which is based on compatible.
Ok
Quoted 7 lines
drivers/staging/fsl-mc/README.txt line 309
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ It is responsible for bootstrapping the DPAA2 kernel infrastructure. + Key functions include: + -registering a new bus type named "fsl-mc" with the kernel, + and implementing bus call-backs (e.g. match/uevent/dev_groups) + -implemeting APIs for DPAA2 driver registration and for deviceimplemeting? ;)
Thanks...
Quoted 7 lines
drivers/staging/fsl-mc/README.txt line 313
Quoted 4 lines
drivers/staging/fsl-mc/README.txt new file +4
-0+ add/remove + -creates an MSI irq domain + -do a device add of the 'root' DPRC device, which is needed + to bootstrap thingsI think you can find a better way to describe exposure of the root container than "to bootstrap things".
Runtime management of the DPRC/container is by the "DPRC driver" (see below). That driver scans the bus, does device_add() operations, handles hotplug events.
But, how did that DPRC itself driver start? It _is_ a DPAA2 driver as
well... how did the device_add() happen for the root DPRC object? That is the
bootstrapping role of the mc-bus driver. It manufactures the device info needed to add a device to the bus, calls device_add, which in turn causes the DPRC driver to get probed, and things start from there.
After that initial boostrapping, the mc-bus platform driver never does anything again.
Did that help? If so, I'll try to clarify the text. If it still is unclear I'll try to explain more.
Quoted 6 lines
drivers/staging/fsl-mc/README.txt line 317
Quoted 4 lines
drivers/staging/fsl-mc/README.txt new file +4
-0+ + DPRC driver + ----------- + The dprc-driver is bound DPRC objects and does runtime managementbound [to] DPRC
Thanks...
Quoted 27 lines
Quoted 27 lines
drivers/staging/fsl-mc/README.txt new file +27
-0+ of a bus instance. It performs the initial bus scan of the DPRC + and handles interrupts for container events such as hot plug. + + Allocator + ---------- + Certain objects such as DPMCP and DPBP are generic and fungible, + and are intended to be used by other drivers. For example, + the DPAA2 Ethernet driver needs: + -DPMCPs to send MC commands, to configure network interfaces + -DPBPs for network buffer pools + + The allocator driver registers for these allocatable object types + and those objects are bound to the allocator when the bus is probed. + The allocator maintains a pool of objects that are available for + allocation by other DPAA2 drivers. + + DPIO driver + ----------- + The DPIO driver is bound to DPIO objects and provides services that allow + other drivers such as the Ethernet driver to receive and transmit data. + Key services include: + -data availability notifications + -hardware queuing operations (enqueue and dequeue of data) + -hardware buffer pool management + + There is typically one DPIO object per physical CPU for optimum + performance, allowing each CPU to simultaneously enqueue
Quoted 8 lines
drivers/staging/fsl-mc/README.txt line 349
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ and dequeue data. + + The DPIO driver operates on behalf of all DPAA2 drivers + active in the kernel-- Ethernet, crypto, compression, + etc.I'm not quite sure what this means? Where do I MMIO into when I want to transmit a packet?
The MMIO region is in the DPIO, but you (e.g. Ethernet) don't touch it directly. You call a DPIO driver API (lightweight). The DPIO driver provides APIs to put/get data on/from queues for all drivers that access the queuing system-- Ethernet, crypto, compression, pattern matcher, etc.
Thanks,
Stuart
Quoted 247 lines · Stuart Yoder
Quoted 157 lines
-----Original Message-----
From: Alexander Graf [mailto:[email protected]]
Sent: Sunday, August 09, 2015 9:25 AM
To: Yoder Stuart-B08248; [email protected]; Rivera Jose-B46482; katz Itai-RM05202
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v2] staging: fsl-mc: add DPAA2 overview readmeQuoted 145 lines · Stuart Yoder
add README file providing an overview of the DPAA2 architecture and how it is integrated in Linux
Signed-off-by Stuart Yoder <[email protected]>---
(quoted diff) +0
-1-v2: added changelog text drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++ drivers/staging/fsl-mc/TODO | 4 - 2 files changed, 364 insertions(+), 4 deletions(-) create mode 100644 drivers/staging/fsl-mc/README.txtdrivers/staging/fsl-mc/README.txt new file +127
-0@@ -0,0 +1,364 @@ +Copyright (C) 2015 Freescale Semiconductor Inc. + +DPAA2 (Data Path Acceleration Architecture Gen2) +------------------------------------------------ + +This document provides an overview of the Freescale DPAA2 architecture +and how it is integrated into the Linux kernel. + +Contents summary + -DPAA2 overview + -Overview of DPAA2 objects + -DPAA2 Linux driver architecture overview + -bus driver + -dprc driver + -allocator + -dpio driver + -Ethernet + -mac + +DPAA2 Overview +-------------- + +DPAA2 is a hardware architecture designed for high-speeed network +packet processing. DPAA2 consists of sophisticated mechanisms for +processing Ethernet packets, queue management, buffer management, +autonomous L2 switching, virtual Ethernet bridging, and accelerator +(e.g. crypto) sharing. + +A DPAA2 hardware component called the Management Complex (or MC) manages the +DPAA2 hardware resources. The MC provides an object-based abstraction for +software drivers to use the DPAA2 hardware. + +The MC uses DPAA2 hardware resources such as queues, buffer pools, and +network ports to create functional objects/devices such as network +interfaces, an L2 switch, or accelerator instances. + +The MC provides memory-mapped I/O command interfaces (MC portals) +which DPAA2 software drivers use to operate on DPAA2 objects: + + +--------------------------------------+ + | OS | + | DPAA2 drivers | + | | | + +-----------------------------|--------+ + | + | (create,discover,connect + | config,use,destroy) + | + DPAA2 | + +------------------------| mc portal |-+ + | | | + | +- - - - - - - - - - - - -V- - -+ | + | | | | + | | Management Complex (MC) | | + | | | | + | +- - - - - - - - - - - - - - - -+ | + | | + | Hardware Hardware | + | Resources Objects | + | --------- ------- | + | -queues -DPRC | + | -buffer pools -DPMCP | + | -Eth MACs/ports -DPIO | + | -network interface -DPNI | + | profiles -DPMAC | + | -queue portals -DPBP | + | -MC portals ... | + | ... | + | | + +--------------------------------------+ + +The MC mediates operations such as create, discover, +connect, configuration, and destroy. Fast-path operations +on data, such as packet transmit/receive, are not mediated by +the MC and are done directly using memory mapped regions in +DPIO objects. + +Overview of DPAA2 Objects +------------------------- +The section provides a brief overview of some key objects +in the DPAA2 hardware. A simple scenario is described illustrating +the objects involved in creating a network interfaces. + +-DPRC (Datapath Resource Container) + + A DPRC is an container object that holds all the other + types of DPAA2 objects. In the example diagram below there + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) + in the container. + + +---------------------------------------------------------+ + | DPRC | + | | + | +-------+ +-------+ +-------+ +-------+ +-------+ | + | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | + | +-------+ +-------+ +-------+ +---+---+ +---+---+ | + | | DPMCP | | DPIO | | + | +-------+ +-------+ | + | | DPMCP | | + | +-------+ | + | | + +---------------------------------------------------------+ + + From the point of view of an OS, a DPRC is bus-like. Like + a plug-and-play bus, such as PCI, DPRC commands can be used to + enumerate the contents of the DPRC, discover the hardware + objects present (including mappable regions and interrupts). + + dprc.1 (bus) + | + +--+--------+-------+-------+-------+ + | | | | | + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 + dpmcp.2 dpio.2 + dpmcp.3 + + Hardware objects can be created and destroyed dynamically, providing + the ability to hot plug/unplug objects in and out of the DPRC. + + A DPRC has a mappable mmio region (an MC portal) that can be used + to send MC commands. It has an interrupt for status events (like + hotplug). + + All objects in a container share the same hardware "isolation context". + This means that with respect to an IOMMU the isolation granularity + is at the DPRC (container) level, not at the individual object + level.Quoted 8 lines
drivers/staging/fsl-mc/README.txt line 133
Quoted 6 lines
drivers/staging/fsl-mc/README.txt new file +6
-0+ + DPRCs can be defined statically and populated with objects + via a config file passed to the MC when firmware starts + it. There is also a Linux user space tool called "restool" + that can be used to create/destroy containers and objects + dynamically.Is this tool publicly available yet?
Not yet.
Quoted 5 lines
Also, I find the naming unfortunate
for a tool that potentially will get included in general purpose distributions. Naming it "dpaa2-restool" for example would make it much clearer what its purpose is and would give you a nice namespace to add more tools to later.Probably a good idea.
Quoted 53 lines
Quoted 53 lines
drivers/staging/fsl-mc/README.txt new file +53
-0+ +-DPAA2 Objects for an Ethernet Network Interface + + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX + queuing mechanisms, configuration mechanisms, buffer management, + physical ports, and interrupts. DPAA2 uses a more granular approach + utilizing multiple hardware objects. Each object has specialized + functions, and are used together by software to provide Ethernet network + interface functionality. This approach provides efficient use of finite + hardware resources, flexibility, and performance advantages. + + The diagram below shows the objects needed for a simple + network interface configuration on a system with 2 CPUs. + + +---+---+ +---+---+ + CPU0 CPU1 + +---+---+ +---+---+ + | | + +---+---+ +---+---+ + DPIO DPIO + +---+---+ +---+---+ + \ / + \ / + \ / + +---+---+ + DPNI --- DPBP,DPMCP + +---+---+ + | + | + +---+---+ + DPMAC + +---+---+ + | + port/PHY + + Below the objects are described. For each object a brief description + is provided along with a summary of the kinds of operations the object + supports and a summary of key resources of the object (mmio regions + and irqs). + + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a + hardware device that connects to an Ethernet PHY and allows + physical transmission and reception of Ethernet frames. + -mmio regions: none + -irqs: dpni link change + -commands: set link up/down, link config, get stats, + irq config, enable, reset + + -DPNI (Datapath Network Interface): contains TX/RX queues, + network interface configuration, and rx buffer pool configuration + mechanisms. + -mmio regions: none + -irqs: link stateQuoted 8 lines
drivers/staging/fsl-mc/README.txt line 191
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ -commands: port config, offload config, queue config, + parse/classify config, irq config, enable, reset + + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue + packets and do hardware buffer pool management operations. ForI think you may want to explain the difference between TX/RX queues and "interfaces to enqueue and dequeue packets" ;).
So, the queues are literally just the queues themselves (identified by queue #). They are accessible via the DPIO mmio region. So to enqueue something you write a descriptor to the DPIO mmio region, which includes the target queue #.
So the architecture separates the interface to access the queues from the queues themselves. DPIOs are intended to be shared among all DPAA2 drivers in the kernel that interact with queues.
Quoted 5 lines
Since a CPU can only do one thing at a time, you just need 1 DPIO
queuing interface per CPU for optimal performance. But, there are
thousands of queues that could exists in the system.Will expand the description to clarify this.
Awesome :). With that clarification the diagrams make much more sense.
Quoted 7 lines
Quoted 4 lines
drivers/staging/fsl-mc/README.txt line 192
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +1
-0+ optimum performance there is typically DPIO per CPU. This allowstypically [one] DPIO?
Yes, as mentioned above, one DPIO per CPU is optimal. But, you could have just 1 DPIO period and 8 cpus sharing it.
So I did guess right, just wanted to make sure you add the "one" :).
Quoted 147 lines
Quoted 108 lines
Quoted 108 lines
drivers/staging/fsl-mc/README.txt new file +107
-0+ each CPU to perform simultaneous enqueue/dequeue operations. + -mmio regions: queue operations, buffer mgmt + -irqs: data availability, congestion notification, buffer + pool depletion + -commands: irq config, enable, reset + + -DPBP (Datapath Buffer Pool): represents a hardware buffer + pool. + -mmio regions: none + -irqs: none + -commands: enable, reset + + -DPMCP (Datapath MC Portal): provides an MC command portal. + Used by drivers to send commands to the MC to manage + objects. + -mmio regions: MC command portal + -irqs: command completion + -commands: irq config, enable, reset + + Object Connections + ------------------ + Some objects have explicit relationships that must + be configured: + + -DPNI <--> DPMAC + -DPNI <--> DPNI + -DPNI <--> L2-switch-port + A DPNI must be connected to something such as a DPMAC, + another DPNI, or L2 switch port. The DPNI connection + is made via a DPRC command. + + +-------+ +-------+ + | DPNI | | DPMAC | + +---+---+ +---+---+ + | | + +==========+ + + -DPNI <--> DPBP + A network interface requires a 'buffer pool' (DPBP + object) which provides a list of pointers to memory + where received Ethernet data is to be copied. The + Ethernet driver configures the DPBPs associated with + the network interface. + + Interrupts + ---------- + All interrupts generated by DPAA2 objects are message + interrupts. At the hardware level message interrupts + generated by devices will normally have 3 components-- + 1) a non-spoofable 'device-id' expressed on the hardware + bus, 2) an address, 3) a data value. + + In the case of DPAA2 devices/objects, all objects in the + same container/DPRC share the same 'device-id'. + For ARM-based SoC this is the same as the stream ID. + + +DPAA2 Linux Driver Overview +--------------------------- + +This section provides an overview of the Linux kernel drivers for +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" +drivers and 2) functional object drivers (such as Ethernet). + +As described previously, a DPRC is a container that holds the other +types of DPAA2 objects. It is functionally similar to a plug-and-play +bus controller. + +Each object in the DPRC is a Linux "device" and is bound to a driver. +The diagram below shows the Linux drivers involved in a networking +scenario and the objects bound to each driver. A brief description +of each driver follows. + + +------------+ + | OS Network | + | Stack | + +------------+ +------------+ + | Allocator |. . . . . . . | Ethernet | + |(dpmcp,dpbp)| | (dpni) | + +-.----------+ +---+---+----+ + . . ^ | + . . <data avail, | |<enqueue, + . . tx confirm> | | dequeue> + +-------------+ . | | + | DPRC driver | . +---+---V----+ +---------+ + | (dprc) | . . . . . .| DPIO driver| | MAC | + +----------+--+ | (dpio) | | (dpmac) | + | +------+-----+ +-----+---+ + |<dev add/remove> | | + | | | + +----+--------------+ | +--+---+ + | mc-bus driver | | | PHY | + | | | |driver| + | /fsl-mc@80c000000 | | +--+---+ + +-------------------+ | | + | | + ================================ HARDWARE =========|=================|====== + DPIO | + | | + DPNI---DPBP | + | | + DPMAC | + | | + PHY ---------------+ + ===================================================|======================== + +A brief description of each driver is provided below.Quoted 8 lines
drivers/staging/fsl-mc/README.txt line 304
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ + mc-bus driver + ------------- + The mc-bus driver is a platform driver and is probed from an + "/fsl-mc@xxxx" node in the device tree passed in by boot firmware.Probably better to describe the actual binding here which is based on compatible.
Ok
Quoted 7 lines
drivers/staging/fsl-mc/README.txt line 309
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ It is responsible for bootstrapping the DPAA2 kernel infrastructure. + Key functions include: + -registering a new bus type named "fsl-mc" with the kernel, + and implementing bus call-backs (e.g. match/uevent/dev_groups) + -implemeting APIs for DPAA2 driver registration and for deviceimplemeting? ;)
Thanks...
Quoted 7 lines
drivers/staging/fsl-mc/README.txt line 313
Quoted 4 lines
drivers/staging/fsl-mc/README.txt new file +4
-0+ add/remove + -creates an MSI irq domain + -do a device add of the 'root' DPRC device, which is needed + to bootstrap thingsI think you can find a better way to describe exposure of the root container than "to bootstrap things".
Runtime management of the DPRC/container is by the "DPRC driver" (see below). That driver scans the bus, does
device_add()operations, handles hotplug events.But, how did that DPRC itself driver start? It _is_ a DPAA2 driver as
well... how did thedevice_add()happen for the root DPRC object? That is the
bootstrapping role of the mc-bus driver. It manufactures the device info needed to add a device to the bus, callsdevice_add, which in turn causes the DPRC driver to get probed, and things start from there.
Quoted 5 lines
After that initial boostrapping, the mc-bus platform driver never does anything again.
Did that help? If so, I'll try to clarify the text. If it still is unclear I'll try to explain more.
I do understand what the code does, I just found the wording clumsy. Basically in a nutshell, the mc-bus driver exposes the root DPRC.
Quoted 46 lines
Quoted 7 lines
drivers/staging/fsl-mc/README.txt line 317
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +4
-0+ + DPRC driver + ----------- + The dprc-driver is bound DPRC objects and does runtime managementbound [to] DPRC
Thanks...
Quoted 27 lines
Quoted 27 lines
drivers/staging/fsl-mc/README.txt new file +27
-0+ of a bus instance. It performs the initial bus scan of the DPRC + and handles interrupts for container events such as hot plug. + + Allocator + ---------- + Certain objects such as DPMCP and DPBP are generic and fungible, + and are intended to be used by other drivers. For example, + the DPAA2 Ethernet driver needs: + -DPMCPs to send MC commands, to configure network interfaces + -DPBPs for network buffer pools + + The allocator driver registers for these allocatable object types + and those objects are bound to the allocator when the bus is probed. + The allocator maintains a pool of objects that are available for + allocation by other DPAA2 drivers. + + DPIO driver + ----------- + The DPIO driver is bound to DPIO objects and provides services that allow + other drivers such as the Ethernet driver to receive and transmit data. + Key services include: + -data availability notifications + -hardware queuing operations (enqueue and dequeue of data) + -hardware buffer pool management + + There is typically one DPIO object per physical CPU for optimum + performance, allowing each CPU to simultaneously enqueueQuoted 8 lines
drivers/staging/fsl-mc/README.txt line 349
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ and dequeue data. + + The DPIO driver operates on behalf of all DPAA2 drivers + active in the kernel-- Ethernet, crypto, compression, + etc.I'm not quite sure what this means? Where do I MMIO into when I want to transmit a packet?
Quoted 4 lines
The MMIO region is in the DPIO, but you (e.g. Ethernet) don't touch it directly. You call a DPIO driver API (lightweight). The DPIO driver provides APIs to put/get data on/from queues for all drivers that access the queuing system-- Ethernet, crypto, compression, pattern matcher, etc.
That's an interesting design, thanks for the clarification.
Alex
Hi Stuart,
I am by no means a native speaker, but I have proof-read my fair share of articles and theses, so here are a bunch of suggestions which might or might not help improve you document.
Quoted 22 lines · Stuart Yoder
add README file providing an overview of the DPAA2 architecture and how it is integrated in Linux
Signed-off-by Stuart Yoder <[email protected]>---
(quoted diff) +0
-1-v2: added changelog text drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++ drivers/staging/fsl-mc/TODO | 4 - 2 files changed, 364 insertions(+), 4 deletions(-) create mode 100644 drivers/staging/fsl-mc/README.txtdrivers/staging/fsl-mc/README.txt new file +4
-0@@ -0,0 +1,364 @@ +Copyright (C) 2015 Freescale Semiconductor Inc. + +DPAA2 (Data Path Acceleration Architecture Gen2) +------------------------------------------------
drivers/staging/fsl-mc/README.txt line 14
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ +This document provides an overview of the Freescale DPAA2 architecture +and how it is integrated into the Linux kernel. + +Contents summary + -DPAA2 overview + -Overview of DPAA2 objects + -DPAA2 Linux driver architecture overview + -bus driver + -dprc driver
- DPRC driver
drivers/staging/fsl-mc/README.txt line 16
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -allocator + -dpio driver
- DPIO driver
drivers/staging/fsl-mc/README.txt line 18
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -Ethernet + -mac
mac -> MAC
Quoted 58 lines
drivers/staging/fsl-mc/README.txt new file +58
-0+ +DPAA2 Overview +-------------- + +DPAA2 is a hardware architecture designed for high-speeed network +packet processing. DPAA2 consists of sophisticated mechanisms for +processing Ethernet packets, queue management, buffer management, +autonomous L2 switching, virtual Ethernet bridging, and accelerator +(e.g. crypto) sharing. + +A DPAA2 hardware component called the Management Complex (or MC) manages the +DPAA2 hardware resources. The MC provides an object-based abstraction for +software drivers to use the DPAA2 hardware. + +The MC uses DPAA2 hardware resources such as queues, buffer pools, and +network ports to create functional objects/devices such as network +interfaces, an L2 switch, or accelerator instances. + +The MC provides memory-mapped I/O command interfaces (MC portals) +which DPAA2 software drivers use to operate on DPAA2 objects: + + +--------------------------------------+ + | OS | + | DPAA2 drivers | + | | | + +-----------------------------|--------+ + | + | (create,discover,connect + | config,use,destroy) + | + DPAA2 | + +------------------------| mc portal |-+ + | | | + | +- - - - - - - - - - - - -V- - -+ | + | | | | + | | Management Complex (MC) | | + | | | | + | +- - - - - - - - - - - - - - - -+ | + | | + | Hardware Hardware | + | Resources Objects | + | --------- ------- | + | -queues -DPRC | + | -buffer pools -DPMCP | + | -Eth MACs/ports -DPIO | + | -network interface -DPNI | + | profiles -DPMAC | + | -queue portals -DPBP | + | -MC portals ... | + | ... | + | | + +--------------------------------------+ + +The MC mediates operations such as create, discover, +connect, configuration, and destroy. Fast-path operations +on data, such as packet transmit/receive, are not mediated by +the MC and are done directly using memory mapped regions in +DPIO objects.
drivers/staging/fsl-mc/README.txt line 86
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ +Overview of DPAA2 Objects +------------------------- +The section provides a brief overview of some key objects +in the DPAA2 hardware. A simple scenario is described illustrating +the objects involved in creating a network interfaces. + +-DPRC (Datapath Resource Container) + + A DPRC is an container object that holds all the other
... is a container ...
Quoted 8 lines
drivers/staging/fsl-mc/README.txt new file +8
-0+ types of DPAA2 objects. In the example diagram below there + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) + in the container. + + +---------------------------------------------------------+ + | DPRC | + | | + | +-------+ +-------+ +-------+ +-------+ +-------+ |
drivers/staging/fsl-mc/README.txt line 104
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | + | +-------+ +-------+ +-------+ +---+---+ +---+---+ | + | | DPMCP | | DPIO | | + | +-------+ +-------+ | + | | DPMCP | | + | +-------+ | + | | + +---------------------------------------------------------+ + + From the point of view of an OS, a DPRC is bus-like. Like
maybe replace "is bus-like" with "behaves similar to a bus" or "can be viewed as a bus"
drivers/staging/fsl-mc/README.txt line 105
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ a plug-and-play bus, such as PCI, DPRC commands can be used to
maybe replace "Like a pnp bus..." with "Similar to a plug-and-play bus, such as PCI, DPRC ..."
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ enumerate the contents of the DPRC, discover the hardware + objects present (including mappable regions and interrupts). + + dprc.1 (bus) + |
drivers/staging/fsl-mc/README.txt line 120
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ +--+--------+-------+-------+-------+ + | | | | | + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 + dpmcp.2 dpio.2 + dpmcp.3 + + Hardware objects can be created and destroyed dynamically, providing + the ability to hot plug/unplug objects in and out of the DPRC. + + A DPRC has a mappable mmio region (an MC portal) that can be used
- mmio -> MMIO (not sure whether mappable MMIO is redundant or not)
- a MC portal
Quoted 11 lines
drivers/staging/fsl-mc/README.txt new file +11
-0+ to send MC commands. It has an interrupt for status events (like + hotplug). + + All objects in a container share the same hardware "isolation context". + This means that with respect to an IOMMU the isolation granularity + is at the DPRC (container) level, not at the individual object + level. + + DPRCs can be defined statically and populated with objects + via a config file passed to the MC when firmware starts + it. There is also a Linux user space tool called "restool"
drivers/staging/fsl-mc/README.txt line 141
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ that can be used to create/destroy containers and objects + dynamically. + +-DPAA2 Objects for an Ethernet Network Interface + + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX + queuing mechanisms, configuration mechanisms, buffer management, + physical ports, and interrupts. DPAA2 uses a more granular approach + utilizing multiple hardware objects. Each object has specialized + functions, and are used together by software to provide Ethernet network
Each object provides specialized functions. Groups of these objects are used by the software to provide Ethernet network interface functionality.
Quoted 18 lines
drivers/staging/fsl-mc/README.txt new file +18
-0+ interface functionality. This approach provides efficient use of finite + hardware resources, flexibility, and performance advantages. + + The diagram below shows the objects needed for a simple + network interface configuration on a system with 2 CPUs. + + +---+---+ +---+---+ + CPU0 CPU1 + +---+---+ +---+---+ + | | + +---+---+ +---+---+ + DPIO DPIO + +---+---+ +---+---+ + \ / + \ / + \ / + +---+---+ + DPNI --- DPBP,DPMCP
drivers/staging/fsl-mc/README.txt line 169
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ +---+---+ + | + | + +---+---+ + DPMAC + +---+---+ + | + port/PHY + + Below the objects are described. For each object a brief description
The objects depicted in this figure are described below.
drivers/staging/fsl-mc/README.txt line 172
Quoted 3 lines
drivers/staging/fsl-mc/README.txt new file +3
-0+ is provided along with a summary of the kinds of operations the object + supports and a summary of key resources of the object (mmio regions + and irqs).
mmio -> MMIO
irqs -> IRQs
drivers/staging/fsl-mc/README.txt line 177
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a + hardware device that connects to an Ethernet PHY and allows + physical transmission and reception of Ethernet frames. + -mmio regions: none
mmio -> MMIO
drivers/staging/fsl-mc/README.txt line 178
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ -irqs: dpni link change
irqs -> IRQs
dpni -> DPNI
drivers/staging/fsl-mc/README.txt line 180
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -commands: set link up/down, link config, get stats, + irq config, enable, reset
irq -> IRQ
drivers/staging/fsl-mc/README.txt line 183
Quoted 3 lines
drivers/staging/fsl-mc/README.txt new file +3
-0+ + -DPNI (Datapath Network Interface): contains TX/RX queues, + network interface configuration, and rx buffer pool configuration
rx -> RX
drivers/staging/fsl-mc/README.txt line 185
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ mechanisms. + -mmio regions: none
mmio -> MMIO
drivers/staging/fsl-mc/README.txt line 186
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ -irqs: link state
irqs -> IRQs
drivers/staging/fsl-mc/README.txt line 188
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -commands: port config, offload config, queue config, + parse/classify config, irq config, enable, reset
irq -> IRQ
drivers/staging/fsl-mc/README.txt line 192
Quoted 4 lines
drivers/staging/fsl-mc/README.txt new file +4
-0+ + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue + packets and do hardware buffer pool management operations. For + optimum performance there is typically DPIO per CPU. This allows
For optimum performance there is typically one DPIO assigned to each CPU
drivers/staging/fsl-mc/README.txt line 193
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ each CPU to perform simultaneous enqueue/dequeue operations.
This allows different CPUs to simultaneously perform enqueue/dequeue operations.
drivers/staging/fsl-mc/README.txt line 194
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ -mmio regions: queue operations, buffer mgmt
mmio -> MMIO
mgmt -> management
drivers/staging/fsl-mc/README.txt line 196
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -irqs: data availability, congestion notification, buffer + pool depletion
irqs -> IRQs
drivers/staging/fsl-mc/README.txt line 197
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ -commands: irq config, enable, reset
irq -> IRQ
drivers/staging/fsl-mc/README.txt line 203
Quoted 6 lines
drivers/staging/fsl-mc/README.txt new file +6
-0+ + -DPBP (Datapath Buffer Pool): represents a hardware buffer + pool. + -mmio regions: none + -irqs: none + -commands: enable, reset
mmio -> MMIO
irqs -> IRQs
drivers/staging/fsl-mc/README.txt line 210
Quoted 7 lines
drivers/staging/fsl-mc/README.txt new file +7
-0+ + -DPMCP (Datapath MC Portal): provides an MC command portal. + Used by drivers to send commands to the MC to manage + objects. + -mmio regions: MC command portal + -irqs: command completion + -commands: irq config, enable, reset
mmio -> MMIO
irqs -> IRQs
irq -> IRQ
Quoted 82 lines
drivers/staging/fsl-mc/README.txt new file +82
-0+ + Object Connections + ------------------ + Some objects have explicit relationships that must + be configured: + + -DPNI <--> DPMAC + -DPNI <--> DPNI + -DPNI <--> L2-switch-port + A DPNI must be connected to something such as a DPMAC, + another DPNI, or L2 switch port. The DPNI connection + is made via a DPRC command. + + +-------+ +-------+ + | DPNI | | DPMAC | + +---+---+ +---+---+ + | | + +==========+ + + -DPNI <--> DPBP + A network interface requires a 'buffer pool' (DPBP + object) which provides a list of pointers to memory + where received Ethernet data is to be copied. The + Ethernet driver configures the DPBPs associated with + the network interface. + + Interrupts + ---------- + All interrupts generated by DPAA2 objects are message + interrupts. At the hardware level message interrupts + generated by devices will normally have 3 components-- + 1) a non-spoofable 'device-id' expressed on the hardware + bus, 2) an address, 3) a data value. + + In the case of DPAA2 devices/objects, all objects in the + same container/DPRC share the same 'device-id'. + For ARM-based SoC this is the same as the stream ID. + + +DPAA2 Linux Driver Overview +--------------------------- + +This section provides an overview of the Linux kernel drivers for +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" +drivers and 2) functional object drivers (such as Ethernet). + +As described previously, a DPRC is a container that holds the other +types of DPAA2 objects. It is functionally similar to a plug-and-play +bus controller. + +Each object in the DPRC is a Linux "device" and is bound to a driver. +The diagram below shows the Linux drivers involved in a networking +scenario and the objects bound to each driver. A brief description +of each driver follows. + + +------------+ + | OS Network | + | Stack | + +------------+ +------------+ + | Allocator |. . . . . . . | Ethernet | + |(dpmcp,dpbp)| | (dpni) | + +-.----------+ +---+---+----+ + . . ^ | + . . <data avail, | |<enqueue, + . . tx confirm> | | dequeue> + +-------------+ . | | + | DPRC driver | . +---+---V----+ +---------+ + | (dprc) | . . . . . .| DPIO driver| | MAC | + +----------+--+ | (dpio) | | (dpmac) | + | +------+-----+ +-----+---+ + |<dev add/remove> | | + | | | + +----+--------------+ | +--+---+ + | mc-bus driver | | | PHY | + | | | |driver| + | /fsl-mc@80c000000 | | +--+---+ + +-------------------+ | | + | | + ================================ HARDWARE =========|=================|====== + DPIO | + | | + DPNI---DPBP |
drivers/staging/fsl-mc/README.txt line 302
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ | | + DPMAC | + | | + PHY ---------------+ + ===================================================|======================== + +A brief description of each driver is provided below. + + mc-bus driver + -------------
mc-bus -> MC-Bus or MC-bus
drivers/staging/fsl-mc/README.txt line 311
Quoted 9 lines
drivers/staging/fsl-mc/README.txt new file +9
-0+ The mc-bus driver is a platform driver and is probed from an + "/fsl-mc@xxxx" node in the device tree passed in by boot firmware. + It is responsible for bootstrapping the DPAA2 kernel infrastructure. + Key functions include: + -registering a new bus type named "fsl-mc" with the kernel, + and implementing bus call-backs (e.g. match/uevent/dev_groups) + -implemeting APIs for DPAA2 driver registration and for device + add/remove + -creates an MSI irq domain
irq -> IRQ
Quoted 24 lines
drivers/staging/fsl-mc/README.txt new file +24
-0+ -do a device add of the 'root' DPRC device, which is needed + to bootstrap things + + DPRC driver + ----------- + The dprc-driver is bound DPRC objects and does runtime management + of a bus instance. It performs the initial bus scan of the DPRC + and handles interrupts for container events such as hot plug. + + Allocator + ---------- + Certain objects such as DPMCP and DPBP are generic and fungible, + and are intended to be used by other drivers. For example, + the DPAA2 Ethernet driver needs: + -DPMCPs to send MC commands, to configure network interfaces + -DPBPs for network buffer pools + + The allocator driver registers for these allocatable object types + and those objects are bound to the allocator when the bus is probed. + The allocator maintains a pool of objects that are available for + allocation by other DPAA2 drivers. + + DPIO driver + -----------
drivers/staging/fsl-mc/README.txt line 345
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ The DPIO driver is bound to DPIO objects and provides services that allow + other drivers such as the Ethernet driver to receive and transmit data. + Key services include: + -data availability notifications + -hardware queuing operations (enqueue and dequeue of data) + -hardware buffer pool management + + There is typically one DPIO object per physical CPU for optimum + performance, allowing each CPU to simultaneously enqueue + and dequeue data.
allowing different CPUs to simultaneously ...
Quoted 24 lines
drivers/staging/fsl-mc/README.txt new file +19
-0+ + The DPIO driver operates on behalf of all DPAA2 drivers + active in the kernel-- Ethernet, crypto, compression, + etc. + + Ethernet + -------- + The Ethernet driver is bound to a DPNI and implements the kernel + interfaces needed to connect the DPAA2 network interface to + the network stack. + + Each DPNI corresponds to a Linux network interface. + + MAC driver + ---------- + An Ethernet PHY is an off-chip, board specific component and is managed + by the appropriate PHY driver via an mdio bus. The MAC driver + plays a role of being a proxy between the PHY driver and the + MC. It does this proxy via the MC commands to a DPMAC object.drivers/staging/fsl-mc/TODO +0
-0
drivers/staging/fsl-mc/TODO line 3
Quoted 7 lines
drivers/staging/fsl-mc/TODO +0
-4@@ -1,7 +1,3 @@ -* Add README file (with ASCII art) describing relationships between - DPAA2 objects and how combine them to make a NIC, an LS2 switch, etc. - Also, define all acronyms used. - * Decide if multiple root fsl-mc buses will be supported per Linux instance, and if so add support for this.
Cheers
Tillmann
Quoted 484 lines
-----Original Message-----
From: Tillmann Heidsieck [mailto:[email protected]]
Sent: Tuesday, August 11, 2015 6:01 AM
To: Yoder Stuart-B08248; [email protected]; Rivera Jose-B46482; katz Itai-RM05202
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v2] staging: fsl-mc: add DPAA2 overview readmeHi Stuart,
I am by no means a native speaker, but I have proof-read my fair share of articles and theses, so here are a bunch of suggestions which might or might not help improve you document.
Quoted 22 lines · Stuart Yoder
add README file providing an overview of the DPAA2 architecture and how it is integrated in Linux
Signed-off-by Stuart Yoder <[email protected]>---
(quoted diff) +0
-1-v2: added changelog text drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++ drivers/staging/fsl-mc/TODO | 4 - 2 files changed, 364 insertions(+), 4 deletions(-) create mode 100644 drivers/staging/fsl-mc/README.txtdrivers/staging/fsl-mc/README.txt new file +4
-0@@ -0,0 +1,364 @@ +Copyright (C) 2015 Freescale Semiconductor Inc. + +DPAA2 (Data Path Acceleration Architecture Gen2) +------------------------------------------------drivers/staging/fsl-mc/README.txt line 14
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ +This document provides an overview of the Freescale DPAA2 architecture +and how it is integrated into the Linux kernel. + +Contents summary + -DPAA2 overview + -Overview of DPAA2 objects + -DPAA2 Linux driver architecture overview + -bus driver + -dprc driver- DPRC driver
drivers/staging/fsl-mc/README.txt line 16
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -allocator + -dpio driver- DPIO driver
drivers/staging/fsl-mc/README.txt line 18
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -Ethernet + -macmac -> MAC
Quoted 58 lines
drivers/staging/fsl-mc/README.txt new file +58
-0+ +DPAA2 Overview +-------------- + +DPAA2 is a hardware architecture designed for high-speeed network +packet processing. DPAA2 consists of sophisticated mechanisms for +processing Ethernet packets, queue management, buffer management, +autonomous L2 switching, virtual Ethernet bridging, and accelerator +(e.g. crypto) sharing. + +A DPAA2 hardware component called the Management Complex (or MC) manages the +DPAA2 hardware resources. The MC provides an object-based abstraction for +software drivers to use the DPAA2 hardware. + +The MC uses DPAA2 hardware resources such as queues, buffer pools, and +network ports to create functional objects/devices such as network +interfaces, an L2 switch, or accelerator instances. + +The MC provides memory-mapped I/O command interfaces (MC portals) +which DPAA2 software drivers use to operate on DPAA2 objects: + + +--------------------------------------+ + | OS | + | DPAA2 drivers | + | | | + +-----------------------------|--------+ + | + | (create,discover,connect + | config,use,destroy) + | + DPAA2 | + +------------------------| mc portal |-+ + | | | + | +- - - - - - - - - - - - -V- - -+ | + | | | | + | | Management Complex (MC) | | + | | | | + | +- - - - - - - - - - - - - - - -+ | + | | + | Hardware Hardware | + | Resources Objects | + | --------- ------- | + | -queues -DPRC | + | -buffer pools -DPMCP | + | -Eth MACs/ports -DPIO | + | -network interface -DPNI | + | profiles -DPMAC | + | -queue portals -DPBP | + | -MC portals ... | + | ... | + | | + +--------------------------------------+ + +The MC mediates operations such as create, discover, +connect, configuration, and destroy. Fast-path operations +on data, such as packet transmit/receive, are not mediated by +the MC and are done directly using memory mapped regions in +DPIO objects.drivers/staging/fsl-mc/README.txt line 86
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ +Overview of DPAA2 Objects +------------------------- +The section provides a brief overview of some key objects +in the DPAA2 hardware. A simple scenario is described illustrating +the objects involved in creating a network interfaces. + +-DPRC (Datapath Resource Container) + + A DPRC is an container object that holds all the other... is a container ...
Quoted 8 lines
drivers/staging/fsl-mc/README.txt new file +8
-0+ types of DPAA2 objects. In the example diagram below there + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) + in the container. + + +---------------------------------------------------------+ + | DPRC | + | | + | +-------+ +-------+ +-------+ +-------+ +-------+ |drivers/staging/fsl-mc/README.txt line 104
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | + | +-------+ +-------+ +-------+ +---+---+ +---+---+ | + | | DPMCP | | DPIO | | + | +-------+ +-------+ | + | | DPMCP | | + | +-------+ | + | | + +---------------------------------------------------------+ + + From the point of view of an OS, a DPRC is bus-like. Likemaybe replace "is bus-like" with "behaves similar to a bus" or "can be viewed as a bus"
drivers/staging/fsl-mc/README.txt line 105
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ a plug-and-play bus, such as PCI, DPRC commands can be used tomaybe replace "Like a pnp bus..." with "Similar to a plug-and-play bus, such as PCI, DPRC ..."
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ enumerate the contents of the DPRC, discover the hardware + objects present (including mappable regions and interrupts). + + dprc.1 (bus) + |drivers/staging/fsl-mc/README.txt line 120
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ +--+--------+-------+-------+-------+ + | | | | | + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 + dpmcp.2 dpio.2 + dpmcp.3 + + Hardware objects can be created and destroyed dynamically, providing + the ability to hot plug/unplug objects in and out of the DPRC. + + A DPRC has a mappable mmio region (an MC portal) that can be used- mmio -> MMIO (not sure whether mappable MMIO is redundant or not)
- a MC portalQuoted 11 lines
drivers/staging/fsl-mc/README.txt new file +11
-0+ to send MC commands. It has an interrupt for status events (like + hotplug). + + All objects in a container share the same hardware "isolation context". + This means that with respect to an IOMMU the isolation granularity + is at the DPRC (container) level, not at the individual object + level. + + DPRCs can be defined statically and populated with objects + via a config file passed to the MC when firmware starts + it. There is also a Linux user space tool called "restool"drivers/staging/fsl-mc/README.txt line 141
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ that can be used to create/destroy containers and objects + dynamically. + +-DPAA2 Objects for an Ethernet Network Interface + + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX + queuing mechanisms, configuration mechanisms, buffer management, + physical ports, and interrupts. DPAA2 uses a more granular approach + utilizing multiple hardware objects. Each object has specialized + functions, and are used together by software to provide Ethernet networkEach object provides specialized functions. Groups of these objects are used by the software to provide Ethernet network interface functionality.
Quoted 18 lines
drivers/staging/fsl-mc/README.txt new file +18
-0+ interface functionality. This approach provides efficient use of finite + hardware resources, flexibility, and performance advantages. + + The diagram below shows the objects needed for a simple + network interface configuration on a system with 2 CPUs. + + +---+---+ +---+---+ + CPU0 CPU1 + +---+---+ +---+---+ + | | + +---+---+ +---+---+ + DPIO DPIO + +---+---+ +---+---+ + \ / + \ / + \ / + +---+---+ + DPNI --- DPBP,DPMCPdrivers/staging/fsl-mc/README.txt line 169
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ +---+---+ + | + | + +---+---+ + DPMAC + +---+---+ + | + port/PHY + + Below the objects are described. For each object a brief descriptionThe objects depicted in this figure are described below.
drivers/staging/fsl-mc/README.txt line 172
Quoted 3 lines
drivers/staging/fsl-mc/README.txt new file +3
-0+ is provided along with a summary of the kinds of operations the object + supports and a summary of key resources of the object (mmio regions + and irqs).mmio -> MMIO
irqs -> IRQsdrivers/staging/fsl-mc/README.txt line 177
Quoted 5 lines
drivers/staging/fsl-mc/README.txt new file +5
-0+ + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a + hardware device that connects to an Ethernet PHY and allows + physical transmission and reception of Ethernet frames. + -mmio regions: nonemmio -> MMIO
drivers/staging/fsl-mc/README.txt line 178
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ -irqs: dpni link changeirqs -> IRQs
dpni -> DPNIdrivers/staging/fsl-mc/README.txt line 180
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -commands: set link up/down, link config, get stats, + irq config, enable, resetirq -> IRQ
drivers/staging/fsl-mc/README.txt line 183
Quoted 3 lines
drivers/staging/fsl-mc/README.txt new file +3
-0+ + -DPNI (Datapath Network Interface): contains TX/RX queues, + network interface configuration, and rx buffer pool configurationrx -> RX
drivers/staging/fsl-mc/README.txt line 185
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ mechanisms. + -mmio regions: nonemmio -> MMIO
drivers/staging/fsl-mc/README.txt line 186
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ -irqs: link stateirqs -> IRQs
drivers/staging/fsl-mc/README.txt line 188
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -commands: port config, offload config, queue config, + parse/classify config, irq config, enable, resetirq -> IRQ
drivers/staging/fsl-mc/README.txt line 192
Quoted 4 lines
drivers/staging/fsl-mc/README.txt new file +4
-0+ + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue + packets and do hardware buffer pool management operations. For + optimum performance there is typically DPIO per CPU. This allowsFor optimum performance there is typically one DPIO assigned to each CPU
drivers/staging/fsl-mc/README.txt line 193
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ each CPU to perform simultaneous enqueue/dequeue operations.This allows different CPUs to simultaneously perform enqueue/dequeue operations.
drivers/staging/fsl-mc/README.txt line 194
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ -mmio regions: queue operations, buffer mgmtmmio -> MMIO
mgmt -> managementdrivers/staging/fsl-mc/README.txt line 196
Quoted 2 lines
drivers/staging/fsl-mc/README.txt new file +2
-0+ -irqs: data availability, congestion notification, buffer + pool depletionirqs -> IRQs
drivers/staging/fsl-mc/README.txt line 197
Quoted 1 line
drivers/staging/fsl-mc/README.txt new file +1
-0+ -commands: irq config, enable, resetirq -> IRQ
drivers/staging/fsl-mc/README.txt line 203
Quoted 6 lines
drivers/staging/fsl-mc/README.txt new file +6
-0+ + -DPBP (Datapath Buffer Pool): represents a hardware buffer + pool. + -mmio regions: none + -irqs: none + -commands: enable, resetmmio -> MMIO
irqs -> IRQsdrivers/staging/fsl-mc/README.txt line 210
Quoted 7 lines
drivers/staging/fsl-mc/README.txt new file +7
-0+ + -DPMCP (Datapath MC Portal): provides an MC command portal. + Used by drivers to send commands to the MC to manage + objects. + -mmio regions: MC command portal + -irqs: command completion + -commands: irq config, enable, resetmmio -> MMIO
irqs -> IRQs
irq -> IRQQuoted 82 lines
drivers/staging/fsl-mc/README.txt new file +82
-0+ + Object Connections + ------------------ + Some objects have explicit relationships that must + be configured: + + -DPNI <--> DPMAC + -DPNI <--> DPNI + -DPNI <--> L2-switch-port + A DPNI must be connected to something such as a DPMAC, + another DPNI, or L2 switch port. The DPNI connection + is made via a DPRC command. + + +-------+ +-------+ + | DPNI | | DPMAC | + +---+---+ +---+---+ + | | + +==========+ + + -DPNI <--> DPBP + A network interface requires a 'buffer pool' (DPBP + object) which provides a list of pointers to memory + where received Ethernet data is to be copied. The + Ethernet driver configures the DPBPs associated with + the network interface. + + Interrupts + ---------- + All interrupts generated by DPAA2 objects are message + interrupts. At the hardware level message interrupts + generated by devices will normally have 3 components-- + 1) a non-spoofable 'device-id' expressed on the hardware + bus, 2) an address, 3) a data value. + + In the case of DPAA2 devices/objects, all objects in the + same container/DPRC share the same 'device-id'. + For ARM-based SoC this is the same as the stream ID. + + +DPAA2 Linux Driver Overview +--------------------------- + +This section provides an overview of the Linux kernel drivers for +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" +drivers and 2) functional object drivers (such as Ethernet). + +As described previously, a DPRC is a container that holds the other +types of DPAA2 objects. It is functionally similar to a plug-and-play +bus controller. + +Each object in the DPRC is a Linux "device" and is bound to a driver. +The diagram below shows the Linux drivers involved in a networking +scenario and the objects bound to each driver. A brief description +of each driver follows. + + +------------+ + | OS Network | + | Stack | + +------------+ +------------+ + | Allocator |. . . . . . . | Ethernet | + |(dpmcp,dpbp)| | (dpni) | + +-.----------+ +---+---+----+ + . . ^ | + . . <data avail, | |<enqueue, + . . tx confirm> | | dequeue> + +-------------+ . | | + | DPRC driver | . +---+---V----+ +---------+ + | (dprc) | . . . . . .| DPIO driver| | MAC | + +----------+--+ | (dpio) | | (dpmac) | + | +------+-----+ +-----+---+ + |<dev add/remove> | | + | | | + +----+--------------+ | +--+---+ + | mc-bus driver | | | PHY | + | | | |driver| + | /fsl-mc@80c000000 | | +--+---+ + +-------------------+ | | + | | + ================================ HARDWARE =========|=================|====== + DPIO | + | | + DPNI---DPBP |drivers/staging/fsl-mc/README.txt line 302
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ | | + DPMAC | + | | + PHY ---------------+ + ===================================================|======================== + +A brief description of each driver is provided below. + + mc-bus driver + -------------mc-bus -> MC-Bus or MC-bus
drivers/staging/fsl-mc/README.txt line 311
Quoted 9 lines
drivers/staging/fsl-mc/README.txt new file +9
-0+ The mc-bus driver is a platform driver and is probed from an + "/fsl-mc@xxxx" node in the device tree passed in by boot firmware. + It is responsible for bootstrapping the DPAA2 kernel infrastructure. + Key functions include: + -registering a new bus type named "fsl-mc" with the kernel, + and implementing bus call-backs (e.g. match/uevent/dev_groups) + -implemeting APIs for DPAA2 driver registration and for device + add/remove + -creates an MSI irq domainirq -> IRQ
Quoted 24 lines
drivers/staging/fsl-mc/README.txt new file +24
-0+ -do a device add of the 'root' DPRC device, which is needed + to bootstrap things + + DPRC driver + ----------- + The dprc-driver is bound DPRC objects and does runtime management + of a bus instance. It performs the initial bus scan of the DPRC + and handles interrupts for container events such as hot plug. + + Allocator + ---------- + Certain objects such as DPMCP and DPBP are generic and fungible, + and are intended to be used by other drivers. For example, + the DPAA2 Ethernet driver needs: + -DPMCPs to send MC commands, to configure network interfaces + -DPBPs for network buffer pools + + The allocator driver registers for these allocatable object types + and those objects are bound to the allocator when the bus is probed. + The allocator maintains a pool of objects that are available for + allocation by other DPAA2 drivers. + + DPIO driver + -----------drivers/staging/fsl-mc/README.txt line 345
Quoted 10 lines
drivers/staging/fsl-mc/README.txt new file +10
-0+ The DPIO driver is bound to DPIO objects and provides services that allow + other drivers such as the Ethernet driver to receive and transmit data. + Key services include: + -data availability notifications + -hardware queuing operations (enqueue and dequeue of data) + -hardware buffer pool management + + There is typically one DPIO object per physical CPU for optimum + performance, allowing each CPU to simultaneously enqueue + and dequeue data.allowing different CPUs to simultaneously ...
Quoted 23 lines
drivers/staging/fsl-mc/README.txt new file +19
-0+ + The DPIO driver operates on behalf of all DPAA2 drivers + active in the kernel-- Ethernet, crypto, compression, + etc. + + Ethernet + -------- + The Ethernet driver is bound to a DPNI and implements the kernel + interfaces needed to connect the DPAA2 network interface to + the network stack. + + Each DPNI corresponds to a Linux network interface. + + MAC driver + ---------- + An Ethernet PHY is an off-chip, board specific component and is managed + by the appropriate PHY driver via an mdio bus. The MAC driver + plays a role of being a proxy between the PHY driver and the + MC. It does this proxy via the MC commands to a DPMAC object.drivers/staging/fsl-mc/TODO +0
-0
Quoted 8 lines
drivers/staging/fsl-mc/TODO line 3
Quoted 8 lines
drivers/staging/fsl-mc/TODO +0
-4@@ -1,7 +1,3 @@ -* Add README file (with ASCII art) describing relationships between - DPAA2 objects and how combine them to make a NIC, an LS2 switch, etc. - Also, define all acronyms used. - * Decide if multiple root fsl-mc buses will be supported per Linux instance, and if so add support for this.
Appreciate the detailed review. Like most of your suggestions and will incorporate on the next update to this document.
Stuart ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�