dlb2: introduce DLB 2.0 device driver
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v3 Latest19 patches 6 repliesIn review
- v219 patches 3 repliesSuperseded
- v120 patches 25 repliesSuperseded
Changes in v3 from the cover letter
- Remove DLB2_PCI_REG_READ/WRITE macros
Patches in v3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/19 | dlb2: add skeleton for DLB 2.0 driverDocumentation/ Documentation/ MAINTAINERS +8 more | +621 | 6 replies | |
| 2/19 | dlb2: initialize PF devicedrivers/ drivers/ drivers/ +2 more | +235 | 0 replies | |
| 3/19 | dlb2: add resource and device initializationdrivers/ drivers/ drivers/ +6 more | +754 | 0 replies | |
| 4/19 | dlb2: add device ioctl layer and first three ioctlsDocumentation/ drivers/ drivers/ +8 more | +428 | 0 replies | |
| 5/19 | dlb2: add sched domain config and reset supportdrivers/ drivers/ drivers/ +7 more | +7252 | 0 replies | |
| 6/19 | dlb2: add runtime power-management supportdrivers/ drivers/ drivers/ | +104 | 0 replies | |
| 7/19 | dlb2: add queue create and queue-depth-get ioctlsdrivers/ drivers/ drivers/ +5 more | +1116 | 0 replies | |
| 8/19 | dlb2: add ioctl to configure ports, query poll modedrivers/ drivers/ drivers/ +5 more | +1400 | 0 replies | |
| 9/19 | dlb2: add port mmap supportdrivers/ drivers/ drivers/ +7 more | +650 | 0 replies | |
| 10/19 | dlb2: add start domain ioctldrivers/ drivers/ drivers/ +3 more | +189 | 0 replies | |
| 11/19 | dlb2: add queue map and unmap ioctlsdrivers/ drivers/ drivers/ +3 more | +1032 | 0 replies | |
| 12/19 | dlb2: add port enable/disable ioctlsdrivers/ drivers/ drivers/ +3 more | +762 | 0 replies | |
| 13/19 | dlb2: add CQ interrupt supportdrivers/ drivers/ drivers/ +8 more | +987 | 0 replies | |
| 14/19 | dlb2: add domain alert supportdrivers/ drivers/ drivers/ +1 more | +297 | 0 replies | |
| 15/19 | dlb2: add sequence-number management ioctlsdrivers/ drivers/ drivers/ +3 more | +308 | 0 replies | |
| 16/19 | dlb2: add cos bandwidth get/set ioctlsdrivers/ drivers/ drivers/ +3 more | +209 | 0 replies | |
| 17/19 | dlb2: add device FLR supportDocumentation/ drivers/ drivers/ +5 more | +424 | 0 replies | |
| 18/19 | dlb2: add basic PF sysfs interfacesDocumentation/ drivers/ drivers/ +1 more | +602 | 0 replies | |
| 19/19 | dlb2: add ingress error handlingDocumentation/ drivers/ drivers/ +3 more | +520 | 0 replies |
Reviews trailers per patch
All 19 patches have a Reviewed-by or Acked-by.
| Patch | Magnus Karlsson | Björn Töpel |
|---|---|---|
| 1/19 | R-b | |
| 2/19 | R-b | |
| 3/19 | R-b | |
| 4/19 | R-b | |
| 5/19 | R-b | |
| 6/19 | R-b | |
| 7/19 | R-b | |
| 8/19 | R-b | |
| 9/19 | R-b | |
| 10/19 | R-b | |
| 11/19 | R-b | |
| 12/19 | R-b | |
| 13/19 | R-b | |
| 14/19 | R-b | |
| 15/19 | R-b | |
| 16/19 | R-b | |
| 17/19 | R-b | |
| 18/19 | R-b | |
| 19/19 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH v3 00/19] dlb2: introduce DLB 2.0 device driver
This commit introduces a new misc device driver for the Intel(r) Dynamic Load Balancer 2.0 (Intel(r) DLB 2.0). The Intel DLB 2.0 is a PCIe device that provides load-balanced, prioritized scheduling of core-to-core communication.
The Intel DLB 2.0 consists of queues and arbiters that connect producer cores and consumer cores. The device implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced distribution.
Intel DLB 2.0 can be used in an event-driven programming model, such as DPDK's Event Device Library[2]. Such frameworks are commonly used in packet processing pipelines that benefit from the framework's multi-core scalability, dynamic load-balancing, and variety of packet distribution and synchronization schemes.
These distribution schemes include "parallel" (packets are load-balanced across multiple cores and processed in parallel), "ordered" (similar to "parallel" but packets are reordered into ingress order by the device), and "atomic" (packet flows are scheduled to a single core at a time such that locks are not required to access per-flow data, and dynamically migrated to ensure load-balance).
The fundamental unit of communication through the device is a queue entry (QE), which consists of 8B of data and 8B of metadata (destination queue, priority, etc.). The data field can be any type that fits within 8B.
A core's interface to the device, a "port," consists of a memory-mappable region through which the core enqueues a queue entry, and an in-memory queue (the "consumer queue") to which the device schedules QEs. Each QE is enqueued to a device-managed queue, and from there scheduled to a port. Software specifies the "linking" of queues and ports; i.e. which ports the device is allowed to schedule to for a given queue. The device uses a credit scheme to prevent overflow of the on-device queue storage.
Applications can interface directly with the device by mapping the port's memory and MMIO regions into the application's address space for enqueue and dequeue operations, but call into the kernel driver for configuration operations. An application can also be polling- or interrupt-driven; Intel DLB 2.0 supports both modes of operation.
Device resources -- i.e. ports, queues, and credits -- are contained within a scheduling domain. Scheduling domains are isolated from one another; a port can only enqueue to and dequeue from queues within its scheduling domain. A scheduling domain's resources are configured through a scheduling domain file, which is acquired through an ioctl.
Intel DLB 2.0 supports SR-IOV and Scalable IOV, and allows for a flexible division of its resources among the PF and its virtual devices. The virtual devices are incapable of configuring the device directly; they use a hardware mailbox to proxy configuration requests to the PF driver. This driver supports both PF and virtual devices, as there is significant code re-use between the two, with device-specific behavior handled through a callback interface. Virtualization support will be added in a later patch set.
The dlb driver uses ioctls as its primary interface (it makes use of sysfs as well, to a lesser extent). The dlb device file supports a different ioctl interface than the scheduling domain file; the dlb device file is used for device-wide operations (including scheduling domain creation), and the scheduling domain file supports operations on the scheduling domain's resources (primarily resource configuration).
[1] https://builders.intel.com/docs/networkbuilders/SKU-343247-001US-queue-management-and-load-balancing-on-intel-architecture.pdf
[2] https://doc.dpdk.org/guides/prog_guide/eventdev.html
v3:
- Remove DLB2_PCI_REG_READ/WRITE macros
v2:
- Change driver license to GPLv2 only
- Expand Kconfig help text and remove unnecessary (R)s
- Remove unnecessary prints
- Add a new entry in ioctl-number.rst
- Convert the ioctl handler into a switch statement
- Correct some instances of IOWR that should have been IOR
- Align macro blocks
- Don't break ioctl ABI when introducing new commands
- Remove indirect pointers from ioctl data structures
- Remove the get-sched-domain-fd ioctl command
Gage Eads (19):
dlb2: add skeleton for DLB 2.0 driver
dlb2: initialize PF device
dlb2: add resource and device initialization
dlb2: add device ioctl layer and first three ioctls
dlb2: add sched domain config and reset support
dlb2: add runtime power-management support
dlb2: add queue create and queue-depth-get ioctls
dlb2: add ioctl to configure ports, query poll mode
dlb2: add port mmap support
dlb2: add start domain ioctl
dlb2: add queue map and unmap ioctls
dlb2: add port enable/disable ioctls
dlb2: add CQ interrupt support
dlb2: add domain alert support
dlb2: add sequence-number management ioctls
dlb2: add cos bandwidth get/set ioctls
dlb2: add device FLR support
dlb2: add basic PF sysfs interfaces
dlb2: add ingress error handling
Documentation/ABI/testing/sysfs-driver-dlb2 | 202 + Documentation/misc-devices/dlb2.rst | 310 + Documentation/misc-devices/index.rst | 1 + Documentation/userspace-api/ioctl/ioctl-number.rst | 1 + MAINTAINERS | 7 + drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/dlb2/Kconfig | 17 + drivers/misc/dlb2/Makefile | 13 + drivers/misc/dlb2/dlb2_bitmap.h | 286 + drivers/misc/dlb2/dlb2_file.c | 133 + drivers/misc/dlb2/dlb2_file.h | 19 + drivers/misc/dlb2/dlb2_hw_types.h | 353 + drivers/misc/dlb2/dlb2_intr.c | 137 + drivers/misc/dlb2/dlb2_intr.h | 30 + drivers/misc/dlb2/dlb2_ioctl.c | 892 +++ drivers/misc/dlb2/dlb2_ioctl.h | 14 + drivers/misc/dlb2/dlb2_main.c | 1091 +++ drivers/misc/dlb2/dlb2_main.h | 285 + drivers/misc/dlb2/dlb2_pf_ops.c | 1286 ++++ drivers/misc/dlb2/dlb2_regs.h | 3702 ++++++++++ drivers/misc/dlb2/dlb2_resource.c | 7117 ++++++++++++++++++++ drivers/misc/dlb2/dlb2_resource.h | 924 +++ include/linux/pci_ids.h | 2 + include/uapi/linux/dlb2_user.h | 1045 +++ 25 files changed, 17869 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-driver-dlb2 create mode 100644 Documentation/misc-devices/dlb2.rst create mode 100644 drivers/misc/dlb2/Kconfig create mode 100644 drivers/misc/dlb2/Makefile create mode 100644 drivers/misc/dlb2/dlb2_bitmap.h create mode 100644 drivers/misc/dlb2/dlb2_file.c create mode 100644 drivers/misc/dlb2/dlb2_file.h create mode 100644 drivers/misc/dlb2/dlb2_hw_types.h create mode 100644 drivers/misc/dlb2/dlb2_intr.c create mode 100644 drivers/misc/dlb2/dlb2_intr.h create mode 100644 drivers/misc/dlb2/dlb2_ioctl.c create mode 100644 drivers/misc/dlb2/dlb2_ioctl.h create mode 100644 drivers/misc/dlb2/dlb2_main.c create mode 100644 drivers/misc/dlb2/dlb2_main.h create mode 100644 drivers/misc/dlb2/dlb2_pf_ops.c create mode 100644 drivers/misc/dlb2/dlb2_regs.h create mode 100644 drivers/misc/dlb2/dlb2_resource.c create mode 100644 drivers/misc/dlb2/dlb2_resource.h create mode 100644 include/uapi/linux/dlb2_user.h
base-commit: dd5597245d35cfbb0890b8a868028aa1d2018701
Signature · 1 line
2.13.6