Skip to content

dlb2: introduce DLB 2.0 device driver

Gage Eads lkml 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest19 patches 6 repliesIn review
  2. v219 patches 3 repliesSuperseded
  3. v120 patches 25 repliesSuperseded

Changes in v3 from the cover letter

- Remove DLB2_PCI_REG_READ/WRITE macros

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/19dlb2: add skeleton for DLB 2.0 driverDocumentation/misc-devices/dlb2.rst Documentation/misc-devices/index.rst MAINTAINERS +8 more+621 -06 replies
2/19dlb2: initialize PF devicedrivers/misc/dlb2/Makefile drivers/misc/dlb2/dlb2_hw_types.h drivers/misc/dlb2/dlb2_main.c +2 more+235 -30 replies
3/19dlb2: add resource and device initializationdrivers/misc/dlb2/Makefile drivers/misc/dlb2/dlb2_bitmap.h drivers/misc/dlb2/dlb2_hw_types.h +6 more+754 -30 replies
4/19dlb2: add device ioctl layer and first three ioctlsDocumentation/userspace-api/ioctl/ioctl-number.rst drivers/misc/dlb2/Makefile drivers/misc/dlb2/dlb2_bitmap.h +8 more+428 -00 replies
5/19dlb2: add sched domain config and reset supportdrivers/misc/dlb2/dlb2_bitmap.h drivers/misc/dlb2/dlb2_hw_types.h drivers/misc/dlb2/dlb2_ioctl.c +7 more+7252 -20 replies
6/19dlb2: add runtime power-management supportdrivers/misc/dlb2/dlb2_main.c drivers/misc/dlb2/dlb2_main.h drivers/misc/dlb2/dlb2_pf_ops.c+104 -00 replies
7/19dlb2: add queue create and queue-depth-get ioctlsdrivers/misc/dlb2/dlb2_ioctl.c drivers/misc/dlb2/dlb2_ioctl.h drivers/misc/dlb2/dlb2_main.c +5 more+1116 -00 replies
8/19dlb2: add ioctl to configure ports, query poll modedrivers/misc/dlb2/dlb2_hw_types.h drivers/misc/dlb2/dlb2_ioctl.c drivers/misc/dlb2/dlb2_main.c +5 more+1400 -00 replies
9/19dlb2: add port mmap supportdrivers/misc/dlb2/Makefile drivers/misc/dlb2/dlb2_file.c drivers/misc/dlb2/dlb2_file.h +7 more+650 -00 replies
10/19dlb2: add start domain ioctldrivers/misc/dlb2/dlb2_ioctl.c drivers/misc/dlb2/dlb2_main.h drivers/misc/dlb2/dlb2_pf_ops.c +3 more+189 -00 replies
11/19dlb2: add queue map and unmap ioctlsdrivers/misc/dlb2/dlb2_ioctl.c drivers/misc/dlb2/dlb2_main.h drivers/misc/dlb2/dlb2_pf_ops.c +3 more+1032 -00 replies
12/19dlb2: add port enable/disable ioctlsdrivers/misc/dlb2/dlb2_ioctl.c drivers/misc/dlb2/dlb2_main.h drivers/misc/dlb2/dlb2_pf_ops.c +3 more+762 -00 replies
13/19dlb2: add CQ interrupt supportdrivers/misc/dlb2/Makefile drivers/misc/dlb2/dlb2_hw_types.h drivers/misc/dlb2/dlb2_intr.c +8 more+987 -10 replies
14/19dlb2: add domain alert supportdrivers/misc/dlb2/dlb2_ioctl.c drivers/misc/dlb2/dlb2_main.c drivers/misc/dlb2/dlb2_main.h +1 more+297 -00 replies
15/19dlb2: add sequence-number management ioctlsdrivers/misc/dlb2/dlb2_ioctl.c drivers/misc/dlb2/dlb2_main.h drivers/misc/dlb2/dlb2_pf_ops.c +3 more+308 -00 replies
16/19dlb2: add cos bandwidth get/set ioctlsdrivers/misc/dlb2/dlb2_ioctl.c drivers/misc/dlb2/dlb2_main.h drivers/misc/dlb2/dlb2_pf_ops.c +3 more+209 -00 replies
17/19dlb2: add device FLR supportDocumentation/ABI/testing/sysfs-driver-dlb2 drivers/misc/dlb2/dlb2_intr.c drivers/misc/dlb2/dlb2_intr.h +5 more+424 -110 replies
18/19dlb2: add basic PF sysfs interfacesDocumentation/ABI/testing/sysfs-driver-dlb2 drivers/misc/dlb2/dlb2_main.c drivers/misc/dlb2/dlb2_main.h +1 more+602 -00 replies
19/19dlb2: add ingress error handlingDocumentation/ABI/testing/sysfs-driver-dlb2 drivers/misc/dlb2/dlb2_hw_types.h drivers/misc/dlb2/dlb2_main.h +3 more+520 -10 replies

Reviews trailers per patch

All 19 patches have a Reviewed-by or Acked-by.

PatchMagnus KarlssonBjörn Töpel
1/19R-b
2/19R-b
3/19R-b
4/19R-b
5/19R-b
6/19R-b
7/19R-b
8/19R-b
9/19R-b
10/19R-b
11/19R-b
12/19R-b
13/19R-b
14/19R-b
15/19R-b
16/19R-b
17/19R-b
18/19R-b
19/19R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [PATCH v3 00/19] dlb2: introduce DLB 2.0 device driver

This commit introduces a new misc device driver for the Intel(r) Dynamic Load Balancer 2.0 (Intel(r) DLB 2.0). The Intel DLB 2.0 is a PCIe device that provides load-balanced, prioritized scheduling of core-to-core communication.

The Intel DLB 2.0 consists of queues and arbiters that connect producer cores and consumer cores. The device implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced distribution.

Intel DLB 2.0 can be used in an event-driven programming model, such as DPDK's Event Device Library[2]. Such frameworks are commonly used in packet processing pipelines that benefit from the framework's multi-core scalability, dynamic load-balancing, and variety of packet distribution and synchronization schemes.

These distribution schemes include "parallel" (packets are load-balanced across multiple cores and processed in parallel), "ordered" (similar to "parallel" but packets are reordered into ingress order by the device), and "atomic" (packet flows are scheduled to a single core at a time such that locks are not required to access per-flow data, and dynamically migrated to ensure load-balance).

The fundamental unit of communication through the device is a queue entry (QE), which consists of 8B of data and 8B of metadata (destination queue, priority, etc.). The data field can be any type that fits within 8B.

A core's interface to the device, a "port," consists of a memory-mappable region through which the core enqueues a queue entry, and an in-memory queue (the "consumer queue") to which the device schedules QEs. Each QE is enqueued to a device-managed queue, and from there scheduled to a port. Software specifies the "linking" of queues and ports; i.e. which ports the device is allowed to schedule to for a given queue. The device uses a credit scheme to prevent overflow of the on-device queue storage.

Applications can interface directly with the device by mapping the port's memory and MMIO regions into the application's address space for enqueue and dequeue operations, but call into the kernel driver for configuration operations. An application can also be polling- or interrupt-driven; Intel DLB 2.0 supports both modes of operation.

Device resources -- i.e. ports, queues, and credits -- are contained within a scheduling domain. Scheduling domains are isolated from one another; a port can only enqueue to and dequeue from queues within its scheduling domain. A scheduling domain's resources are configured through a scheduling domain file, which is acquired through an ioctl.

Intel DLB 2.0 supports SR-IOV and Scalable IOV, and allows for a flexible division of its resources among the PF and its virtual devices. The virtual devices are incapable of configuring the device directly; they use a hardware mailbox to proxy configuration requests to the PF driver. This driver supports both PF and virtual devices, as there is significant code re-use between the two, with device-specific behavior handled through a callback interface. Virtualization support will be added in a later patch set.

The dlb driver uses ioctls as its primary interface (it makes use of sysfs as well, to a lesser extent). The dlb device file supports a different ioctl interface than the scheduling domain file; the dlb device file is used for device-wide operations (including scheduling domain creation), and the scheduling domain file supports operations on the scheduling domain's resources (primarily resource configuration).

[1] https://builders.intel.com/docs/networkbuilders/SKU-343247-001US-queue-management-and-load-balancing-on-intel-architecture.pdf
[2] https://doc.dpdk.org/guides/prog_guide/eventdev.html

v3:
- Remove DLB2_PCI_REG_READ/WRITE macros

v2:
- Change driver license to GPLv2 only
- Expand Kconfig help text and remove unnecessary (R)s
- Remove unnecessary prints
- Add a new entry in ioctl-number.rst
- Convert the ioctl handler into a switch statement
- Correct some instances of IOWR that should have been IOR
- Align macro blocks
- Don't break ioctl ABI when introducing new commands
- Remove indirect pointers from ioctl data structures
- Remove the get-sched-domain-fd ioctl command

Gage Eads (19):
dlb2: add skeleton for DLB 2.0 driver
dlb2: initialize PF device
dlb2: add resource and device initialization
dlb2: add device ioctl layer and first three ioctls
dlb2: add sched domain config and reset support
dlb2: add runtime power-management support
dlb2: add queue create and queue-depth-get ioctls
dlb2: add ioctl to configure ports, query poll mode
dlb2: add port mmap support
dlb2: add start domain ioctl
dlb2: add queue map and unmap ioctls
dlb2: add port enable/disable ioctls
dlb2: add CQ interrupt support
dlb2: add domain alert support
dlb2: add sequence-number management ioctls
dlb2: add cos bandwidth get/set ioctls
dlb2: add device FLR support
dlb2: add basic PF sysfs interfaces
dlb2: add ingress error handling

 Documentation/ABI/testing/sysfs-driver-dlb2        |  202 +
 Documentation/misc-devices/dlb2.rst                |  310 +
 Documentation/misc-devices/index.rst               |    1 +
 Documentation/userspace-api/ioctl/ioctl-number.rst |    1 +
 MAINTAINERS                                        |    7 +
 drivers/misc/Kconfig                               |    1 +
 drivers/misc/Makefile                              |    1 +
 drivers/misc/dlb2/Kconfig                          |   17 +
 drivers/misc/dlb2/Makefile                         |   13 +
 drivers/misc/dlb2/dlb2_bitmap.h                    |  286 +
 drivers/misc/dlb2/dlb2_file.c                      |  133 +
 drivers/misc/dlb2/dlb2_file.h                      |   19 +
 drivers/misc/dlb2/dlb2_hw_types.h                  |  353 +
 drivers/misc/dlb2/dlb2_intr.c                      |  137 +
 drivers/misc/dlb2/dlb2_intr.h                      |   30 +
 drivers/misc/dlb2/dlb2_ioctl.c                     |  892 +++
 drivers/misc/dlb2/dlb2_ioctl.h                     |   14 +
 drivers/misc/dlb2/dlb2_main.c                      | 1091 +++
 drivers/misc/dlb2/dlb2_main.h                      |  285 +
 drivers/misc/dlb2/dlb2_pf_ops.c                    | 1286 ++++
 drivers/misc/dlb2/dlb2_regs.h                      | 3702 ++++++++++
 drivers/misc/dlb2/dlb2_resource.c                  | 7117 ++++++++++++++++++++
 drivers/misc/dlb2/dlb2_resource.h                  |  924 +++
 include/linux/pci_ids.h                            |    2 +
 include/uapi/linux/dlb2_user.h                     | 1045 +++
 25 files changed, 17869 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-driver-dlb2
 create mode 100644 Documentation/misc-devices/dlb2.rst
 create mode 100644 drivers/misc/dlb2/Kconfig
 create mode 100644 drivers/misc/dlb2/Makefile
 create mode 100644 drivers/misc/dlb2/dlb2_bitmap.h
 create mode 100644 drivers/misc/dlb2/dlb2_file.c
 create mode 100644 drivers/misc/dlb2/dlb2_file.h
 create mode 100644 drivers/misc/dlb2/dlb2_hw_types.h
 create mode 100644 drivers/misc/dlb2/dlb2_intr.c
 create mode 100644 drivers/misc/dlb2/dlb2_intr.h
 create mode 100644 drivers/misc/dlb2/dlb2_ioctl.c
 create mode 100644 drivers/misc/dlb2/dlb2_ioctl.h
 create mode 100644 drivers/misc/dlb2/dlb2_main.c
 create mode 100644 drivers/misc/dlb2/dlb2_main.h
 create mode 100644 drivers/misc/dlb2/dlb2_pf_ops.c
 create mode 100644 drivers/misc/dlb2/dlb2_regs.h
 create mode 100644 drivers/misc/dlb2/dlb2_resource.c
 create mode 100644 drivers/misc/dlb2/dlb2_resource.h
 create mode 100644 include/uapi/linux/dlb2_user.h

base-commit: dd5597245d35cfbb0890b8a868028aa1d2018701

Signature · 1 line
2.13.6