New Microsemi PCI Switch Management Driver
Subsystems (4)
Logan Gunthorpe
maintainerv6 cover New Microsemi PCI Switch Management Driver
Hi,
Hopefully this is the last change necessary. The new code to handle unbind follows Jason's suggestions and is a lot easier to reason about. It's also been tested interrupting a number of operations.
Thanks,
Logan
--
Changes since v5:
* Reworked cleanup during unbind again. This time we follow the pattern roughly suggested by Jason Gunthorpe: we use an alive flag that's checked with a rw semaphore held by the cdev ops. Except instead of using the rwsem we just use the already existing mrpc_mutex. (Seeing the vast majority of locations that would use the semaphore overlap with the existing mutex.)
* Added a small performance enhancement so the code reads less io memory if the user is quick in submitting their read request.
Changes since v4:
* Turns out pushing the pci release code into the device release function didn't work as I would have liked. If you try to unbind the device with an instance open, then you hit a kernel bug at drivers/pci/msi.c:371. (This didn't occur in v3.)
To solve this, we've moved the pci release code back into the unregister function and reintroduced an alive flag. This time, however, the alive flag is protected by mrpc_mutex and we're very careful about what happens to devices still in use (they should all be released through the timeout path and an ENODEV error returned to userspace; while new commands are blocked with the same error).
Changes since v3:
* Removed stdev_is_alive and moved pci deinit functions
into the device release function which only occurs after all cdev instances are closed. (per Bjorn)
* Reduced locking in read/write functions (per Bjorn)
* Use pci_alloc_irq_vectors to vastly improve ISR initialization (Bjorn)
* A few other minor nits and cleanup as noticed by Bjorn
Changes since v2:
* Collected reviewed and tested tags
* Very minor fix to the error path in the create function
Changes since v1:
* Rebased onto 4.10-rc6 (cleanly)
* Split the patch into a few more easily digestible patches (as suggested by Greg Kroah-Hartman)
* Folded switchtec.c into switchtec.h (per Greg)
* Fixed a bunch of 32bit build warnings caught by the kbuild test robot
* Fixed some issues in the documentation so it has a proper reStructredText format (as noted by Jonathan Corbet)
* Fixed padding and sizes in the IOCTL structures as noticed by Emil Velikov and used pahole to verify their consistency across 32 and 64 bit builds
* Reworked one of the IOCTL interfaces to be more future proof (per Emil).
Changes since RFC:
* Fixed incorrect use of the drive model as pointed out by Greg Kroah-Hartman
* Used devm functions as suggested by Keith Busch
* Added a handful of sysfs attributes to the switchtec class
* Added a handful of IOCTLs to the switchtec device
* A number of miscellaneous bug fixes
--
Hi,
This is a continuation of the RFC we posted lasted month [1] which proposes a management driver for Microsemi's Switchtec line of PCI switches. This hardware is still looking to be used in the Open Compute Platform
To make this entirely clear: the Switchtec products are compliant with the PCI specifications and are supported today with the standard in-kernel driver. However, these devices also expose a management endpoint on a separate PCI function address which can be used to perform some advanced operations. This is a driver for that function. See the patch for more information.
Since the RFC, we've made the changes requested by Greg Kroah-Hartman and Keith Busch, and we've also fleshed out a number of features. We've added a couple of IOCTLs and sysfs attributes which are documented in the patch. Significant work has also been done on the userspace tool which is available under a GPL license at [2]. We've also had testing done by some of the interested parties.
We hope to see this work included in either 4.11 or 4.12 assuming a smooth review process.
The patch is based off of the v4.10-rc6 release.
Thanks for your review,
Logan
[1] https://www.spinics.net/lists/linux-pci/msg56897.html
[2] https://github.com/sbates130272/switchtec-user
Logan Gunthorpe (4):
MicroSemi Switchtec management interface driver
switchtec: Add user interface documentation
switchtec: Add sysfs attributes to the Switchtec driver
switchtec: Add IOCTLs to the Switchtec driver
Documentation/ABI/testing/sysfs-class-switchtec | 96 ++ Documentation/ioctl/ioctl-number.txt | 1 + Documentation/switchtec.txt | 80 ++ MAINTAINERS | 11 + drivers/pci/Kconfig | 1 + drivers/pci/Makefile | 1 + drivers/pci/switch/Kconfig | 13 + drivers/pci/switch/Makefile | 1 + drivers/pci/switch/switchtec.c | 1599 +++++++++++++++++++++++ include/uapi/linux/switchtec_ioctl.h | 132 ++ 10 files changed, 1935 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-class-switchtec create mode 100644 Documentation/switchtec.txt create mode 100644 drivers/pci/switch/Kconfig create mode 100644 drivers/pci/switch/Makefile create mode 100644 drivers/pci/switch/switchtec.c create mode 100644 include/uapi/linux/switchtec_ioctl.h
--
2.1.4
Logan Gunthorpe
maintainerPATCH v6 1/4 MicroSemi Switchtec management interface driver
Microsemi's "Switchtec" line of PCI switch devices is already well supported by the kernel with standard PCI switch drivers. However, the Switchtec device advertises a special management endpoint with a separate PCI function address and class code. This endpoint enables some additional functionality which includes:
* Packet and Byte Counters
* Switch Firmware Upgrades
* Event and Error logs
* Querying port link status
* Custom user firmware commands
This patch introduces the switchtec kernel module which provides PCI driver that exposes a char device. The char device provides userspace access to this interface through read, write and (optionally) poll calls.
A userspace tool and library which utilizes this interface is available at [1]. This tool takes inspiration (and borrows some code) from nvme-cli [2]. The tool is largely complete at this time but additional features may be added in the future.
[1] https://github.com/sbates130272/switchtec-user
[2] https://github.com/linux-nvme/nvme-cli
--- MAINTAINERS | 8 + drivers/pci/Kconfig | 1 + drivers/pci/Makefile | 1 + drivers/pci/switch/Kconfig | 13 + drivers/pci/switch/Makefile | 1 + drivers/pci/switch/switchtec.c | 1005 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 1029 insertions(+) create mode 100644 drivers/pci/switch/Kconfig create mode 100644 drivers/pci/switch/Makefile create mode 100644 drivers/pci/switch/switchtec.c
MAINTAINERS +8 -0
@@ -9506,6 +9506,14 @@ S: Maintained F: Documentation/devicetree/bindings/pci/aardvark-pci.txt F: drivers/pci/host/pci-aardvark.c +PCI DRIVER FOR MICROSEMI SWITCHTEC +M: Kurt Schwemmer <kurt.schwemmer at microsemi.com> +M: Stephen Bates <stephen.bates at microsemi.com> +M: Logan Gunthorpe <logang at deltatee.com> +L: linux-pci at vger.kernel.org +S: Maintained +F: drivers/pci/switch/switchtec* + PCI DRIVER FOR NVIDIA TEGRA M: Thierry Reding <thierry.reding at gmail.com> L: linux-tegra at vger.kernel.org
drivers/pci/Kconfig +1 -0
@@ -133,3 +133,4 @@ config PCI_HYPERV source "drivers/pci/hotplug/Kconfig" source "drivers/pci/host/Kconfig" +source "drivers/pci/switch/Kconfig"
drivers/pci/Makefile +1 -0
@@ -68,3 +68,4 @@ ccflags-$(CONFIG_PCI_DEBUG) := -DDEBUG # PCI host controller drivers obj-y += host/ +obj-y += switch/
drivers/pci/switch/Kconfig new file +13 -0
@@ -0,0 +1,13 @@ +menu "PCI switch controller drivers" + depends on PCI + +config PCI_SW_SWITCHTEC + tristate "MicroSemi Switchtec PCIe Switch Management Driver" + help + Enables support for the management interface for the MicroSemi + Switchtec series of PCIe switches. Supports userspace access + to submit MRPC commands to the switch via /dev/switchtecX + devices. See <file:Documentation/switchtec.txt> for more + information. + +endmenu
drivers/pci/switch/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-$(CONFIG_PCI_SW_SWITCHTEC) += switchtec.o
drivers/pci/switch/switchtec.c new file +1005 -0
1,005 lines not on this page · raw message
Signature · 1 line
2.1.4
Logan Gunthorpe
maintainerPATCH v6 2/4 switchtec: Add user interface documentation
In reply to Logan GunthorpeThis adds standard documentation for the sysfs switchtec attributes and a RST formatted text file which documents the char device interface. Jonathan Corbet has indicated he will move this to a new user-space developer documentation book once it's created.
--- Documentation/switchtec.txt | 53 +++++++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 54 insertions(+) create mode 100644 Documentation/switchtec.txt
Documentation/switchtec.txt new file +53 -0
@@ -0,0 +1,53 @@ +======================== +Linux Switchtec Support +======================== + +Microsemi's "Switchtec" line of PCI switch devices is already +supported by the kernel with standard PCI switch drivers. However, the +Switchtec device advertises a special management endpoint which +enables some additional functionality. This includes: + +* Packet and Byte Counters +* Firmware Upgrades +* Event and Error logs +* Querying port link status +* Custom user firmware commands + +The switchtec kernel module implements this functionality. + + +Interface +========= + +The primary means of communicating with the Switchtec management firmware is +through the Memory-mapped Remote Procedure Call (MRPC) interface. +Commands are submitted to the interface with a 4-byte command +identifier and up to 1KB of command specific data. The firmware will +respond with a 4 bytes return code and up to 1KB of command specific +data. The interface only processes a single command at a time. + + +Userspace Interface +=================== + +The MRPC interface will be exposed to userspace through a simple char +device: /dev/switchtec#, one for each management endpoint in the system. + +The char device has the following semantics: + +* A write must consist of at least 4 bytes and no more than 1028 bytes. + The first four bytes will be interpreted as the command to run and + the remainder will be used as the input data. A write will send the + command to the firmware to begin processing. + +* Each write must be followed by exactly one read. Any double write will + produce an error and any read that doesn't follow a write will + produce an error. + +* A read will block until the firmware completes the command and return + the four bytes of status plus up to 1024 bytes of output data. (The + length will be specified by the size parameter of the read call -- + reading less than 4 bytes will produce an error. + +* The poll call will also be supported for userspace applications that + need to do other things while waiting for the command to complete.
MAINTAINERS +1 -0
@@ -9512,6 +9512,7 @@ M: Stephen Bates <stephen.bates at microsemi.com> M: Logan Gunthorpe <logang at deltatee.com> L: linux-pci at vger.kernel.org S: Maintained +F: Documentation/switchtec.txt F: drivers/pci/switch/switchtec* PCI DRIVER FOR NVIDIA TEGRA
Signature · 1 line
2.1.4
Logan Gunthorpe
maintainerPATCH v6 3/4 switchtec: Add sysfs attributes to the Switchtec driver
In reply to Logan GunthorpeThis patch adds a few read-only sysfs attributes which provide some device information that is exposed from the devices. Primarily component and device names and versions. These are documented in Documentation/ABI/testing/sysfs-class-switchtec.
--- Documentation/ABI/testing/sysfs-class-switchtec | 96 ++++++++++++++++++++ MAINTAINERS | 1 + drivers/pci/switch/switchtec.c | 113 ++++++++++++++++++++++++ 3 files changed, 210 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-class-switchtec
Documentation/ABI/testing/sysfs-class-switchtec new file +96 -0
@@ -0,0 +1,96 @@ +switchtec - Microsemi Switchtec PCI Switch Management Endpoint + +For details on this subsystem look at Documentation/switchtec.txt. + +What: /sys/class/switchtec +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: The switchtec class subsystem folder. + Each registered switchtec driver is represented by a switchtecX + subfolder (X being an integer >= 0). + + +What: /sys/class/switchtec/switchtec[0-9]+/component_id +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Component identifier as stored in the hardware (eg. PM8543) + (read only) +Values: arbitrary string. + + +What: /sys/class/switchtec/switchtec[0-9]+/component_revision +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Component revision stored in the hardware (read only) +Values: integer. + + +What: /sys/class/switchtec/switchtec[0-9]+/component_vendor +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Component vendor as stored in the hardware (eg. MICROSEM) + (read only) +Values: arbitrary string. + + +What: /sys/class/switchtec/switchtec[0-9]+/device_version +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Device version as stored in the hardware (read only) +Values: integer. + + +What: /sys/class/switchtec/switchtec[0-9]+/fw_version +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Currently running firmware version (read only) +Values: integer (in hexadecimal). + + +What: /sys/class/switchtec/switchtec[0-9]+/partition +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Partition number for this device in the switch (read only) +Values: integer. + + +What: /sys/class/switchtec/switchtec[0-9]+/partition_count +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Total number of partitions in the switch (read only) +Values: integer. + + +What: /sys/class/switchtec/switchtec[0-9]+/product_id +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Product identifier as stored in the hardware (eg. PSX 48XG3) + (read only) +Values: arbitrary string. + + +What: /sys/class/switchtec/switchtec[0-9]+/product_revision +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Product revision stored in the hardware (eg. RevB) + (read only) +Values: arbitrary string. + + +What: /sys/class/switchtec/switchtec[0-9]+/product_vendor +Date: 05-Jan-2017 +KernelVersion: v4.11 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: Product vendor as stored in the hardware (eg. MICROSEM) + (read only) +Values: arbitrary string.
MAINTAINERS +1 -0
@@ -9513,6 +9513,7 @@ M: Logan Gunthorpe <logang at deltatee.com> L: linux-pci at vger.kernel.org S: Maintained F: Documentation/switchtec.txt +F: Documentation/ABI/testing/sysfs-class-switchtec F: drivers/pci/switch/switchtec* PCI DRIVER FOR NVIDIA TEGRA
drivers/pci/switch/switchtec.c +113 -0
125 lines not on this page · raw message
Signature · 1 line
2.1.4
Logan Gunthorpe
maintainerPATCH v6 4/4 switchtec: Add IOCTLs to the Switchtec driver
In reply to Logan GunthorpeThis patch introduces a couple of special IOCTLs which are provided to:
* Inform userspace of firmware partition locations
* Pass event counts and allow userspace to wait on events
* Translate between PFF numbers used by the switch to port numbers.
--- Documentation/ioctl/ioctl-number.txt | 1 + Documentation/switchtec.txt | 27 ++ MAINTAINERS | 1 + drivers/pci/switch/switchtec.c | 481 +++++++++++++++++++++++++++++++++++ include/uapi/linux/switchtec_ioctl.h | 132 ++++++++++ 5 files changed, 642 insertions(+) create mode 100644 include/uapi/linux/switchtec_ioctl.h
Documentation/ioctl/ioctl-number.txt +1 -0
@@ -191,6 +191,7 @@ Code Seq#(hex) Include File Comments 'W' 00-1F linux/watchdog.h conflict! 'W' 00-1F linux/wanrouter.h conflict! (pre 3.9) 'W' 00-3F sound/asound.h conflict! +'W' 40-5F drivers/pci/switch/switchtec.c 'X' all fs/xfs/xfs_fs.h conflict! and fs/xfs/linux-2.6/xfs_ioctl32.h and include/linux/falloc.h
Documentation/switchtec.txt +27 -0
@@ -51,3 +51,30 @@ The char device has the following semantics: * The poll call will also be supported for userspace applications that need to do other things while waiting for the command to complete. + +The following IOCTLs are also supported by the device: + +* SWITCHTEC_IOCTL_FLASH_INFO - Retrieve firmware length and number + of partitions in the device. + +* SWITCHTEC_IOCTL_FLASH_PART_INFO - Retrieve address and lengeth for + any specified partition in flash. + +* SWITCHTEC_IOCTL_EVENT_SUMMARY - Read a structure of bitmaps + indicating all uncleared events. + +* SWITCHTEC_IOCTL_EVENT_CTL - Get the current count, clear and set flags + for any event. This ioctl takes in a switchtec_ioctl_event_ctl struct + with the event_id, index and flags set (index being the partition or PFF + number for non-global events). It returns whether the event has + occurred, the number of times and any event specific data. The flags + can be used to clear the count or enable and disable actions to + happen when the event occurs. + By using the SWITCHTEC_IOCTL_EVENT_FLAG_EN_POLL flag, + you can set an event to trigger a poll command to return with + POLLPRI. In this way, userspace can wait for events to occur. + +* SWITCHTEC_IOCTL_PFF_TO_PORT and SWITCHTEC_IOCTL_PORT_TO_PFF convert + between PCI Function Framework number (used by the event system) + and Switchtec Logic Port ID and Partition number (which is more + user friendly).
MAINTAINERS +1 -0
@@ -9515,6 +9515,7 @@ S: Maintained F: Documentation/switchtec.txt F: Documentation/ABI/testing/sysfs-class-switchtec F: drivers/pci/switch/switchtec* +F: include/uapi/linux/switchtec_ioctl.h PCI DRIVER FOR NVIDIA TEGRA M: Thierry Reding <thierry.reding at gmail.com>
drivers/pci/switch/switchtec.c +481 -0
516 lines not on this page · raw message
include/uapi/linux/switchtec_ioctl.h new file +132 -0
132 lines not on this page · raw message
Signature · 1 line
2.1.4
Logan Gunthorpe
maintainerHey,
Ah, sorry, sorry. I noticed another minor mistake. I'll send a v7 in a second.
Thanks,
Logan
Quoted 138 lines · Logan Gunthorpe
Hi,
Hopefully this is the last change necessary. The new code to handle unbind follows Jason's suggestions and is a lot easier to reason about. It's also been tested interrupting a number of operations.
Thanks,
Logan
--
Changes since v5:
* Reworked cleanup during unbind again. This time we follow the pattern roughly suggested by Jason Gunthorpe: we use an alive flag that's checked with a rw semaphore held by the cdev ops. Except instead of using the rwsem we just use the already existing
mrpc_mutex. (Seeing the vast majority of locations that would use the semaphore overlap with the existing mutex.)
* Added a small performance enhancement so the code reads less io memory if the user is quick in submitting their read request.Changes since v4:
* Turns out pushing the pci release code into the device release function didn't work as I would have liked. If you try to unbind the device with an instance open, then you hit a kernel bug at
drivers/pci/msi.c:371. (This didn't occur in v3.)To solve this, we've moved the pci release code back into the unregister function and reintroduced an alive flag. This time, however, the alive flag is protected by
mrpc_mutexand we're very careful about what happens to devices still in use (they should all be released through the timeout path and an ENODEV error returned to userspace; while new commands are blocked with the same error).Changes since v3:
* Removed
stdev_is_aliveand moved pci deinit functions
into the device release function which only occurs after all cdev instances are closed. (per Bjorn)
* Reduced locking in read/write functions (per Bjorn)
* Usepci_alloc_irq_vectorsto vastly improve ISR initialization (Bjorn)
* A few other minor nits and cleanup as noticed by BjornChanges since v2:
* Collected reviewed and tested tags
* Very minor fix to the error path in the create functionChanges since v1:
* Rebased onto 4.10-rc6 (cleanly)
* Split the patch into a few more easily digestible patches (as suggested by Greg Kroah-Hartman)
* Foldedswitchtec.cintoswitchtec.h(per Greg)
* Fixed a bunch of 32bit build warnings caught by the kbuild test robot
* Fixed some issues in the documentation so it has a proper reStructredText format (as noted by Jonathan Corbet)
* Fixed padding and sizes in the IOCTL structures as noticed by Emil Velikov and used pahole to verify their consistency across 32 and 64 bit builds
* Reworked one of the IOCTL interfaces to be more future proof (per Emil).Changes since RFC:
* Fixed incorrect use of the drive model as pointed out by Greg Kroah-Hartman
* Used devm functions as suggested by Keith Busch
* Added a handful of sysfs attributes to the switchtec class
* Added a handful of IOCTLs to the switchtec device
* A number of miscellaneous bug fixes--
Hi,
This is a continuation of the RFC we posted lasted month [1] which proposes a management driver for Microsemi's Switchtec line of PCI switches. This hardware is still looking to be used in the Open Compute Platform
To make this entirely clear: the Switchtec products are compliant with the PCI specifications and are supported today with the standard in-kernel driver. However, these devices also expose a management endpoint on a separate PCI function address which can be used to perform some advanced operations. This is a driver for that function. See the patch for more information.
Since the RFC, we've made the changes requested by Greg Kroah-Hartman and Keith Busch, and we've also fleshed out a number of features. We've added a couple of IOCTLs and sysfs attributes which are documented in the patch. Significant work has also been done on the userspace tool which is available under a GPL license at [2]. We've also had testing done by some of the interested parties.
We hope to see this work included in either 4.11 or 4.12 assuming a smooth review process.
The patch is based off of the v4.10-rc6 release.
Thanks for your review,
Logan
[1] https://www.spinics.net/lists/linux-pci/msg56897.html
[2] https://github.com/sbates130272/switchtec-userLogan Gunthorpe (4):
MicroSemi Switchtec management interface driver
switchtec: Add user interface documentation
switchtec: Add sysfs attributes to the Switchtec driver
switchtec: Add IOCTLs to the Switchtec driverDocumentation/ABI/testing/sysfs-class-switchtec | 96 ++ Documentation/ioctl/ioctl-number.txt | 1 + Documentation/switchtec.txt | 80 ++ MAINTAINERS | 11 + drivers/pci/Kconfig | 1 + drivers/pci/Makefile | 1 + drivers/pci/switch/Kconfig | 13 + drivers/pci/switch/Makefile | 1 + drivers/pci/switch/switchtec.c | 1599 +++++++++++++++++++++++ include/uapi/linux/switchtec_ioctl.h | 132 ++ 10 files changed, 1935 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-class-switchtec create mode 100644 Documentation/switchtec.txt create mode 100644 drivers/pci/switch/Kconfig create mode 100644 drivers/pci/switch/Makefile create mode 100644 drivers/pci/switch/switchtec.c create mode 100644 include/uapi/linux/switchtec_ioctl.h--
2.1.4