PCIe Enclosure LED Management
v6 cover PCIe Enclosure LED Management
Patchset is named as PCIe Enclosure LED Management because it adds two features:
- Native PCIe Enclosure Management (NPEM)
- PCIe SSD Status LED Management (DSM)
Both are pattern oriented standards, they tell which "indication" should blink. It doesn't control physical LED or pattern visualization.
Overall, driver is simple but it was not simple to fit it into interfaces we have in kernel (We considered leds and enclosure interfaces). It reuses leds interface, this approach seems to be the best because:
- leds are actively maintained, no new interface added.
- leds do not require any extensions, enclosure needs to be adjusted first.
There are trade-offs:
- "brightness" is the name of sysfs file to control led. It is not natural to use brightness to set patterns, that is why multiple led devices are created (one per indication);
- Update of one led may affect other leds, led triggers may not work as expected.
Changes from v1:
- Renamed "pattern" to indication.
- DSM support added.
- fixed nits reported by Bjorn.
Changes from v2:
- Introduce lazy loading to allow DELL _DSM quirks to work, reported by Stuart.
- leds class initcall moved up in Makefile, proposed by Dan.
- fix other nits reported by Dan and Iipo.
Changes from v3:
- Remove unnecessary packed attr.
- Fix doc issue reported by lkp.
- Fix read_poll_timeout() error handling reported by Iipo.
- Minor fixes reported by Christoph.
Changes from v4:
- Use 0 / 1 instead of LED_OFF/LED_ON, suggested by Marek.
- Documentation added, suggested by Bjorn.
Change from v5:
- Remove unnecessary _packed, reported by Christoph.
- Changed "led" to "LED" and other typos suggested by Randy.
Mariusz Tkaczyk (3):
leds: Init leds class earlier
PCI/NPEM: Add Native PCIe Enclosure Management support PCI/NPEM: Add _DSM PCIe SSD status LED management
Documentation/ABI/testing/sysfs-bus-pci | 72 +++ drivers/Makefile | 4 +- drivers/pci/Kconfig | 9 + drivers/pci/Makefile | 1 + drivers/pci/npem.c | 590 ++++++++++++++++++++++++ drivers/pci/pci.h | 8 + drivers/pci/probe.c | 2 + drivers/pci/remove.c | 2 + include/linux/pci.h | 3 + include/uapi/linux/pci_regs.h | 35 ++ 10 files changed, 725 insertions(+), 1 deletion(-) create mode 100644 drivers/pci/npem.c
Signature · 1 line
2.35.3
PATCH v6 1/3 leds: Init leds class earlier
NPEM driver will require leds class, there is an init-order conflict. Make sure that LEDs initialization happens first and add comment.
--- drivers/Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/Makefile +3 -1
@@ -17,6 +17,9 @@ obj-$(CONFIG_PINCTRL) += pinctrl/ obj-$(CONFIG_GPIOLIB) += gpio/ obj-y += pwm/ +# LEDs must come before PCI, it is needed by NPEM driver +obj-y += leds/ + obj-y += pci/ obj-$(CONFIG_PARISC) += parisc/
@@ -130,7 +133,6 @@ obj-$(CONFIG_CPU_IDLE) += cpuidle/ obj-y += mmc/ obj-y += ufs/ obj-$(CONFIG_MEMSTICK) += memstick/ -obj-y += leds/ obj-$(CONFIG_INFINIBAND) += infiniband/ obj-y += firmware/ obj-$(CONFIG_CRYPTO) += crypto/
Signature · 1 line
2.35.3
PATCH v6 2/3 PCI/NPEM: Add Native PCIe Enclosure Management support
In reply to Mariusz TkaczykNative PCIe Enclosure Management (NPEM, PCIe r6.1 sec 6.28) allows managing LED in storage enclosures. NPEM is indication oriented and it does not give direct access to LED. Although each of the indications *could* represent an individual LED, multiple indications could also be represented as a single, multi-color LED or a single LED blinking in a specific interval. The specification leaves that open.
Each enabled indication (capability register bit on) is represented as a ledclass_dev which can be controlled through sysfs. For every ledclass device only 2 brightness states are allowed: LED_ON (1) or LED_OFF (0). It is corresponding to NPEM control register (Indication bit on/off).
Ledclass devices appear in sysfs as child devices (subdirectory) of PCI device which has an NPEM Extended Capability and indication is enabled in NPEM capability register. For example, these are leds created for pcieport "10000:02:05.0" on my setup:
leds/
├── 10000:02:05.0:enclosure:fail ├── 10000:02:05.0:enclosure:locate ├── 10000:02:05.0:enclosure:ok └── 10000:02:05.0:enclosure:rebuild
They can be also found in "/sys/class/leds" directory. Parent PCIe device bdf is used to guarantee uniqueness across leds subsystem.
To enable/disable fail indication "brightness" file can be edited:
echo 1 > ./leds/10000:02:05.0:enclosure:fail/brightness echo 0 > ./leds/10000:02:05.0:enclosure:fail/brightness
PCIe r6.1, sec 7.9.19.2 defines the possible indications.
Multiple indications for same parent PCIe device can conflict and hardware may update them when processing new request. To avoid issues, driver refresh all indications by reading back control register.
Driver is projected to be exclusive NPEM extended capability manager. It waits up to 1 second after imposing new request, it doesn't verify if controller is busy before write, assuming that mutex lock gives protection from concurrent updates. Driver is not registered if _DSM LED management is available.
NPEM is a PCIe extended capability so it should be registered in pcie_init_capabilities() but it is not possible due to LED dependency. Parent pci_device must be added earlier for led_classdev_register() to be successful. NPEM does not require configuration on kernel side, it is safe to register LED devices later.
--- Documentation/ABI/testing/sysfs-bus-pci | 63 ++++ drivers/pci/Kconfig | 9 + drivers/pci/Makefile | 1 + drivers/pci/npem.c | 449 ++++++++++++++++++++++++ drivers/pci/pci.h | 8 + drivers/pci/probe.c | 2 + drivers/pci/remove.c | 2 + include/linux/pci.h | 3 + include/uapi/linux/pci_regs.h | 35 ++ 9 files changed, 572 insertions(+) create mode 100644 drivers/pci/npem.c
Documentation/ABI/testing/sysfs-bus-pci +63 -0
@@ -500,3 +500,66 @@ Description: console drivers from the device. Raw users of pci-sysfs resourceN attributes must be terminated prior to resizing. Success of the resizing operation is not guaranteed. + +What: /sys/bus/pci/devices/.../leds/*:enclosure:*/brightness +What: /sys/class/leds/*:enclosure:*/brightness +Date: August 2024 +KernelVersion: 6.12 +Description: + LED indications on PCIe storage enclosures which are controlled + through the NPEM interface (Native PCIe Enclosure Management, + PCIe r6.1 sec 6.28) are accessible as LED class devices, both + below /sys/class/leds and below NPEM-capable PCI devices. + + Although these LED class devices could be manipulated manually, + in practice they are typically manipulated automatically by an + application such as ledmon(8). + + The name of a LED class device is as follows: + <bdf>:enclosure:<indication> + where: + + - <bdf> is the domain, bus, device and function number + (e.g. 10000:02:05.0) + - <indication> is a short description of the LED indication + + Valid indications per PCIe r6.1 table 6-27 are: + + - ok (drive is functioning normally) + - locate (drive is being identified by an admin) + - fail (drive is not functioning properly) + - rebuild (drive is part of an array that is rebuilding) + - pfa (drive is predicted to fail soon) + - hotspare (drive is marked to be used as a replacement) + - ica (drive is part of an array that is degraded) + - ifa (drive is part of an array that is failed) + - idt (drive is not the right type for the connector) + - disabled (drive is disabled, removal is safe) + - specific0 to specific7 (enclosure-specific indications) + + Broadly, the indications fall into one of these categories: + + - to signify drive state (ok, locate, fail, idt, disabled) + - to signify drive role or state in a software RAID array + (rebuild, pfa, hotspare, ica, ifa) + - to signify any other role or state (specific0 to specific7) + + Mandatory indications per PCIe r6.1 sec 7.9.19.2 comprise: + ok, locate, fail, rebuild. All others are optional. + An LED class device is only visible if the corresponding + indication is supported by the device. + + To manipulate the indications, write 0 (LED_OFF) or 1 (LED_ON) + to the "brightness" file. Note that manipulating an indication + may implicitly manipulate other indications at the vendor's + discretion. E.g. when the user lights up the "ok" indication, + the vendor may choose to automatically turn off the "fail" + indication. The current state of an indication can be + retrieved by reading its "brightness" file. + + The PCIe Base Specification allows vendors leeway to choose + different colors or blinking patterns for the indications, + but they typically follow the IBPI standard. E.g. the "locate" + indication is usually presented as one or two LEDs blinking at + 4 Hz frequency: + https://en.wikipedia.org/wiki/International_Blinking_Pattern_Interpretation
drivers/pci/Kconfig +9 -0
@@ -143,6 +143,15 @@ config PCI_IOV If unsure, say N. +config PCI_NPEM + bool "Native PCIe Enclosure Management" + depends on LEDS_CLASS=y + help + Support for Native PCIe Enclosure Management. It allows managing LED + indications in storage enclosures. Enclosure must support following + indications: OK, Locate, Fail, Rebuild. Other indications are + optional. + config PCI_PRI bool "PCI PRI support" select PCI_ATS
drivers/pci/Makefile +1 -0
@@ -35,6 +35,7 @@ obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o obj-$(CONFIG_VGA_ARB) += vgaarb.o obj-$(CONFIG_PCI_DOE) += doe.o obj-$(CONFIG_PCI_DYNAMIC_OF_NODES) += of_property.o +obj-$(CONFIG_PCI_NPEM) += npem.o # Endpoint library must be initialized before its users obj-$(CONFIG_PCI_ENDPOINT) += endpoint/
drivers/pci/npem.c new file +449 -0
449 lines not on this page · raw message
drivers/pci/pci.h +8 -0
@@ -398,6 +398,14 @@ static inline void pci_doe_destroy(struct pci_dev *pdev) { }static inline void pci_doe_disconnected (struct pci_dev *pdev) { }#endif +#ifdef CONFIG_PCI_NPEM +void pci_npem_create (struct pci_dev *dev); +void pci_npem_remove (struct pci_dev *dev); +#else +static inline void pci_npem_create (struct pci_dev *dev) { } +static inline void pci_npem_remove (struct pci_dev *dev) { } +#endif +/** * pci_dev_set_io_state - Set the new error state if possible. *
drivers/pci/probe.c +2 -0
@@ -2593,6 +2593,8 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus) dev->match_driver = false; ret =device_add (&dev->dev);WARN_ON (ret <0 ); + +pci_npem_create (dev); }struct pci_dev *pci_scan_single_device (struct pci_bus *bus,int devfn)
drivers/pci/remove.c +2 -0
@@ -34,6 +34,8 @@ static void pci_destroy_dev(struct pci_dev *dev)if (!dev->dev.kobj.parent)return ; +pci_npem_remove (dev); +device_del (&dev->dev);down_write (&pci_bus_sem);
include/linux/pci.h +3 -0
@@ -516,6 +516,9 @@ struct pci_dev {#endif #ifdef CONFIG_PCI_DOE struct xarray doe_mbs;/* Data Object Exchange mailboxes */ +#endif +#ifdef CONFIG_PCI_NPEM +struct npem *npem;/* Native PCIe Enclosure Management */ #endif u16 acs_cap;/* ACS Capability offset */ phys_addr_t rom;/* Physical address if not from BAR */
include/uapi/linux/pci_regs.h +35 -0
@@ -740,6 +740,7 @@#define PCI_EXT_CAP_ID_DVSEC 0x23/* Designated Vendor-Specific */ #define PCI_EXT_CAP_ID_DLF 0x25/* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26/* Physical Layer 16.0 GT/s */ +#define PCI_EXT_CAP_ID_NPEM 0x29/* Native PCIe Enclosure Management */ #define PCI_EXT_CAP_ID_PL_32GT 0x2A/* Physical Layer 32.0 GT/s */ #define PCI_EXT_CAP_ID_DOE 0x2E/* Data Object Exchange */ #define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_DOE
@@ -1121,6 +1122,40 @@#define PCI_PL_16GT_LE_CTRL_USP_TX_PRESET_MASK 0x000000F0#define PCI_PL_16GT_LE_CTRL_USP_TX_PRESET_SHIFT 4 +/* Native PCIe Enclosure Management */ +#define PCI_NPEM_CAP 0x04/* NPEM capability register */ +#define PCI_NPEM_CAP_CAPABLE 0x00000001/* NPEM Capable */ + +#define PCI_NPEM_CTRL 0x08/* NPEM control register */ +#define PCI_NPEM_CTRL_ENABLE 0x00000001/* NPEM Enable */ + +/* +* Native PCIe Enclosure Management indication bits and Reset command bit +* are corresponding for capability and control registers. +*/ +#define PCI_NPEM_CMD_RESET 0x00000002/* NPEM Reset Command */ +#define PCI_NPEM_IND_OK 0x00000004/* NPEM indication OK */ +#define PCI_NPEM_IND_LOCATE 0x00000008/* NPEM indication Locate */ +#define PCI_NPEM_IND_FAIL 0x00000010/* NPEM indication Fail */ +#define PCI_NPEM_IND_REBUILD 0x00000020/* NPEM indication Rebuild */ +#define PCI_NPEM_IND_PFA 0x00000040/* NPEM indication Predicted Failure Analysis */ +#define PCI_NPEM_IND_HOTSPARE 0x00000080/* NPEM indication Hot Spare */ +#define PCI_NPEM_IND_ICA 0x00000100/* NPEM indication In Critical Array */ +#define PCI_NPEM_IND_IFA 0x00000200/* NPEM indication In Failed Array */ +#define PCI_NPEM_IND_IDT 0x00000400/* NPEM indication Invalid Device Type */ +#define PCI_NPEM_IND_DISABLED 0x00000800/* NPEM indication Disabled */ +#define PCI_NPEM_IND_SPEC_0 0x01000000 +#define PCI_NPEM_IND_SPEC_1 0x02000000 +#define PCI_NPEM_IND_SPEC_2 0x04000000 +#define PCI_NPEM_IND_SPEC_3 0x08000000 +#define PCI_NPEM_IND_SPEC_4 0x10000000 +#define PCI_NPEM_IND_SPEC_5 0x20000000 +#define PCI_NPEM_IND_SPEC_6 0x40000000 +#define PCI_NPEM_IND_SPEC_7 0x80000000 + +#define PCI_NPEM_STATUS 0x0c/* NPEM status register */ +#define PCI_NPEM_STATUS_CC 0x00000001/* NPEM Command completed */ +/* Data Object Exchange */ #define PCI_DOE_CAP 0x04/* DOE Capabilities Register */ #define PCI_DOE_CAP_INT_SUP 0x00000001/* Interrupt Support */
Signature · 1 line
2.35.3
Re: [PATCH v6 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
Quoted 9 lines · Mariusz Tkaczyk
Native PCIe Enclosure Management (NPEM, PCIe r6.1 sec 6.28) allows managing LED in storage enclosures. NPEM is indication oriented and it does not give direct access to LED. Although each of the indications *could* represent an individual LED, multiple indications could also be represented as a single, multi-color LED or a single LED blinking in a specific interval. The specification leaves that open.
...
Quoted 7 lines
Driver is projected to be exclusive NPEM extended capability manager. It waits up to 1 second after imposing new request, it doesn't verify if controller is busy before write, assuming that mutex lock gives protection from concurrent updates.
Driver is not registered if
_DSMLED management
is available.
IMO we should drop this sentence (more details below).
Quoted 7 lines
NPEM is a PCIe extended capability so it should be registered in
pcie_init_capabilities()but it is not possible due to LED dependency. Parentpci_devicemust be added earlier forled_classdev_register()to be successful. NPEM does not require configuration on kernel side, it is safe to register LED devices later.
I can update this myself, no need to repost just for this, but I think these links are pointless because they're useless except for PCI-SIG members, and I don't want to rely them being permalinks anyway.
A reference like "PCIe r6.1" is universally and permanently meaningful.
Quoted 4 lines
drivers/pci/npem.c new file +4
-0+struct npem { +struct pci_dev *dev; +const struct npem_ops *ops; +struct mutex lock;
drivers/pci/npem.c line 139
Quoted 10 lines
drivers/pci/npem.c new file +10
-0+u16 pos; +u32 supported_indications; +u32 active_indications; + +/* +* Use lazy loading for active_indications to not play with initcalls. +* It is needed to allow _DSM initialization on DELL platforms, where +* ACPI_IPMI must be loaded first. +*/ + unsigned int active_inds_initialized:1;
What's going on here? I hope we can at least move this to the _DSM patch since it seems related to that, not to the NPEM capability. I don't understand the initcall reference or what "lazy loading" means.
Is there some existing ACPI ordering that guarantees ACPI_IPMI happens first? Why do we need some Dell-specific thing here?
What is ACPI_IPMI? I guess it refers to the "acpi_ipmi" module, acpi_ipmi.c?
Quoted 32 lines
drivers/pci/npem.c new file +31
-0+#define DSM_GUID GUID_INIT(0x5d524d9d, 0xfff9, 0x4d4b, 0x8c, 0xb7, 0x74, 0x7e,\ + 0xd5, 0x1e, 0x19, 0x4d) +#define GET_SUPPORTED_STATES_DSM 1 +#define GET_STATE_DSM 2 +#define SET_STATE_DSM 3 + +static const guid_t dsm_guid =DSM_GUID ; + +static bool npem_has_dsm (struct pci_dev *pdev) +{ +acpi_handle handle; + + handle =ACPI_HANDLE (&pdev->dev); +if (!handle) +return false; + +return acpi_check_dsm (handle, &dsm_guid,0x1 , +BIT (GET_SUPPORTED_STATES_DSM ) | +BIT (GET_STATE_DSM ) |BIT (SET_STATE_DSM )); +} +void pci_npem_create (struct pci_dev *dev) +{ +const struct npem_ops *ops = &npem_ops; +int pos =0 , ret; +u32 cap; + +if (!npem_has_dsm (dev)) { + pos =pci_find_ext_capability (dev,PCI_EXT_CAP_ID_NPEM ); +if (pos ==0 ) +return ; +
drivers/pci/npem.c line 342
Quoted 10 lines
drivers/pci/npem.c new file +10
-0+if (pci_read_config_dword (dev, pos +PCI_NPEM_CAP , &cap) !=0 || + (cap &PCI_NPEM_CAP_CAPABLE ) ==0 ) +return ; + }else { +/* +* OS should use the DSM for LED control if it is available +* PCI Firmware Spec r3.3 sec 4.7. +*/ +return ; + }
I know this is sort of a transient state since the next patch adds full _DSM support, but I do think (a) the fact that NPEM will stop working simply because firmware adds _DSM support is unexpected behavior, and (b) npem_has_dsm() and the other ACPI-related stuff would fit better in the next patch. It's a little strange to have them mixed here.
include/uapi/linux/pci_regs.h line 1130
Quoted 8 lines
+++
b/include/uapi/linux/pci_regs.h
...include/uapi/linux/pci_regs.h +5
-0+#define PCI_NPEM_CAP 0x04/* NPEM capability register */ +#define PCI_NPEM_CAP_CAPABLE 0x00000001/* NPEM Capable */ + +#define PCI_NPEM_CTRL 0x08/* NPEM control register */ +#define PCI_NPEM_CTRL_ENABLE 0x00000001/* NPEM Enable */
Spaces instead of tabs after #define, as you did below (mostly), would make the diff prettier.
include/uapi/linux/pci_regs.h line 1157
Quoted 7 lines
include/uapi/linux/pci_regs.h +5
-0+#define PCI_NPEM_CMD_RESET 0x00000002/* NPEM Reset Command */ +#define PCI_NPEM_IND_OK 0x00000004/* NPEM indication OK */ +#define PCI_NPEM_IND_LOCATE 0x00000008/* NPEM indication Locate */ +#define PCI_NPEM_STATUS 0x0c/* NPEM status register */ +#define PCI_NPEM_STATUS_CC 0x00000001/* NPEM Command completed */
Ditto.
Bjorn
Re: [PATCH v6 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
Quoted 11 lines · Bjorn Helgaas
drivers/pci/npem.c line 139
Quoted 6 lines · Mariusz Tkaczyk
drivers/pci/npem.c new file +6
-0+/* +* Use lazy loading for active_indications to not play with initcalls. +* It is needed to allow _DSM initialization on DELL platforms, where +* ACPI_IPMI must be loaded first. +*/ + unsigned int active_inds_initialized:1;What's going on here? I hope we can at least move this to the
_DSMpatch since it seems related to that, not to the NPEM capability. I don't understand the initcall reference or what "lazy loading" means.
In previous iterations of this series, the status of all LEDs was read on PCI device enumeration. That was done so that when user space reads the brightness is sysfs, it gets the correct value. The value is cached, it's not re-read from the register on every brightness read.
(It's not guaranteed that all LEDs are off on enumeration. E.g. boot firmware may have fiddled with them, or the enclosure itself may have turned some of them on by itself, typically the "ok" LED.)
However Stuart reported issues when the _DSM interface is used on Dell servers, because the _DSM requires IPMI drivers to access the NPEM registers. He got a ton of errors when LED status was read on enumeration because that was simply too early. Start of thread:
https://lore.kernel.org/all/[email protected]/
The solution is to read LED status lazily, when brightness is read or written for the first time through sysfs. At that point, IPMI drivers are typically loaded. Stuart reported success with this approach. There is still a possibility that users may see issues if they access brightness before IPMI drivers are loaded. Those drivers may be modules and user space might overzealously try to access brightness before they're loaded. Or user space may prevent them from loading by blacklisting or not installing them. In which case users get to keep the pieces.
We discussed various alternative approaches in the above-linked thread but concluded that this pragmatic solution is the simplest that does the job for all but the most pathological use cases.
We wanted to make this work on Dell servers, but at the same time minimize the contortions that we need to go through to accommodate their quirky implementation.
The code uses lazy initialization of LED status even in the native NPEM case because it would make the code more complex to use early initialization for direct NPEM register access and lazy initialization for _DSM-mediated register access.
Quoted 5 lines
Is there some existing ACPI ordering that guarantees
ACPI_IPMIhappens first? Why do we need some Dell-specific thing here?What is
ACPI_IPMI? I guess it refers to the "acpi_ipmi" module,acpi_ipmi.c?
As it turned out in the above-linked thread, just forcing ACPI_IPMI=y for NPEM is not sufficient because additional (Dell-specific) IPMI drivers need to be loaded as well for NPEM register access to work through _DSM.
Quoted 20 lines
Quoted 20 lines
drivers/pci/npem.c new file +20
-0+void pci_npem_create (struct pci_dev *dev) +{ +const struct npem_ops *ops = &npem_ops; +int pos =0 , ret; +u32 cap; + +if (!npem_has_dsm (dev)) { + pos =pci_find_ext_capability (dev,PCI_EXT_CAP_ID_NPEM ); +if (pos ==0 ) +return ; + +if (pci_read_config_dword (dev, pos +PCI_NPEM_CAP , &cap) !=0 || + (cap &PCI_NPEM_CAP_CAPABLE ) ==0 ) +return ; + }else { +/* +* OS should use the DSM for LED control if it is available +* PCI Firmware Spec r3.3 sec 4.7. +*/ +return ;
Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ }I know this is sort of a transient state since the next patch adds full
_DSMsupport, but I do think (a) the fact that NPEM will stop working simply because firmware adds_DSMsupport is unexpected behavior, and (b)npem_has_dsm()and the other ACPI-related stuff would fit better in the next patch. It's a little strange to have them mixed here.
PCI Firmware Spec r3.3 sec 4.7 says:
"OSPM should use this _DSM when available. If this _DSM is not
available, OSPM should use Native PCIe Enclosure Management (NPEM) or SCSI Enclosure Services (SES) instead, if available."
I realize that a "should" is not a "must", so Linux would in principle be allowed to use direct register access despite presence of the _DSM.
However that doesn't feel safe. If the _DSM is present, I think it's fair to assume that the platform firmware wants to control at least a portion of the LEDs itself. Accessing those LEDs directly, behind the platform firmware's back, may cause issues. Not exposing the LEDs to the user in the _DSM case therefore seems safer.
Which is why the ACPI stuff to query for _DSM presence is already in this patch instead of the succeeding one.
Thanks,
Lukas
Re: [PATCH v6 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
Quoted 18 lines · Lukas Wunner
Quoted 11 lines · Bjorn Helgaas
drivers/pci/npem.c line 139
Quoted 6 lines · Mariusz Tkaczyk
drivers/pci/npem.c new file +6
-0+/* +* Use lazy loading for active_indications to not play with initcalls. +* It is needed to allow _DSM initialization on DELL platforms, where +* ACPI_IPMI must be loaded first. +*/ + unsigned int active_inds_initialized:1;What's going on here? I hope we can at least move this to the
_DSMpatch since it seems related to that, not to the NPEM capability. I don't understand the initcall reference or what "lazy loading" means.In previous iterations of this series, the status of all LEDs was read on PCI device enumeration. That was done so that when user space reads the brightness is sysfs, it gets the correct value. The value is cached, it's not re-read from the register on every brightness read.
Quoted 8 lines
(It's not guaranteed that all LEDs are off on enumeration. E.g. boot firmware may have fiddled with them, or the enclosure itself may have turned some of them on by itself, typically the "ok" LED.)
However Stuart reported issues when the
_DSMinterface is used on Dell servers, because the_DSMrequires IPMI drivers to access the NPEM registers. He got a ton of errors when LED status was read on enumeration because that was simply too early.
The dependency of _DSM on IPMI sounds like a purely ACPI problem. Is there no mechanism in ACPI to express that dependency?
If _DSM claims the function is supported before the IPMI driver is ready, that sounds like a BIOS defect to me.
If we're stuck with this, maybe the comment can be reworded. "Lazy loading" in a paragraph that also mentions initcalls and the "ACPI_IPMI" module makes it sound like we're talking about loading the *module* lazily, not just (IIUC) reading the LED status lazily.
Maybe it could also explicitly say that the GET_STATE_DSM function depends on IPMI.
I'm unhappy that we're getting our arm twisted here. If functionality depends on IPMI, there really needs to be a way for OSPM to manage that dependency. If we're working around a firmware defect, we need to be clear about that.
Quoted 38 lines
Quoted 20 lines
Quoted 20 lines
drivers/pci/npem.c new file +20
-0+void pci_npem_create (struct pci_dev *dev) +{ +const struct npem_ops *ops = &npem_ops; +int pos =0 , ret; +u32 cap; + +if (!npem_has_dsm (dev)) { + pos =pci_find_ext_capability (dev,PCI_EXT_CAP_ID_NPEM ); +if (pos ==0 ) +return ; + +if (pci_read_config_dword (dev, pos +PCI_NPEM_CAP , &cap) !=0 || + (cap &PCI_NPEM_CAP_CAPABLE ) ==0 ) +return ; + }else { +/* +* OS should use the DSM for LED control if it is available +* PCI Firmware Spec r3.3 sec 4.7. +*/ +return ;Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ }I know this is sort of a transient state since the next patch adds full
_DSMsupport, but I do think (a) the fact that NPEM will stop working simply because firmware adds_DSMsupport is unexpected behavior, and (b)npem_has_dsm()and the other ACPI-related stuff would fit better in the next patch. It's a little strange to have them mixed here.PCI Firmware Spec r3.3 sec 4.7 says:
"OSPM should use this
_DSMwhen available. If this_DSMis not
available, OSPM should use Native PCIe Enclosure Management (NPEM) or SCSI Enclosure Services (SES) instead, if available."I realize that a "should" is not a "must", so Linux would in principle be allowed to use direct register access despite presence of the
_DSM.
Quoted 8 lines
However that doesn't feel safe. If the
_DSMis present, I think it's fair to assume that the platform firmware wants to control at least a portion of the LEDs itself. Accessing those LEDs directly, behind the platform firmware's back, may cause issues. Not exposing the LEDs to the user in the_DSMcase therefore seems safer.Which is why the ACPI stuff to query for
_DSMpresence is already in this patch instead of the succeeding one.
The spec is regrettably vague about this, but that assumption isn't unreasonable. It does deserve a more explicit callout in the commit log and probably a dmesg note about why NPEM used to work but no longer does.
Bjorn
Re: [PATCH v6 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
Quoted 41 lines · On Thu, 15 Aug 2024 12:42:48 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 20 lines · Lukas Wunner
Quoted 13 lines · Bjorn Helgaas
drivers/pci/npem.c line 139
Quoted 8 lines · Mariusz Tkaczyk
(quoted diff) +2
-0+ /* + * Use lazy loading for active_indications to not play withinitcalls.
(quoted diff) +1
-0+ * It is needed to allow _DSM initialization on DELLplatforms, where
drivers/pci/npem.c new file +3
-0+* ACPI_IPMI must be loaded first. +*/ + unsigned int active_inds_initialized:1;What's going on here? I hope we can at least move this to the
_DSMpatch since it seems related to that, not to the NPEM capability. I don't understand the initcall reference or what "lazy loading" means.In previous iterations of this series, the status of all LEDs was read on PCI device enumeration. That was done so that when user space reads the brightness is sysfs, it gets the correct value. The value is cached, it's not re-read from the register on every brightness read.
Quoted 8 lines
(It's not guaranteed that all LEDs are off on enumeration. E.g. boot firmware may have fiddled with them, or the enclosure itself may have turned some of them on by itself, typically the "ok" LED.)
However Stuart reported issues when the
_DSMinterface is used on Dell servers, because the_DSMrequires IPMI drivers to access the NPEM registers. He got a ton of errors when LED status was read on enumeration because that was simply too early.The dependency of
_DSMon IPMI sounds like a purely ACPI problem. Is there no mechanism in ACPI to express that dependency?If
_DSMclaims the function is supported before the IPMI driver is ready, that sounds like a BIOS defect to me.If we're stuck with this, maybe the comment can be reworded. "Lazy loading" in a paragraph that also mentions initcalls and the "
ACPI_IPMI" module makes it sound like we're talking about loading the *module* lazily, not just (IIUC) reading the LED status lazily.
Quoted 7 lines
Maybe it could also explicitly say that the
GET_STATE_DSMfunction depends on IPMI.I'm unhappy that we're getting our arm twisted here. If functionality depends on IPMI, there really needs to be a way for OSPM to manage that dependency. If we're working around a firmware defect, we need to be clear about that.
Hi,
I will move active_inds_initialized:1 to DSM commit and I will add better justification. For NPEM commit, get_active_indications() will be called once in pci_npem_init() to avoid referring _DSM specific issues in NPEM commit.
Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 24 lines
drivers/pci/npem.c new file +8
-0+void pci_npem_create (struct pci_dev *dev) +{ +const struct npem_ops *ops = &npem_ops; +int pos =0 , ret; +u32 cap; + +if (!npem_has_dsm (dev)) { + pos = pci_find_ext_capability(dev,
PCI_EXT_CAP_ID_NPEM);drivers/pci/npem.c new file +4
-0+if (pos ==0 ) +return ; + + if (pci_read_config_dword(dev, pos + PCI_NPEM_CAP,&cap) != 0 ||
drivers/pci/npem.c new file +5
-0+ (cap & PCI_NPEM_CAP_CAPABLE) == 0) +return ; + }else { +/* +* OS should use the DSM for LED control if it is available
drivers/pci/npem.c new file +3
-0+* PCI Firmware Spec r3.3 sec 4.7. +*/ +return ;Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ }I know this is sort of a transient state since the next patch adds full
_DSMsupport, but I do think (a) the fact that NPEM will stop working simply because firmware adds_DSMsupport is unexpected behavior, and (b)npem_has_dsm()and the other ACPI-related stuff would fit better in the next patch. It's a little strange to have them mixed here.PCI Firmware Spec r3.3 sec 4.7 says:
"OSPM should use this
_DSMwhen available. If this_DSMis not
available, OSPM should use Native PCIe Enclosure Management (NPEM) or SCSI Enclosure Services (SES) instead, if available."I realize that a "should" is not a "must", so Linux would in principle be allowed to use direct register access despite presence of the
_DSM.However that doesn't feel safe. If the
_DSMis present, I think it's fair to assume that the platform firmware wants to control at least a portion of the LEDs itself. Accessing those LEDs directly, behind the platform firmware's back, may cause issues. Not exposing the LEDs to the user in the_DSMcase therefore seems safer.
Quoted 8 lines
Quoted 2 lines
Which is why the ACPI stuff to query for
_DSMpresence is already in this patch instead of the succeeding one.The spec is regrettably vague about this, but that assumption isn't unreasonable. It does deserve a more explicit callout in the commit log and probably a dmesg note about why NPEM used to work but no longer does.
In fact, there is theoretical case that after firmware update DSM is no longer available and NPEM is chosen. Given that, I will log chosen backed instead of trying to predict a change. It is easier to implement it this way. User can compare working/not-working dmesg logs to see a difference so printing backend used is enough I think.
Thanks,
Mariusz
Re: [PATCH v6 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
In reply to Bjorn HelgaasQuoted 2 lines · Bjorn Helgaas
The dependency of
_DSMon IPMI sounds like a purely ACPI problem. Is there no mechanism in ACPI to express that dependency?
Unfortunately there doesn't seem to be one. :(
Quoted 8 lines
If
_DSMclaims the function is supported before the IPMI driver is ready, that sounds like a BIOS defect to me.If we're stuck with this, maybe the comment can be reworded. "Lazy loading" in a paragraph that also mentions initcalls and the "
ACPI_IPMI" module makes it sound like we're talking about loading the *module* lazily, not just (IIUC) reading the LED status lazily.
Quoted 7 lines
Maybe it could also explicitly say that the
GET_STATE_DSMfunction depends on IPMI.I'm unhappy that we're getting our arm twisted here. If functionality depends on IPMI, there really needs to be a way for OSPM to manage that dependency. If we're working around a firmware defect, we need to be clear about that.
AFAICS lazy initialization of active indications was architected such that it is retried on every LED access until it succeeds:
npem->active_inds_initialized is only set to true once npem->ops->get_active_indications() returns successfully. I'm assuming that the DSM method fails as it should on inaccessibility of the IPMI OpRegion.
So users may see errors in dmesg when they access LEDs if IPMI drivers have not been loaded yet, but once they're loaded, those errors will go away and LED access should start working flawlessly.
This way of lazily initializing the cached active_indications bit mask doesn't cost us much as far as code complexity is concerned, but should make things work 99.999% of the time on quirky platforms.
Thanks,
Lukas
PATCH v6 3/3 PCI/NPEM: Add _DSM PCIe SSD status LED management
In reply to Mariusz TkaczykDevice Specific Method PCIe SSD Status LED Management (_DSM) defined in PCI Firmware Spec r3.3 sec 4.7 provides a way to manage LEDs via ACPI.
The design is similar to NPEM defined in PCIe Base Specification r6.1 sec 6.28:
- both standards are indication oriented,
- _DSM supported bits are corresponding to NPEM capability register bits
- _DSM control bits are corresponding to NPEM control register bits.
_DSM does not support enclosure specific indications and special NPEM commands NPEM_ENABLE and NPEM_RESET.
_DSM is implemented as a second op in NPEM driver. The standard used in background is not presented to user. The interface is accessed same as NPEM.
According to spec, _DSM has higher priority and availability of _DSM in not limited to devices with NPEM support. It is followed in implementation.
--- Documentation/ABI/testing/sysfs-bus-pci | 9 ++ drivers/pci/npem.c | 147 +++++++++++++++++++++++- 2 files changed, 153 insertions(+), 3 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +9 -0
@@ -563,3 +563,12 @@ Description: indication is usually presented as one or two LEDs blinking at 4 Hz frequency: https://en.wikipedia.org/wiki/International_Blinking_Pattern_Interpretation + + PCI Firmware Specification r3.3 sec 4.7 defines a DSM interface + to facilitate shared access by operating system and platform + firmware to a device's NPEM registers. The kernel will use + this DSM interface where available, instead of accessing NPEM + registers directly. The DSM interface does not support the + enclosure-specific indications "specific0" to "specific7", + hence the corresponding LED class devices are unavailable if + the DSM interface is used.
drivers/pci/npem.c +144 -3
173 lines not on this page · raw message
Signature · 1 line
2.35.3
[+cc Lee, linux-leds for comment on using the LED subsystem as described in patch 2/3; would be nice to have a reviewed-by or ack for this. Thread at
https://lore.kernel.org/r/[email protected]]
Quoted 14 lines · Mariusz Tkaczyk
Patchset is named as PCIe Enclosure LED Management because it adds two features:
- Native PCIe Enclosure Management (NPEM)
- PCIe SSD Status LED Management (DSM)Both are pattern oriented standards, they tell which "indication" should blink. It doesn't control physical LED or pattern visualization.
Overall, driver is simple but it was not simple to fit it into interfaces we have in kernel (We considered leds and enclosure interfaces). It reuses leds interface, this approach seems to be the best because:
- leds are actively maintained, no new interface added.
- leds do not require any extensions, enclosure needs to be adjusted first.
Quoted 6 lines
There are trade-offs:
- "brightness" is the name of sysfs file to control led. It is not natural to use brightness to set patterns, that is why multiple led devices are created (one per indication);
- Update of one led may affect other leds, led triggers may not work as expected.
v1 at https://lore.kernel.org/r/[email protected]
Quoted 4 lines
Changes from v1:
- Renamed "pattern" to indication.
- DSM support added.
- fixed nits reported by Bjorn.
v2 at https://lore.kernel.org/r/[email protected]
Quoted 5 lines
Changes from v2:
- Introduce lazy loading to allow DELL_DSMquirks to work, reported by Stuart.
- leds class initcall moved up inMakefile, proposed by Dan.
- fix other nits reported by Dan and Iipo.
v3 at https://lore.kernel.org/r/[email protected]
Quoted 5 lines
Changes from v3:
- Remove unnecessary packed attr.
- Fix doc issue reported by lkp.
- Fixread_poll_timeout()error handling reported by Iipo.
- Minor fixes reported by Christoph.
v4 at https://lore.kernel.org/r/[email protected]
Quoted 3 lines
Changes from v4:
- Use 0 / 1 instead of LED_OFF/LED_ON, suggested by Marek.
- Documentation added, suggested by Bjorn.
v5 at https://lore.kernel.org/r/[email protected]
Quoted 8 lines
Change from v5:
- Remove unnecessary_packed, reported by Christoph.
- Changed "led" to "LED" and other typos suggested by Randy.Suggested-by Lukas Wunner <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Ilpo Järvinen <[email protected]>Tested-by Stuart Hayes <[email protected]>
Evidently you intend these tags to be applied to each patch, but b4 doesn't distribute tags from the cover letter across each individual patch.
You included Christoph's Reviewed-by directly in patches 1 and 2, but not Ilpo's. I didn't dig through the previous postings to verify all this.
Quoted 35 lines
Cc Arnd Bergmann <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Dan Williams <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc Ilpo Jarvinen <[email protected]>Cc Lukas Wunner <[email protected]>Cc Keith Busch <[email protected]>Cc Marek Behun <[email protected]>Cc Pavel Machek <[email protected]>Cc Randy Dunlap <[email protected]>Cc Andy Shevchenko <[email protected]>Cc Stuart Hayes <[email protected]>Mariusz Tkaczyk (3):
leds: Init leds class earlier
PCI/NPEM: Add Native PCIe Enclosure Management support PCI/NPEM: Add_DSMPCIe SSD status LED managementDocumentation/ABI/testing/sysfs-bus-pci | 72 +++ drivers/Makefile | 4 +- drivers/pci/Kconfig | 9 + drivers/pci/Makefile | 1 + drivers/pci/npem.c | 590 ++++++++++++++++++++++++ drivers/pci/pci.h | 8 + drivers/pci/probe.c | 2 + drivers/pci/remove.c | 2 + include/linux/pci.h | 3 + include/uapi/linux/pci_regs.h | 35 ++ 10 files changed, 725 insertions(+), 1 deletion(-) create mode 100644 drivers/pci/npem.cSignature · 1 line
2.35.3
Lee Jones
maintainerQuoted 4 lines · Bjorn Helgaas
[+cc Lee, linux-leds for comment on using the LED subsystem as described in patch 2/3; would be nice to have a reviewed-by or ack for this. Thread at
https://lore.kernel.org/r/[email protected]]
I usually say things like "if you want to act as an X-device, author an X-driver and put it in X-subsystem". However it looks like the LED class is already heavily abused in similar ways as this, so there's not a great deal I can say about it.
Signature · 1 line
Lee Jones [李琼斯]