PCIe Enclosure LED Management
v3 cover PCIe Enclosure LED Management
Patchset is named as PCIe Enclosure LED Management because it adds two features:
- Native PCIe Enclosure Management (NPEM)
- PCIe SSD Status LED Management (DSM)
Both are pattern oriented standards, they tell which "indication" should blink. It doesn't control physical LED or pattern visualization.
Overall, driver is simple but it was not simple to fit it into interfaces we have in kernel (We considered leds and enclosure interfaces). It reuses leds interface, this approach seems to be the best because:
- leds are actively maintained, no new interface added.
- leds do not require any extensions, enclosure needs to be adjusted first.
There are trade-offs:
- "brightness" is the name of sysfs file to control led. It is not natural to use brightness to set patterns, that is why multiple led devices are created (one per indication);
- Update of one led may affect other leds, led triggers may not work as expected.
It was tested with _DSM but I do some minor updates after review. Stuart please retest.
Changes from v1:
- Renamed "pattern" to indication.
- DSM support added.
- fixed nits reported by Bjorn.
Changes from v2:
- Introduce lazy loading to allow DELL _DSM quirks to work, reported by Stuart.
- leds class initcall moved up in Makefile, proposed by Dan.
- fix other nits reported by Dan and Iipo.
No feedback received from leds maintainers. No updates in Kconifg.
Mariusz Tkaczyk (3):
leds: Init leds class earlier
PCI/NPEM: Add Native PCIe Enclosure Management support PCI/NPEM: Add _DSM PCIe SSD status LED management
drivers/Makefile | 4 +- drivers/pci/Kconfig | 9 + drivers/pci/Makefile | 1 + drivers/pci/npem.c | 588 ++++++++++++++++++++++++++++++++++ drivers/pci/pci.h | 8 + drivers/pci/probe.c | 2 + drivers/pci/remove.c | 2 + include/linux/pci.h | 3 + include/uapi/linux/pci_regs.h | 35 ++ 9 files changed, 651 insertions(+), 1 deletion(-) create mode 100644 drivers/pci/npem.c
Signature · 1 line
2.35.3
PATCH v3 1/3 leds: Init leds class earlier
NPEM driver will require leds class, there is an init-order conflict. Make sure that LEDs initialization happens first and add comment.
--- drivers/Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/Makefile +3 -1
@@ -17,6 +17,9 @@ obj-$(CONFIG_PINCTRL) += pinctrl/ obj-$(CONFIG_GPIOLIB) += gpio/ obj-y += pwm/ +# LEDs must come before PCI, it is needed by NPEM driver +obj-y += leds/ + obj-y += pci/ obj-$(CONFIG_PARISC) += parisc/
@@ -130,7 +133,6 @@ obj-$(CONFIG_CPU_IDLE) += cpuidle/ obj-y += mmc/ obj-y += ufs/ obj-$(CONFIG_MEMSTICK) += memstick/ -obj-y += leds/ obj-$(CONFIG_INFINIBAND) += infiniband/ obj-y += firmware/ obj-$(CONFIG_CRYPTO) += crypto/
Signature · 1 line
2.35.3
Re: [PATCH v3 1/3] leds: Init leds class earlier
Looks good:
PATCH v3 2/3 PCI/NPEM: Add Native PCIe Enclosure Management support
In reply to Mariusz TkaczykNative PCIe Enclosure Management (NPEM, PCIe r6.1 sec 6.28) allows managing LED in storage enclosures. NPEM is indication oriented and it does not give direct access to LED. Although each of the indications *could* represent an individual LED, multiple indications could also be represented as a single, multi-color LED or a single LED blinking in a specific interval. The specification leaves that open.
Each enabled indication (capability register bit on) is represented as a ledclass_dev which can be controlled through sysfs. For every ledclass device only 2 brightness states are allowed: LED_ON (1) or LED_OFF (0). It is corresponding to NPEM control register (Indication bit on/off).
Ledclass devices appear in sysfs as child devices (subdirectory) of PCI device which has an NPEM Extended Capability and indication is enabled in NPEM capability register. For example, these are leds created for pcieport "10000:02:05.0" on my setup:
leds/
├── 10000:02:05.0:enclosure:fail ├── 10000:02:05.0:enclosure:locate ├── 10000:02:05.0:enclosure:ok └── 10000:02:05.0:enclosure:rebuild
They can be also found in "/sys/class/leds" directory. Parent PCIe device bdf is used to guarantee uniqueness across leds subsystem.
To enable/disable fail indication "brightness" file can be edited:
echo 1 > ./leds/10000:02:05.0:enclosure:fail/brightness echo 0 > ./leds/10000:02:05.0:enclosure:fail/brightness
PCIe r6.1, sec 7.9.19.2 defines the possible indications.
Multiple indications for same parent PCIe device can conflict and hardware may update them when processing new request. To avoid issues, driver refresh all indications by reading back control register.
Driver is projected to be exclusive NPEM extended capability manager. It waits up to 1 second after imposing new request, it doesn't verify if controller is busy before write, assuming that mutex lock gives protection from concurrent updates. Driver is not registered if _DSM LED management is available.
NPEM is a PCIe extended capability so it should be registered in pcie_init_capabilities() but it is not possible due to LED dependency. Parent pci_device must be added earlier for led_classdev_register() to be successful. NPEM does not require configuration on kernel side, it is safe to register LED devices later.
--- drivers/pci/Kconfig | 9 + drivers/pci/Makefile | 1 + drivers/pci/npem.c | 447 ++++++++++++++++++++++++++++++++++ drivers/pci/pci.h | 8 + drivers/pci/probe.c | 2 + drivers/pci/remove.c | 2 + include/linux/pci.h | 3 + include/uapi/linux/pci_regs.h | 35 +++ 8 files changed, 507 insertions(+) create mode 100644 drivers/pci/npem.c
drivers/pci/Kconfig +9 -0
@@ -143,6 +143,15 @@ config PCI_IOV If unsure, say N. +config PCI_NPEM + bool "Native PCIe Enclosure Management" + depends on LEDS_CLASS=y + help + Support for Native PCIe Enclosure Management. It allows managing LED + indications in storage enclosures. Enclosure must support following + indications: OK, Locate, Fail, Rebuild, other indications are + optional. + config PCI_PRI bool "PCI PRI support" select PCI_ATS
drivers/pci/Makefile +1 -0
@@ -34,6 +34,7 @@ obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o obj-$(CONFIG_VGA_ARB) += vgaarb.o obj-$(CONFIG_PCI_DOE) += doe.o obj-$(CONFIG_PCI_DYNAMIC_OF_NODES) += of_property.o +obj-$(CONFIG_PCI_NPEM) += npem.o # Endpoint library must be initialized before its users obj-$(CONFIG_PCI_ENDPOINT) += endpoint/
drivers/pci/npem.c new file +447 -0
447 lines not on this page · raw message
drivers/pci/pci.h +8 -0
@@ -333,6 +333,14 @@ static inline void pci_doe_destroy(struct pci_dev *pdev) { }static inline void pci_doe_disconnected (struct pci_dev *pdev) { }#endif +#ifdef CONFIG_PCI_NPEM +void pci_npem_create (struct pci_dev *dev); +void pci_npem_remove (struct pci_dev *dev); +#else +static inline void pci_npem_create (struct pci_dev *dev) { } +static inline void pci_npem_remove (struct pci_dev *dev) { } +#endif +/** * pci_dev_set_io_state - Set the new error state if possible. *
drivers/pci/probe.c +2 -0
@@ -2582,6 +2582,8 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus) dev->match_driver = false; ret =device_add (&dev->dev);WARN_ON (ret <0 ); + +pci_npem_create (dev); }struct pci_dev *pci_scan_single_device (struct pci_bus *bus,int devfn)
drivers/pci/remove.c +2 -0
@@ -33,6 +33,8 @@ static void pci_destroy_dev(struct pci_dev *dev)if (!dev->dev.kobj.parent)return ; +pci_npem_remove (dev); +device_del (&dev->dev);down_write (&pci_bus_sem);
include/linux/pci.h +3 -0
@@ -517,6 +517,9 @@ struct pci_dev {#endif #ifdef CONFIG_PCI_DOE struct xarray doe_mbs;/* Data Object Exchange mailboxes */ +#endif +#ifdef CONFIG_PCI_NPEM +struct npem *npem;/* Native PCIe Enclosure Management */ #endif u16 acs_cap;/* ACS Capability offset */ phys_addr_t rom;/* Physical address if not from BAR */
include/uapi/linux/pci_regs.h +35 -0
@@ -740,6 +740,7 @@#define PCI_EXT_CAP_ID_DVSEC 0x23/* Designated Vendor-Specific */ #define PCI_EXT_CAP_ID_DLF 0x25/* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26/* Physical Layer 16.0 GT/s */ +#define PCI_EXT_CAP_ID_NPEM 0x29/* Native PCIe Enclosure Management */ #define PCI_EXT_CAP_ID_PL_32GT 0x2A/* Physical Layer 32.0 GT/s */ #define PCI_EXT_CAP_ID_DOE 0x2E/* Data Object Exchange */ #define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_DOE
@@ -1121,6 +1122,40 @@#define PCI_PL_16GT_LE_CTRL_USP_TX_PRESET_MASK 0x000000F0#define PCI_PL_16GT_LE_CTRL_USP_TX_PRESET_SHIFT 4 +/* Native PCIe Enclosure Management */ +#define PCI_NPEM_CAP 0x04/* NPEM capability register */ +#define PCI_NPEM_CAP_CAPABLE 0x00000001/* NPEM Capable */ + +#define PCI_NPEM_CTRL 0x08/* NPEM control register */ +#define PCI_NPEM_CTRL_ENABLE 0x00000001/* NPEM Enable */ + +/* +* Native PCIe Enclosure Management indication bits and Reset command bit +* are corresponding for capability and control registers. +*/ +#define PCI_NPEM_CMD_RESET 0x00000002/* NPEM Reset Command */ +#define PCI_NPEM_IND_OK 0x00000004/* NPEM indication OK */ +#define PCI_NPEM_IND_LOCATE 0x00000008/* NPEM indication Locate */ +#define PCI_NPEM_IND_FAIL 0x00000010/* NPEM indication Fail */ +#define PCI_NPEM_IND_REBUILD 0x00000020/* NPEM indication Rebuild */ +#define PCI_NPEM_IND_PFA 0x00000040/* NPEM indication Predicted Failure Analysis */ +#define PCI_NPEM_IND_HOTSPARE 0x00000080/* NPEM indication Hot Spare */ +#define PCI_NPEM_IND_ICA 0x00000100/* NPEM indication In Critical Array */ +#define PCI_NPEM_IND_IFA 0x00000200/* NPEM indication In Failed Array */ +#define PCI_NPEM_IND_IDT 0x00000400/* NPEM indication Invalid Device Type */ +#define PCI_NPEM_IND_DISABLED 0x00000800/* NPEM indication Disabled */ +#define PCI_NPEM_IND_SPEC_0 0x01000000 +#define PCI_NPEM_IND_SPEC_1 0x02000000 +#define PCI_NPEM_IND_SPEC_2 0x04000000 +#define PCI_NPEM_IND_SPEC_3 0x08000000 +#define PCI_NPEM_IND_SPEC_4 0x10000000 +#define PCI_NPEM_IND_SPEC_5 0x20000000 +#define PCI_NPEM_IND_SPEC_6 0x40000000 +#define PCI_NPEM_IND_SPEC_7 0x80000000 + +#define PCI_NPEM_STATUS 0x0c/* NPEM status register */ +#define PCI_NPEM_STATUS_CC 0x00000001/* NPEM Command completed */ +/* Data Object Exchange */ #define PCI_DOE_CAP 0x04/* DOE Capabilities Register */ #define PCI_DOE_CAP_INT_SUP 0x00000001/* Interrupt Support */
Signature · 1 line
2.35.3
Re: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
drivers/pci/npem.c line 59
Quoted 2 lines
drivers/pci/npem.c new file +2
-0+#define for_each_indication (ind, inds) \ + for (ind = inds; ind->bit; ind++)
I find this more confusing than just seeing the actual for loop in the two places using it, but that my just be personal preference.
/*
* Driver has internal list of supported indications. Ideally, driver should not
s/drivers/the driver/ both times?
Quoted 4 lines
drivers/pci/npem.c new file +4
-0+if (!npem_has_dsm (dev)) { + pos =pci_find_ext_capability (dev,PCI_EXT_CAP_ID_NPEM ); +if (pos ==0 ) +return ;
drivers/pci/npem.c line 440
Quoted 10 lines
drivers/pci/npem.c new file +10
-0+ +if (pci_read_config_dword (dev, pos +PCI_NPEM_CAP , &cap) !=0 || + (cap &PCI_NPEM_CAP_CAPABLE ) ==0 ) +return ; + }else { +/* +* OS should use the DSM for LED control if it is available +* PCI Firmware Spec r3.3 sec 4.7. +*/ +return ;
If you just did a:
if (npem_has_dsm(dev)) return;
we'd save a level of identation for the !dsm case an make the code a bit easier to read. Also I think "OS" above should be "The OS".
Otherwise looks good:
Re: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
Quoted 7 lines · On Fri, 5 Jul 2024 06:02:53 -0700 Christoph Hellwig <[email protected]> wrote:
If you just did a:
if (npem_has_dsm(dev)) return;we'd save a level of identation for the !dsm case an make the code a bit easier to read. Also I think "OS" above should be "The OS".
Thanks for review!
I did it way because in next patch I'm adding DSM. With that I have less changes in second patch.
Mariusz
Re: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
Quoted 2 lines · Mariusz Tkaczyk
I did it way because in next patch I'm adding DSM. With that I have less changes in second patch.
Ok, never mind then.
kernel test robot
build botRe: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
In reply to Mariusz Tkaczykkernel test robot noticed the following build warnings:
Hi Mariusz,
kernel test robot noticed the following build warnings:
[auto build test WARNING on pci/next]
[also build test WARNING on pci/for-linus linus/master v6.10-rc6 next-20240703] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Mariusz-Tkaczyk/leds-Init-leds-class-earlier/20240706-015012 base: https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next patch link: https://lore.kernel.org/r/20240705125436.26057-3-mariusz.tkaczyk%40linux.intel.com patch subject: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support config: microblaze-allyesconfig (https://download.01.org/0day-ci/archive/20240706/[email protected]/config) compiler: microblaze-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240706/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> drivers/pci/npem.c:143: warning: Function parameter or struct member 'active_inds_initialized' not described in 'npem' >> drivers/pci/npem.c:143: warning: Excess struct member 'flags' description in 'npem'
vim +143 drivers/pci/npem.c
111
112 /**
113 * struct npem - NPEM device properties
114 * @dev: PCIe device this driver is attached to
115 * @ops: Backend specific callbacks
116 * @lock: serialized accessing npem device from multiple LED devices
117 * @pos: NPEM backed only, NPEM capability offset
118 * @supported_indications: bit mask of supported indications
119 * non-indication and reserved bits are cleared
120 * @active_indications: bit mask of active indications
121 * non-indication and reserved bits are cleared
122 * @flags: npem device control flags
123 * @led_cnt: Supported LEDs count
124 * @leds: supported LEDs
125 */
126 struct npem {
127 struct pci_dev *dev;
128 const struct npem_ops *ops;
129 struct mutex lock;
130 u16 pos;
131 u32 supported_indications;
132 u32 active_indications;
133
134 /*
135 * Use lazy loading for active_indications to not play with initcalls.
136 * It is needed to allow _DSM initialization on DELL platforms, where
137 * ACPI_IPMI module is needed.
138 */
139 unsigned int active_inds_initialized:1;
140
141 int led_cnt;
142 struct npem_led leds[];
> 143 };
144
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
In reply to Mariusz TkaczykQuoted 49 lines · Mariusz Tkaczyk
Native PCIe Enclosure Management (NPEM, PCIe r6.1 sec 6.28) allows managing LED in storage enclosures. NPEM is indication oriented and it does not give direct access to LED. Although each of the indications *could* represent an individual LED, multiple indications could also be represented as a single, multi-color LED or a single LED blinking in a specific interval. The specification leaves that open.
Each enabled indication (capability register bit on) is represented as a
ledclass_devwhich can be controlled through sysfs. For every ledclass device only 2 brightness states are allowed:LED_ON(1) orLED_OFF(0). It is corresponding to NPEM control register (Indication bit on/off).Ledclass devices appear in sysfs as child devices (subdirectory) of PCI device which has an NPEM Extended Capability and indication is enabled in NPEM capability register. For example, these are leds created for pcieport "10000:02:05.0" on my setup:
leds/
├── 10000:02:05.0:enclosure:fail ├── 10000:02:05.0:enclosure:locate ├── 10000:02:05.0:enclosure:ok └── 10000:02:05.0:enclosure:rebuildThey can be also found in "
/sys/class/leds" directory. Parent PCIe device bdf is used to guarantee uniqueness across leds subsystem.To enable/disable fail indication "brightness" file can be edited:
echo 1 > ./leds/10000:02:05.0:enclosure:fail/brightness echo 0 > ./leds/10000:02:05.0:enclosure:fail/brightnessPCIe r6.1, sec 7.9.19.2 defines the possible indications.
Multiple indications for same parent PCIe device can conflict and hardware may update them when processing new request. To avoid issues, driver refresh all indications by reading back control register.
Driver is projected to be exclusive NPEM extended capability manager. It waits up to 1 second after imposing new request, it doesn't verify if controller is busy before write, assuming that mutex lock gives protection from concurrent updates. Driver is not registered if
_DSMLED management is available.NPEM is a PCIe extended capability so it should be registered in
pcie_init_capabilities()but it is not possible due to LED dependency. Parentpci_devicemust be added earlier forled_classdev_register()to be successful. NPEM does not require configuration on kernel side, it is safe to register LED devices later.
Quoted 3 lines
Suggested-by Lukas Wunner <[email protected]>Signed-off-by Mariusz Tkaczyk <[email protected]>
Looks to be in quite good shape already, one comment below I think should be addressed before this is ready to go.
Quoted 19 lines
drivers/pci/npem.c new file +19
-0+static int npem_set_active_indications (struct npem *npem,u32 inds) +{ +int ctrl, ret, ret_val; +u32 cc_status; + +lockdep_assert_held (&npem->lock); + +/* This bit is always required */ + ctrl = inds |PCI_NPEM_CTRL_ENABLE ; + + ret =npem_write_ctrl (npem, ctrl); +if (ret) +return ret; + +/* +* For the case where a NPEM command has not completed immediately, +* it is recommended that software not continuously “spin” on polling +* the status register, but rather poll under interrupt at a reduced +* rate; for example at 10 ms intervals.
drivers/pci/npem.c line 210
Quoted 10 lines
drivers/pci/npem.c new file +10
-0+* +* PCIe r6.1 sec 6.28 "Implementation Note: Software Polling of NPEM +* Command Completed" +*/ + ret =read_poll_timeout (npem_read_reg, ret_val, + ret_val || (cc_status &PCI_NPEM_STATUS_CC ), +10 *USEC_PER_MSEC ,USEC_PER_SEC , false, npem, +PCI_NPEM_STATUS , &cc_status); +if (ret) +return ret_val;
Will this work as intended?
If ret_val gets set, cond in read_poll_timeout() is true and it returns 0 so the return branch is not taken.
Also, when read_poll_timeout() times out, ret_val might not be non-zero.
Quoted 14 lines
drivers/pci/npem.c new file +14
-0+ +/* +* All writes to control register, including writes that do not change +* the register value, are NPEM commands and should eventually result +* in a command completion indication in the NPEM Status Register. +* +* PCIe Base Specification r6.1 sec 7.9.19.3 +* +* Register may not be updated, or other conflicting bits may be +* cleared. Spec is not strict here. Read NPEM Control register after +* write to keep cache in-sync. +*/ +return npem_get_active_indications (npem, &npem->active_indications); +}
Signature · 1 line
i.
Re: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
Quoted 32 lines · Ilpo Järvinen
Quoted 19 lines
drivers/pci/npem.c new file +19
-0+static int npem_set_active_indications (struct npem *npem,u32 inds) +{ +int ctrl, ret, ret_val; +u32 cc_status; + +lockdep_assert_held (&npem->lock); + +/* This bit is always required */ + ctrl = inds |PCI_NPEM_CTRL_ENABLE ; + + ret =npem_write_ctrl (npem, ctrl); +if (ret) +return ret; + +/* +* For the case where a NPEM command has not completed immediately, +* it is recommended that software not continuously ???spin??? on polling +* the status register, but rather poll under interrupt at a reduced +* rate; for example at 10 ms intervals. drivers/pci/npem.c line 210
Quoted 10 lines
drivers/pci/npem.c new file +10
-0+* +* PCIe r6.1 sec 6.28 "Implementation Note: Software Polling of NPEM +* Command Completed" +*/ + ret =read_poll_timeout (npem_read_reg, ret_val, + ret_val || (cc_status &PCI_NPEM_STATUS_CC ), +10 *USEC_PER_MSEC ,USEC_PER_SEC , false, npem, +PCI_NPEM_STATUS , &cc_status); +if (ret) +return ret_val;Will this work as intended?
Quoted 4 lines
If
ret_valgets set, cond inread_poll_timeout()is true and it returns 0 so the return branch is not taken.Also, when
read_poll_timeout()times out,ret_valmight not be non-zero.
Hm, it seems to me that just having two if-clauses should fix that.
+ ret = read_poll_timeout(npem_read_reg, ret_val,
+ ret_val || (cc_status & PCI_NPEM_STATUS_CC),
+ 10 * USEC_PER_MSEC, USEC_PER_SEC, false, npem,
+ PCI_NPEM_STATUS, &cc_status);
+ if (ret)
+ return ret;
+ if (ret_val)
+ return ret_val;
Does this look correct to you?
Thanks,
Lukas
Re: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
In reply to Ilpo JärvinenQuoted 95 lines · On Mon, 8 Jul 2024 14:33:34 +0300 (EEST) Ilpo Järvinen…
Quoted 49 lines · Mariusz Tkaczyk
Native PCIe Enclosure Management (NPEM, PCIe r6.1 sec 6.28) allows managing LED in storage enclosures. NPEM is indication oriented and it does not give direct access to LED. Although each of the indications *could* represent an individual LED, multiple indications could also be represented as a single, multi-color LED or a single LED blinking in a specific interval. The specification leaves that open.
Each enabled indication (capability register bit on) is represented as a
ledclass_devwhich can be controlled through sysfs. For every ledclass device only 2 brightness states are allowed:LED_ON(1) orLED_OFF(0). It is corresponding to NPEM control register (Indication bit on/off).Ledclass devices appear in sysfs as child devices (subdirectory) of PCI device which has an NPEM Extended Capability and indication is enabled in NPEM capability register. For example, these are leds created for pcieport "10000:02:05.0" on my setup:
leds/
├── 10000:02:05.0:enclosure:fail ├── 10000:02:05.0:enclosure:locate ├── 10000:02:05.0:enclosure:ok └── 10000:02:05.0:enclosure:rebuildThey can be also found in "
/sys/class/leds" directory. Parent PCIe device bdf is used to guarantee uniqueness across leds subsystem.To enable/disable fail indication "brightness" file can be edited:
echo 1 > ./leds/10000:02:05.0:enclosure:fail/brightness echo 0 > ./leds/10000:02:05.0:enclosure:fail/brightnessPCIe r6.1, sec 7.9.19.2 defines the possible indications.
Multiple indications for same parent PCIe device can conflict and hardware may update them when processing new request. To avoid issues, driver refresh all indications by reading back control register.
Driver is projected to be exclusive NPEM extended capability manager. It waits up to 1 second after imposing new request, it doesn't verify if controller is busy before write, assuming that mutex lock gives protection from concurrent updates. Driver is not registered if
_DSMLED management is available.NPEM is a PCIe extended capability so it should be registered in
pcie_init_capabilities()but it is not possible due to LED dependency. Parentpci_devicemust be added earlier forled_classdev_register()to be successful. NPEM does not require configuration on kernel side, it is safe to register LED devices later.Quoted 3 lines
Suggested-by Lukas Wunner <[email protected]>Signed-off-by Mariusz Tkaczyk <[email protected]>Looks to be in quite good shape already, one comment below I think should be addressed before this is ready to go.
Quoted 31 lines
drivers/pci/npem.c new file +17
-0+static int npem_set_active_indications (struct npem *npem,u32 inds) +{ +int ctrl, ret, ret_val; +u32 cc_status; + +lockdep_assert_held (&npem->lock); + +/* This bit is always required */ + ctrl = inds |PCI_NPEM_CTRL_ENABLE ; + + ret =npem_write_ctrl (npem, ctrl); +if (ret) +return ret; + +/* +* For the case where a NPEM command has not completed immediately, +* it is recommended that software not continuously “spin” on polling
(quoted diff) +1
-0+ * the status register, but rather poll under interrupt at areduced
drivers/pci/npem.c new file +3
-0+* rate; for example at 10 ms intervals. +* +* PCIe r6.1 sec 6.28 "Implementation Note: Software Polling of NPEM
drivers/pci/npem.c new file +4
-0+* Command Completed" +*/ + ret = read_poll_timeout(npem_read_reg, ret_val, + ret_val || (cc_status &
PCI_NPEM_STATUS_CC),(quoted diff) +1
-0+ 10 * USEC_PER_MSEC, USEC_PER_SEC, false,npem,
drivers/pci/npem.c line 210
Quoted 3 lines
drivers/pci/npem.c new file +3
-0+ PCI_NPEM_STATUS, &cc_status); +if (ret) +return ret_val;Will this work as intended?
Quoted 5 lines
If
ret_valgets set, cond inread_poll_timeout()is true and it returns 0 so the return branch is not taken.Also, when
read_poll_timeout()times out,ret_valmight not be non-zero.
Yes, it is good catch thanks! What about?
if (ret) return ret; if (ret_val) return ret_val;
If ret is set it means that it times out- we should return that to caller.
If ret_val is set it means that we received error in npem_read_reg()- we should return that (device probably is unreachable).
If read_val is set then we are less interested in ret because error from npem_read_reg() function is more critical, so it is "acceptable" to have ret = 0 in this case.
Mariusz
Re: [PATCH v3 2/3] PCI/NPEM: Add Native PCIe Enclosure Management support
Quoted 111 lines · Mariusz Tkaczyk
Quoted 95 lines · On Mon, 8 Jul 2024 14:33:34 +0300 (EEST) Ilpo Järvinen…
Quoted 49 lines · Mariusz Tkaczyk
Native PCIe Enclosure Management (NPEM, PCIe r6.1 sec 6.28) allows managing LED in storage enclosures. NPEM is indication oriented and it does not give direct access to LED. Although each of the indications *could* represent an individual LED, multiple indications could also be represented as a single, multi-color LED or a single LED blinking in a specific interval. The specification leaves that open.
Each enabled indication (capability register bit on) is represented as a
ledclass_devwhich can be controlled through sysfs. For every ledclass device only 2 brightness states are allowed:LED_ON(1) orLED_OFF(0). It is corresponding to NPEM control register (Indication bit on/off).Ledclass devices appear in sysfs as child devices (subdirectory) of PCI device which has an NPEM Extended Capability and indication is enabled in NPEM capability register. For example, these are leds created for pcieport "10000:02:05.0" on my setup:
leds/
├── 10000:02:05.0:enclosure:fail ├── 10000:02:05.0:enclosure:locate ├── 10000:02:05.0:enclosure:ok └── 10000:02:05.0:enclosure:rebuildThey can be also found in "
/sys/class/leds" directory. Parent PCIe device bdf is used to guarantee uniqueness across leds subsystem.To enable/disable fail indication "brightness" file can be edited:
echo 1 > ./leds/10000:02:05.0:enclosure:fail/brightness echo 0 > ./leds/10000:02:05.0:enclosure:fail/brightnessPCIe r6.1, sec 7.9.19.2 defines the possible indications.
Multiple indications for same parent PCIe device can conflict and hardware may update them when processing new request. To avoid issues, driver refresh all indications by reading back control register.
Driver is projected to be exclusive NPEM extended capability manager. It waits up to 1 second after imposing new request, it doesn't verify if controller is busy before write, assuming that mutex lock gives protection from concurrent updates. Driver is not registered if
_DSMLED management is available.NPEM is a PCIe extended capability so it should be registered in
pcie_init_capabilities()but it is not possible due to LED dependency. Parentpci_devicemust be added earlier forled_classdev_register()to be successful. NPEM does not require configuration on kernel side, it is safe to register LED devices later.Quoted 3 lines
Suggested-by Lukas Wunner <[email protected]>Signed-off-by Mariusz Tkaczyk <[email protected]>Looks to be in quite good shape already, one comment below I think should be addressed before this is ready to go.
Quoted 31 lines
drivers/pci/npem.c new file +17
-0+static int npem_set_active_indications (struct npem *npem,u32 inds) +{ +int ctrl, ret, ret_val; +u32 cc_status; + +lockdep_assert_held (&npem->lock); + +/* This bit is always required */ + ctrl = inds |PCI_NPEM_CTRL_ENABLE ; + + ret =npem_write_ctrl (npem, ctrl); +if (ret) +return ret; + +/* +* For the case where a NPEM command has not completed immediately, +* it is recommended that software not continuously “spin” on polling
(quoted diff) +1
-0+ * the status register, but rather poll under interrupt at areduced
drivers/pci/npem.c new file +3
-0+* rate; for example at 10 ms intervals. +* +* PCIe r6.1 sec 6.28 "Implementation Note: Software Polling of NPEM
drivers/pci/npem.c new file +4
-0+* Command Completed" +*/ + ret = read_poll_timeout(npem_read_reg, ret_val, + ret_val || (cc_status &
PCI_NPEM_STATUS_CC),(quoted diff) +1
-0+ 10 * USEC_PER_MSEC, USEC_PER_SEC, false,npem,
drivers/pci/npem.c line 210
Quoted 3 lines
drivers/pci/npem.c new file +3
-0+ PCI_NPEM_STATUS, &cc_status); +if (ret) +return ret_val;Will this work as intended?
Quoted 5 lines
If
ret_valgets set, cond inread_poll_timeout()is true and it returns 0 so the return branch is not taken.Also, when
read_poll_timeout()times out,ret_valmight not be non-zero.Yes, it is good catch thanks! What about?
if (ret) return ret; if (ret_val) return ret_val;
Quoted 8 lines
If ret is set it means that it times out- we should return that to caller.
If
ret_valis set it means that we received error in npem_read_reg()- we should return that (device probably is unreachable).If
read_valis set then we are less interested in ret because error fromnpem_read_reg()function is more critical, so it is "acceptable" to have ret = 0 in this case.
Yes, I think that will do.
Signature · 1 line
i.
PATCH v3 3/3 PCI/NPEM: Add _DSM PCIe SSD status LED management
In reply to Mariusz TkaczykDevice Specific Method PCIe SSD Status LED Management (_DSM) defined in PCI Firmware Spec r3.3 sec 4.7 provides a way to manage LEDs via ACPI.
The design is similar to NPEM defined in PCIe Base Specification r6.1 sec 6.28:
- both standards are indication oriented,
- _DSM supported bits are corresponding to NPEM capability register bits
- _DSM control bits are corresponding to NPEM control register bits.
_DSM does not support enclosure specific indications and special NPEM commands NPEM_ENABLE and NPEM_RESET.
_DSM is implemented as a second op in NPEM driver. The standard used in background is not presented to user. The interface is accessed same as NPEM.
According to spec, _DSM has higher priority and availability of _DSM in not limited to devices with NPEM support. It is followed in implementation.
--- drivers/pci/npem.c | 147 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 144 insertions(+), 3 deletions(-)
drivers/pci/npem.c +144 -3
@@ -11,6 +11,14 @@* PCIe Base Specification r6.1 sec 6.28 * PCIe Base Specification r6.1 sec 7.9.19 * +* _DSM Definitions for PCIe SSD Status LED +* PCI Firmware Specification, r3.3 sec 4.7 +* +* Two backends are supported to manipulate indications: Direct NPEM register +* access (npem_ops) and indirect access through the ACPI _DSM (dsm_ops). +* _DSM is used if supported, else NPEM. +* +* Copyright (c) 2021-2022 Dell Inc. * Copyright (c) 2023-2024 Intel Corporation * Mariusz Tkaczyk <[email protected]> */
@@ -55,6 +63,21 @@ static const struct indication npem_indications[] = { {0, NULL} }; +/* _DSM PCIe SSD LED States are corresponding to NPEM register values */ +static const struct indication dsm_indications[] = { + {PCI_NPEM_IND_OK ,"enclosure:ok" }, + {PCI_NPEM_IND_LOCATE ,"enclosure:locate" }, + {PCI_NPEM_IND_FAIL ,"enclosure:fail" }, + {PCI_NPEM_IND_REBUILD ,"enclosure:rebuild" }, + {PCI_NPEM_IND_PFA ,"enclosure:pfa" }, + {PCI_NPEM_IND_HOTSPARE ,"enclosure:hotspare" }, + {PCI_NPEM_IND_ICA ,"enclosure:ica" }, + {PCI_NPEM_IND_IFA ,"enclosure:ifa" }, + {PCI_NPEM_IND_IDT ,"enclosure:idt" }, + {PCI_NPEM_IND_DISABLED ,"enclosure:disabled" }, + {0 ,NULL } +}; +#define for_each_indication (ind, inds) \ for (ind = inds; ind->bit; ind++)
@@ -250,6 +273,120 @@ static bool npem_has_dsm(struct pci_dev *pdev) BIT(GET_STATE_DSM) | BIT(SET_STATE_DSM)); } +struct dsm_output { +u16 status; +u8 function_specific_err; +u8 vendor_specific_err; +u32 state; +} __packed; + +/** +* dsm_evaluate() - send DSM PCIe SSD Status LED command +* @pdev: PCI device +* @dsm_func: DSM LED Function +* @output: buffer to copy DSM Response +* @value_to_set: value for SET_STATE_DSM function +* +* To not bother caller with ACPI context, the returned _DSM Output Buffer is +* copied. +*/ +static int dsm_evaluate (struct pci_dev *pdev,u64 dsm_func, +struct dsm_output *output,u32 value_to_set) +{ +acpi_handle handle =ACPI_HANDLE (&pdev->dev); +union acpi_object *out_obj, arg3[2 ]; +union acpi_object *arg3_p =NULL ; + +if (dsm_func ==SET_STATE_DSM ) { + arg3[0 ].type =ACPI_TYPE_PACKAGE ; + arg3[0 ].package.count =1 ; + arg3[0 ].package.elements = &arg3[1 ]; + + arg3[1 ].type =ACPI_TYPE_BUFFER ; + arg3[1 ].buffer.length =4 ; + arg3[1 ].buffer.pointer = (u8 *)&value_to_set; + + arg3_p = arg3; + } + + out_obj =acpi_evaluate_dsm_typed (handle, &dsm_guid,0x1 , dsm_func, + arg3_p,ACPI_TYPE_BUFFER ); +if (!out_obj) +return -EIO ; + +if (out_obj->buffer.length <sizeof (struct dsm_output )) { +ACPI_FREE (out_obj); +return -EIO ; + } + +memcpy (output, out_obj->buffer.pointer,sizeof (struct dsm_output )); + +ACPI_FREE (out_obj); +return 0 ; +} + +static int dsm_get (struct pci_dev *pdev,u64 dsm_func,u32 *buf) +{ +struct dsm_output output; +int ret =dsm_evaluate (pdev, dsm_func, &output,0 ); + +if (ret) +return ret; + +if (output.status !=0 ) +return -EIO ; + + *buf = output.state; +return 0 ; +} + +static int dsm_get_active_indications (struct npem *npem,u32 *buf) +{ +int ret =dsm_get (npem->dev,GET_STATE_DSM , buf); + +/* Filter out not supported indications in response */ + *buf &= npem->supported_indications; +return ret; +} + +static int dsm_set_active_indications (struct npem *npem,u32 value) +{ +struct dsm_output output; +int ret =dsm_evaluate (npem->dev,SET_STATE_DSM , &output, value); + +if (ret) +return ret; + +switch (output.status) { +case 4 : +/* +* Not all bits are set. If this bit is set, the platform +* disregarded some or all of the request state changes. OSPM +* should check the resulting PCIe SSD Status LED States to see +* what, if anything, has changed. +* +* PCI Firmware Specification, r3.3 Table 4-19. +*/ +if (output.function_specific_err !=1 ) +return -EIO ; + fallthrough; +case 0 : +break ; +default : +return -EIO ; + } + + npem->active_indications = output.state; + +return 0 ; +} + +static const struct npem_ops dsm_ops = { + .inds = dsm_indications, + .get_active_indications = dsm_get_active_indications, + .set_active_indications = dsm_set_active_indications, +}; +static int npem_initialize_active_indications (struct npem *npem) {int ret;
@@ -437,11 +574,15 @@ void pci_npem_create(struct pci_dev *dev)* OS should use the DSM for LED control if it is available * PCI Firmware Spec r3.3 sec 4.7. */ -return ; + ret =dsm_get (dev,GET_SUPPORTED_STATES_DSM , &cap); +if (ret) +return ; + + ops = &dsm_ops; } ret =pci_npem_init (dev, ops, pos, cap);if (ret) -pci_err (dev,"Failed to register PCIe Enclosure Management driver, err: %d\n" , - ret); +pci_err (dev,"Failed to register %s PCIe Enclosure Management driver, err: %d\n" , + (ops == &dsm_ops ?"_DSM" :"Native" ), ret); }
Signature · 1 line
2.35.3
Re: [PATCH v3 3/3] PCI/NPEM: Add _DSM PCIe SSD status LED management
drivers/pci/npem.c static bool npem_has_dsm(struct pci_dev *pdev) · line 281
Quoted 6 lines
drivers/pci/npem.c +6
-0+struct dsm_output { +u16 status; +u8 function_specific_err; +u8 vendor_specific_err; +u32 state; +} __packed;
This structure is naturally aligned, so no need for the __packed.
Otherwise looks good (or at least as good as something can look while interacting with the weirdo ACPI interfaces..):
Re: [PATCH v3 3/3] PCI/NPEM: Add _DSM PCIe SSD status LED management
Quoted 8 lines · Christoph Hellwig
drivers/pci/npem.c static bool npem_has_dsm(struct pci_dev *pdev) · line 281
Quoted 6 lines
drivers/pci/npem.c +6
-0+struct dsm_output { +u16 status; +u8 function_specific_err; +u8 vendor_specific_err; +u32 state; +} __packed;This structure is naturally aligned, so no need for the
__packed.
Isn't the compiler free to insert padding wherever it sees fit?
structs passed to ACPI firmware would no longer comply with the spec-prescribed layout then and declaring them __packed seems to be the only way to ensure that doesn't happen.
Thanks,
Lukas
Re: [PATCH v3 3/3] PCI/NPEM: Add _DSM PCIe SSD status LED management
Quoted 11 lines · Lukas Wunner
Quoted 8 lines · Christoph Hellwig
drivers/pci/npem.c static bool npem_has_dsm(struct pci_dev *pdev) · line 281
Quoted 6 lines
drivers/pci/npem.c +6
-0+struct dsm_output { +u16 status; +u8 function_specific_err; +u8 vendor_specific_err; +u32 state; +} __packed;This structure is naturally aligned, so no need for the
__packed.Isn't the compiler free to insert padding wherever it sees fit?
In terms of the standard: yes. It could even reorder members. In terms of not breaking the Linux kernel (or other low-level software) left, right and center: no.
Quoted 3 lines
structs passed to ACPI firmware would no longer comply with the spec-prescribed layout then and declaring them
__packedseems to be the only way to ensure that doesn't happen.
Take a look at just about any driver, file system or network protocol.
Re: [PATCH v3 3/3] PCI/NPEM: Add _DSM PCIe SSD status LED management
In reply to Mariusz TkaczykQuoted 73 lines · Mariusz Tkaczyk
Device Specific Method PCIe SSD Status LED Management (
_DSM) defined in PCI Firmware Spec r3.3 sec 4.7 provides a way to manage LEDs via ACPI.The design is similar to NPEM defined in PCIe Base Specification r6.1 sec 6.28:
- both standards are indication oriented,
-_DSMsupported bits are corresponding to NPEM capability register bits
-_DSMcontrol bits are corresponding to NPEM control register bits.
_DSMdoes not support enclosure specific indications and special NPEM commandsNPEM_ENABLEandNPEM_RESET.
_DSMis implemented as a second op in NPEM driver. The standard used in background is not presented to user. The interface is accessed same as NPEM.According to spec,
_DSMhas higher priority and availability of_DSMin not limited to devices with NPEM support. It is followed in implementation.Suggested-by Lukas Wunner <[email protected]>Signed-off-by Stuart Hayes <[email protected]>Signed-off-by Mariusz Tkaczyk <[email protected]>--- drivers/pci/npem.c | 147 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 144 insertions(+), 3 deletions(-)drivers/pci/npem.c +23
-0@@ -11,6 +11,14 @@* PCIe Base Specification r6.1 sec 6.28 * PCIe Base Specification r6.1 sec 7.9.19 * +* _DSM Definitions for PCIe SSD Status LED +* PCI Firmware Specification, r3.3 sec 4.7 +* +* Two backends are supported to manipulate indications: Direct NPEM register +* access (npem_ops) and indirect access through the ACPI _DSM (dsm_ops). +* _DSM is used if supported, else NPEM. +* +* Copyright (c) 2021-2022 Dell Inc. * Copyright (c) 2023-2024 Intel Corporation * Mariusz Tkaczyk <[email protected]> */ @@ -55,6 +63,21 @@ static const struct indication npem_indications[] = { {0, NULL} }; +/* _DSM PCIe SSD LED States are corresponding to NPEM register values */ +static const struct indication dsm_indications[] = { + {PCI_NPEM_IND_OK ,"enclosure:ok" }, + {PCI_NPEM_IND_LOCATE ,"enclosure:locate" }, + {PCI_NPEM_IND_FAIL ,"enclosure:fail" }, + {PCI_NPEM_IND_REBUILD ,"enclosure:rebuild" }, + {PCI_NPEM_IND_PFA ,"enclosure:pfa" }, + {PCI_NPEM_IND_HOTSPARE ,"enclosure:hotspare" }, + {PCI_NPEM_IND_ICA ,"enclosure:ica" }, + {PCI_NPEM_IND_IFA ,"enclosure:ifa" }, + {PCI_NPEM_IND_IDT ,"enclosure:idt" }, + {PCI_NPEM_IND_DISABLED ,"enclosure:disabled" }, + {0 ,NULL } +}; +#define for_each_indication (ind, inds) \ for (ind = inds; ind->bit; ind++)
drivers/pci/npem.c static bool npem_has_dsm(struct pci_dev *pdev) · line 281
Quoted 9 lines
drivers/pci/npem.c +6
-0@@ -250,6 +273,120 @@ static bool npem_has_dsm(struct pci_dev *pdev) BIT(GET_STATE_DSM) | BIT(SET_STATE_DSM)); } +struct dsm_output { +u16 status; +u8 function_specific_err; +u8 vendor_specific_err; +u32 state; +} __packed;
As mentioned by Christoph, __packed is not required here due to natural alignment (Using __packed will cause other effects beyond just preventing compiler from adding padding which is why it's good to avoid it where it's not needed).
With that removed,
Signature · 1 line
i.
Quoted 20 lines · Mariusz Tkaczyk
Patchset is named as PCIe Enclosure LED Management because it adds two features:
- Native PCIe Enclosure Management (NPEM)
- PCIe SSD Status LED Management (DSM)Both are pattern oriented standards, they tell which "indication" should blink. It doesn't control physical LED or pattern visualization.
Overall, driver is simple but it was not simple to fit it into interfaces we have in kernel (We considered leds and enclosure interfaces). It reuses leds interface, this approach seems to be the best because:
- leds are actively maintained, no new interface added.
- leds do not require any extensions, enclosure needs to be adjusted first.There are trade-offs:
- "brightness" is the name of sysfs file to control led. It is not natural to use brightness to set patterns, that is why multiple led devices are created (one per indication);
- Update of one led may affect other leds, led triggers may not work as expected.
Quoted 3 lines
It was tested with
_DSMbut I do some minor updates after review. Stuart please retest.
Works great for me, thank you!!