Expose and manage PCI device reset
Subsystems (6)
v7 cover Expose and manage PCI device reset
PCI and PCIe devices may support a number of possible reset mechanisms for example Function Level Reset (FLR) provided via Advanced Feature or PCIe capabilities, Power Management reset, bus reset, or device specific reset. Currently the PCI subsystem creates a policy prioritizing these reset methods which provides neither visibility nor control to userspace.
Expose the reset methods available per device to userspace, via sysfs and allow an administrative user or device owner to have ability to manage per device reset method priorities or exclusions. This feature aims to allow greater control of a device for use cases as device assignment, where specific device or platform issues may interact poorly with a given reset method, and for which device specific quirks have not been developed.
Changes in v7:
- Fix the pci_dev_acpi_reset() prototype mismatch in case of CONFIG_ACPI=n
Changes in v6:
- Address Bjorn's and Krzysztof's review comments
- Add Shanker's updated patches along with new "PCI: Setup ACPI_COMPANION early" patch
Changes in v5:
- Rebase the series over pci/reset branch of Bjorn's pci tree to avoid merge conflicts caused by recent changes in existing reset sysfs attribute
Changes in v4:
- Change the order or strlen and strim in reset_method_store function to avoid extra strlen call.
- Use consistent terminology in new pci_reset_mode enum and rename the probe argument of reset functions.
Changes in v3:
- Dropped "PCI: merge slot and bus reset implementations" which was already accepted separately
- Grammar fixes
- Added Shanker's patches which were rebased on v2 of this series
- Added "PCI: Change the type of probe argument in reset functions" and additional user input sanitization code in reset_method_store function per review feedback from Krzysztof
Changes in v2:
- Use byte array instead of bitmap to keep track of ordering of reset methods
- Fix incorrect use of reset_fn field in octeon driver
- Allow writing comma separated list of names of supported reset methods to reset_method sysfs attribute
- Writing empty string instead of "none" to reset_method attribute disables ability of reset the device
Amey Narkhede (5):
PCI: Add pcie_reset_flr to follow calling convention of other reset
methods
PCI: Add new array for keeping track of ordering of reset methods
PCI: Remove reset_fn field from pci_dev
PCI/sysfs: Allow userspace to query and set device reset mechanism
PCI: Change the type of probe argument in reset functions
Shanker Donthineni (3):
PCI: Setup ACPI_COMPANION early
PCI: Add support for ACPI _RST reset method
PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
Documentation/ABI/testing/sysfs-bus-pci | 16 ++ drivers/crypto/cavium/nitrox/nitrox_main.c | 4 +- .../ethernet/cavium/liquidio/lio_vf_main.c | 2 +- drivers/pci/hotplug/pciehp.h | 2 +- drivers/pci/hotplug/pciehp_hpc.c | 4 +- drivers/pci/pci-acpi.c | 39 ++- drivers/pci/pci-sysfs.c | 120 ++++++++- drivers/pci/pci.c | 238 +++++++++++------- drivers/pci/pci.h | 22 +- drivers/pci/pcie/aer.c | 12 +- drivers/pci/probe.c | 6 +- drivers/pci/quirks.c | 54 ++-- drivers/pci/remove.c | 1 - include/linux/pci.h | 16 +- include/linux/pci_hotplug.h | 2 +- 15 files changed, 393 insertions(+), 145 deletions(-)
--
2.31.1
PATCH v7 1/8 PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.
--- drivers/crypto/cavium/nitrox/nitrox_main.c | 4 +- drivers/pci/pci.c | 62 ++++++++++++---------- drivers/pci/pcie/aer.c | 12 ++--- drivers/pci/quirks.c | 9 ++-- include/linux/pci.h | 2 +- 5 files changed, 43 insertions(+), 46 deletions(-)
drivers/crypto/cavium/nitrox/nitrox_main.c +1 -3
@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)return -ENOMEM ; } -/* check flr support */ -if (pcie_has_flr (pdev)) -pcie_flr (pdev); +pcie_reset_flr (pdev,0 );pci_restore_state (pdev);
drivers/pci/pci.c +33 -29
90 lines not on this page · raw message
drivers/pci/pcie/aer.c +5 -7
@@ -1405,13 +1405,11 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev) }if (type ==PCI_EXP_TYPE_RC_EC || type ==PCI_EXP_TYPE_RC_END ) { -if (pcie_has_flr (dev)) { - rc =pcie_flr (dev); -pci_info (dev,"has been reset (%d)\n" , rc); - }else { -pci_info (dev,"not reset (no FLR support)\n" ); - rc = -ENOTTY ; - } + rc =pcie_reset_flr (dev,0 ); +if (!rc) +pci_info (dev,"has been reset\n" ); +else +pci_info (dev,"not reset (no FLR support: %d)\n" , rc); }else { rc =pci_bus_error_reset (dev); pci_info(dev, "%s Port link has been reset (%d)\n",
drivers/pci/quirks.c +3 -6
@@ -3819,7 +3819,7 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)u32 cfg;if (dev->class !=PCI_CLASS_STORAGE_EXPRESS || - !pcie_has_flr (dev) || !pci_resource_start (dev,0 )) +pcie_reset_flr (dev,1 ) || !pci_resource_start (dev,0 ))return -ENOTTY ;if (probe)
@@ -3888,13 +3888,10 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)*/ static int delay_250ms_after_flr (struct pci_dev *dev,int probe) { -if (!pcie_has_flr (dev)) -return -ENOTTY ; +int ret =pcie_reset_flr (dev, probe);if (probe) -return 0 ; - -pcie_flr (dev); +return ret;msleep (250 );
include/linux/pci.h +1 -1
@@ -1225,7 +1225,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,enum pci_bus_speed *speed, enum pcie_link_width *width);void pcie_print_link_status (struct pci_dev *dev); -bool pcie_has_flr (struct pci_dev *dev); +int pcie_reset_flr (struct pci_dev *dev,int probe);int pcie_flr (struct pci_dev *dev);int __pci_reset_function_locked (struct pci_dev *dev);int pci_reset_function (struct pci_dev *dev);
Signature · 1 line
2.31.1
Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Quoted 11 lines · Amey Narkhede
Currently there is separate function
pcie_has_flr()to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new functionpcie_reset_flr()that follows the calling convention of reset methods.Reviewed-by Alex Williamson <[email protected]>Reviewed-by Raphael Norwitz <[email protected]>Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>
Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
In reply to Amey Narkhede[+cc Christoph, since he added pcie_flr()]
Quoted 5 lines · Amey Narkhede
Currently there is separate function
pcie_has_flr()to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new functionpcie_reset_flr()that follows the calling convention of reset methods.
I don't like the fact that we handle FLR differently from other types of reset, so I do like the fact that this makes them more consistent.
Quoted 83 lines
Reviewed-by Alex Williamson <[email protected]>Reviewed-by Raphael Norwitz <[email protected]>Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>--- drivers/crypto/cavium/nitrox/nitrox_main.c | 4 +- drivers/pci/pci.c | 62 ++++++++++++---------- drivers/pci/pcie/aer.c | 12 ++--- drivers/pci/quirks.c | 9 ++-- include/linux/pci.h | 2 +- 5 files changed, 43 insertions(+), 46 deletions(-)drivers/crypto/cavium/nitrox/nitrox_main.c +1
-3@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)return -ENOMEM ; } -/* check flr support */ -if (pcie_has_flr (pdev)) -pcie_flr (pdev); +pcie_reset_flr (pdev,0 );pci_restore_state (pdev);drivers/pci/pci.c +14
-22@@ -4611,32 +4611,12 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev) }EXPORT_SYMBOL (pci_wait_for_pending_transaction); -/** -* pcie_has_flr - check if a device supports function level resets -* @dev: device to check -* -* Returns true if the device advertises support for PCIe function level -* resets. -*/ -bool pcie_has_flr (struct pci_dev *dev) -{ -u32 cap; - -if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) -return false; - -pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); -return cap &PCI_EXP_DEVCAP_FLR ; -} -EXPORT_SYMBOL_GPL (pcie_has_flr); -/** * pcie_flr - initiate a PCIe function level reset * @dev: device to reset * -* Initiate a function level reset on @dev. The caller should ensure the -* device supports FLR before calling this function, e.g. by using the -* pcie_has_flr() helper. +* Initiate a function level reset unconditionally on @dev without +* checking any flags and DEVCAP */ int pcie_flr (struct pci_dev *dev) {@@ -4659,6 +4639,31 @@ int pcie_flr(struct pci_dev *dev) }EXPORT_SYMBOL_GPL (pcie_flr); +/** +* pcie_reset_flr - initiate a PCIe function level reset +* @dev: device to reset +* @probe: If set, only check if the device can be reset this way. +* +* Initiate a function level reset on @dev. +*/ +int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET )
drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4663
Quoted 10 lines
drivers/pci/pci.c +10
-0+return -ENOTTY ; + +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev);
Christoph added pcie_flr() with a60a2b73ba69 ("PCI: Export pcie_flr()"), where the commit log says he split out the probing because "non-core callers already know their hardware."
It *is* reasonable to expect that drivers know whether their device supports FLR so they don't need to probe.
But we don't expose the "probe" argument outside the PCI core for any other reset methods, and I would like to avoid that here.
It seems excessive to have to read PCI_EXP_DEVCAP every time. PCI_EXP_DEVCAP_FLR is a read-only bit, and we should only need to look at it once.
What I would really like here is a single bit in the pci_dev that we could set at enumeration-time, e.g., something like this:
struct pci_dev {
...
unsigned int has_flr:1;
};
void set_pcie_port_type(...) # during enumeration
{
pci_read_config_word(dev, pos + PCI_EXP_DEVCAP, ®16);
if (reg16 & PCI_EXP_DEVCAP_FLR)
dev->has_flr = 1;
}
static void quirk_no_flr(...)
{
dev->has_flr = 0; # get rid of PCI_DEV_FLAGS_NO_FLR_RESET
}
int pcie_flr(...)
{
if (!dev->has_flr)
return -ENOTTY; if (!pci_wait_for_pending_transaction(dev))
...}
I think this should be enough that we could get rid of pcie_has_flr() without having to expose the "probe" argument outside drivers/pci/.
Procedural note: if we *do* have to expose the "probe" argument, can you arrange it to have the correct type before touching the drivers, so we only have to touch the drivers once?
Quoted 101 lines
drivers/pci/pci.c +9
-7+} +EXPORT_SYMBOL_GPL (pcie_reset_flr); +static int pci_af_flr (struct pci_dev *dev,int probe) {int pos;@@ -5139,11 +5144,9 @@ int __pci_reset_function_locked(struct pci_dev *dev) rc =pci_dev_specific_reset (dev,0 );if (rc != -ENOTTY )return rc; -if (pcie_has_flr (dev)) { - rc =pcie_flr (dev); -if (rc != -ENOTTY ) -return rc; - } + rc =pcie_reset_flr (dev,0 ); +if (rc != -ENOTTY ) +return rc; rc =pci_af_flr (dev,0 );if (rc != -ENOTTY )return rc;@@ -5174,8 +5177,9 @@ int pci_probe_reset_function(struct pci_dev *dev) rc =pci_dev_specific_reset (dev,1 );if (rc != -ENOTTY )return rc; -if (pcie_has_flr (dev)) -return 0 ; + rc =pcie_reset_flr (dev,1 ); +if (rc != -ENOTTY ) +return rc; rc =pci_af_flr (dev,1 );if (rc != -ENOTTY )return rc;drivers/pci/pcie/aer.c +5
-7@@ -1405,13 +1405,11 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev) }if (type ==PCI_EXP_TYPE_RC_EC || type ==PCI_EXP_TYPE_RC_END ) { -if (pcie_has_flr (dev)) { - rc =pcie_flr (dev); -pci_info (dev,"has been reset (%d)\n" , rc); - }else { -pci_info (dev,"not reset (no FLR support)\n" ); - rc = -ENOTTY ; - } + rc =pcie_reset_flr (dev,0 ); +if (!rc) +pci_info (dev,"has been reset\n" ); +else +pci_info (dev,"not reset (no FLR support: %d)\n" , rc); }else { rc =pci_bus_error_reset (dev); pci_info(dev, "%s Port link has been reset (%d)\n",drivers/pci/quirks.c +3
-6@@ -3819,7 +3819,7 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)u32 cfg;if (dev->class !=PCI_CLASS_STORAGE_EXPRESS || - !pcie_has_flr (dev) || !pci_resource_start (dev,0 )) +pcie_reset_flr (dev,1 ) || !pci_resource_start (dev,0 ))return -ENOTTY ;if (probe)@@ -3888,13 +3888,10 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)*/ static int delay_250ms_after_flr (struct pci_dev *dev,int probe) { -if (!pcie_has_flr (dev)) -return -ENOTTY ; +int ret =pcie_reset_flr (dev, probe);if (probe) -return 0 ; - -pcie_flr (dev); +return ret;msleep (250 );include/linux/pci.h +1
-1@@ -1225,7 +1225,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,enum pci_bus_speed *speed, enum pcie_link_width *width);void pcie_print_link_status (struct pci_dev *dev); -bool pcie_has_flr (struct pci_dev *dev); +int pcie_reset_flr (struct pci_dev *dev,int probe);int pcie_flr (struct pci_dev *dev);int __pci_reset_function_locked (struct pci_dev *dev);int pci_reset_function (struct pci_dev *dev);Signature · 1 line
2.31.1
Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Quoted 110 lines · On Thu, 17 Jun 2021 16:57:34 -0500 Bjorn Helgaas <[email protected]> wrote:
[+cc Christoph, since he added
pcie_flr()]Quoted 5 lines · Amey Narkhede
Currently there is separate function
pcie_has_flr()to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new functionpcie_reset_flr()that follows the calling convention of reset methods.I don't like the fact that we handle FLR differently from other types of reset, so I do like the fact that this makes them more consistent.
Quoted 83 lines
Reviewed-by Alex Williamson <[email protected]>Reviewed-by Raphael Norwitz <[email protected]>Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>--- drivers/crypto/cavium/nitrox/nitrox_main.c | 4 +- drivers/pci/pci.c | 62 ++++++++++++---------- drivers/pci/pcie/aer.c | 12 ++--- drivers/pci/quirks.c | 9 ++-- include/linux/pci.h | 2 +- 5 files changed, 43 insertions(+), 46 deletions(-)drivers/crypto/cavium/nitrox/nitrox_main.c +1
-3@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)return -ENOMEM ; } -/* check flr support */ -if (pcie_has_flr (pdev)) -pcie_flr (pdev); +pcie_reset_flr (pdev,0 );pci_restore_state (pdev);drivers/pci/pci.c +14
-22@@ -4611,32 +4611,12 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev) }EXPORT_SYMBOL (pci_wait_for_pending_transaction); -/** -* pcie_has_flr - check if a device supports function level resets -* @dev: device to check -* -* Returns true if the device advertises support for PCIe function level -* resets. -*/ -bool pcie_has_flr (struct pci_dev *dev) -{ -u32 cap; - -if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) -return false; - -pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); -return cap &PCI_EXP_DEVCAP_FLR ; -} -EXPORT_SYMBOL_GPL (pcie_has_flr); -/** * pcie_flr - initiate a PCIe function level reset * @dev: device to reset * -* Initiate a function level reset on @dev. The caller should ensure the -* device supports FLR before calling this function, e.g. by using the -* pcie_has_flr() helper. +* Initiate a function level reset unconditionally on @dev without +* checking any flags and DEVCAP */ int pcie_flr (struct pci_dev *dev) {@@ -4659,6 +4639,31 @@ int pcie_flr(struct pci_dev *dev) }EXPORT_SYMBOL_GPL (pcie_flr); +/** +* pcie_reset_flr - initiate a PCIe function level reset +* @dev: device to reset +* @probe: If set, only check if the device can be reset this way. +* +* Initiate a function level reset on @dev. +*/ +int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET )drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4663
Quoted 10 lines
drivers/pci/pci.c +10
-0+return -ENOTTY ; + +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev);Christoph added
pcie_flr()witha60a2b73ba69("PCI: Exportpcie_flr()"), where the commit log says he split out the probing because "non-core callers already know their hardware."
Quoted 2 lines
It *is* reasonable to expect that drivers know whether their device supports FLR so they don't need to probe.
I don't think it changes your suggestion below, but this statement is a little troublesome when we look at devices running in VMs where we've been known to hide various capabilities, or simply quirks where some combination of a known device feature might be otherwise avoided. A more robust driver should try to make fewer assumptions for these cases, if not simply the inability to predict future changes to the hardware.
FLR should be a relatively rare event, but caching for driver API purposes seems reasonable. Thanks,
Alex
Quoted 145 lines
But we don't expose the "probe" argument outside the PCI core for any other reset methods, and I would like to avoid that here.
It seems excessive to have to read
PCI_EXP_DEVCAPevery time.PCI_EXP_DEVCAP_FLRis a read-only bit, and we should only need to look at it once.What I would really like here is a single bit in the
pci_devthat we could set at enumeration-time, e.g., something like this:struct pci_dev { ... unsigned int has_flr:1; }; void set_pcie_port_type(...) # during enumeration { pci_read_config_word(dev, pos + PCI_EXP_DEVCAP, ®16); if (reg16 & PCI_EXP_DEVCAP_FLR) dev->has_flr = 1; } static void quirk_no_flr(...) { dev->has_flr = 0; # get rid of PCI_DEV_FLAGS_NO_FLR_RESET } int pcie_flr(...) { if (!dev->has_flr) return -ENOTTY;if (!pci_wait_for_pending_transaction(dev)) ...}
I think this should be enough that we could get rid of
pcie_has_flr()without having to expose the "probe" argument outsidedrivers/pci/.Procedural note: if we *do* have to expose the "probe" argument, can you arrange it to have the correct type before touching the drivers, so we only have to touch the drivers once?
Quoted 101 lines
drivers/pci/pci.c +9
-7+} +EXPORT_SYMBOL_GPL (pcie_reset_flr); +static int pci_af_flr (struct pci_dev *dev,int probe) {int pos;@@ -5139,11 +5144,9 @@ int __pci_reset_function_locked(struct pci_dev *dev) rc =pci_dev_specific_reset (dev,0 );if (rc != -ENOTTY )return rc; -if (pcie_has_flr (dev)) { - rc =pcie_flr (dev); -if (rc != -ENOTTY ) -return rc; - } + rc =pcie_reset_flr (dev,0 ); +if (rc != -ENOTTY ) +return rc; rc =pci_af_flr (dev,0 );if (rc != -ENOTTY )return rc;@@ -5174,8 +5177,9 @@ int pci_probe_reset_function(struct pci_dev *dev) rc =pci_dev_specific_reset (dev,1 );if (rc != -ENOTTY )return rc; -if (pcie_has_flr (dev)) -return 0 ; + rc =pcie_reset_flr (dev,1 ); +if (rc != -ENOTTY ) +return rc; rc =pci_af_flr (dev,1 );if (rc != -ENOTTY )return rc;drivers/pci/pcie/aer.c +5
-7@@ -1405,13 +1405,11 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev) }if (type ==PCI_EXP_TYPE_RC_EC || type ==PCI_EXP_TYPE_RC_END ) { -if (pcie_has_flr (dev)) { - rc =pcie_flr (dev); -pci_info (dev,"has been reset (%d)\n" , rc); - }else { -pci_info (dev,"not reset (no FLR support)\n" ); - rc = -ENOTTY ; - } + rc =pcie_reset_flr (dev,0 ); +if (!rc) +pci_info (dev,"has been reset\n" ); +else +pci_info (dev,"not reset (no FLR support: %d)\n" , rc); }else { rc =pci_bus_error_reset (dev); pci_info(dev, "%s Port link has been reset (%d)\n",drivers/pci/quirks.c +3
-6@@ -3819,7 +3819,7 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)u32 cfg;if (dev->class !=PCI_CLASS_STORAGE_EXPRESS || - !pcie_has_flr (dev) || !pci_resource_start (dev,0 )) +pcie_reset_flr (dev,1 ) || !pci_resource_start (dev,0 ))return -ENOTTY ;if (probe)@@ -3888,13 +3888,10 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)*/ static int delay_250ms_after_flr (struct pci_dev *dev,int probe) { -if (!pcie_has_flr (dev)) -return -ENOTTY ; +int ret =pcie_reset_flr (dev, probe);if (probe) -return 0 ; - -pcie_flr (dev); +return ret;msleep (250 );include/linux/pci.h +1
-1@@ -1225,7 +1225,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,enum pci_bus_speed *speed, enum pcie_link_width *width);void pcie_print_link_status (struct pci_dev *dev); -bool pcie_has_flr (struct pci_dev *dev); +int pcie_reset_flr (struct pci_dev *dev,int probe);int pcie_flr (struct pci_dev *dev);int __pci_reset_function_locked (struct pci_dev *dev);int pci_reset_function (struct pci_dev *dev);Signature · 1 line
2.31.1
Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
In reply to Bjorn HelgaasQuoted 147 lines · Bjorn Helgaas
[+cc Christoph, since he added
pcie_flr()]Quoted 5 lines · Amey Narkhede
Currently there is separate function
pcie_has_flr()to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new functionpcie_reset_flr()that follows the calling convention of reset methods.I don't like the fact that we handle FLR differently from other types of reset, so I do like the fact that this makes them more consistent.
Quoted 83 lines
Reviewed-by Alex Williamson <[email protected]>Reviewed-by Raphael Norwitz <[email protected]>Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>--- drivers/crypto/cavium/nitrox/nitrox_main.c | 4 +- drivers/pci/pci.c | 62 ++++++++++++---------- drivers/pci/pcie/aer.c | 12 ++--- drivers/pci/quirks.c | 9 ++-- include/linux/pci.h | 2 +- 5 files changed, 43 insertions(+), 46 deletions(-)drivers/crypto/cavium/nitrox/nitrox_main.c +1
-3@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)return -ENOMEM ; } -/* check flr support */ -if (pcie_has_flr (pdev)) -pcie_flr (pdev); +pcie_reset_flr (pdev,0 );pci_restore_state (pdev);drivers/pci/pci.c +14
-22@@ -4611,32 +4611,12 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev) }EXPORT_SYMBOL (pci_wait_for_pending_transaction); -/** -* pcie_has_flr - check if a device supports function level resets -* @dev: device to check -* -* Returns true if the device advertises support for PCIe function level -* resets. -*/ -bool pcie_has_flr (struct pci_dev *dev) -{ -u32 cap; - -if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) -return false; - -pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); -return cap &PCI_EXP_DEVCAP_FLR ; -} -EXPORT_SYMBOL_GPL (pcie_has_flr); -/** * pcie_flr - initiate a PCIe function level reset * @dev: device to reset * -* Initiate a function level reset on @dev. The caller should ensure the -* device supports FLR before calling this function, e.g. by using the -* pcie_has_flr() helper. +* Initiate a function level reset unconditionally on @dev without +* checking any flags and DEVCAP */ int pcie_flr (struct pci_dev *dev) {@@ -4659,6 +4639,31 @@ int pcie_flr(struct pci_dev *dev) }EXPORT_SYMBOL_GPL (pcie_flr); +/** +* pcie_reset_flr - initiate a PCIe function level reset +* @dev: device to reset +* @probe: If set, only check if the device can be reset this way. +* +* Initiate a function level reset on @dev. +*/ +int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET )drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4663
Quoted 10 lines
drivers/pci/pci.c +10
-0+return -ENOTTY ; + +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev);Christoph added
pcie_flr()witha60a2b73ba69("PCI: Exportpcie_flr()"), where the commit log says he split out the probing because "non-core callers already know their hardware."It *is* reasonable to expect that drivers know whether their device supports FLR so they don't need to probe.
But we don't expose the "probe" argument outside the PCI core for any other reset methods, and I would like to avoid that here.
It seems excessive to have to read
PCI_EXP_DEVCAPevery time.PCI_EXP_DEVCAP_FLRis a read-only bit, and we should only need to look at it once.What I would really like here is a single bit in the
pci_devthat we could set at enumeration-time, e.g., something like this:struct pci_dev { ... unsigned int has_flr:1; }; void set_pcie_port_type(...) # during enumeration { pci_read_config_word(dev, pos + PCI_EXP_DEVCAP, ®16); if (reg16 & PCI_EXP_DEVCAP_FLR) dev->has_flr = 1; } static void quirk_no_flr(...) { dev->has_flr = 0; # get rid of PCI_DEV_FLAGS_NO_FLR_RESET } int pcie_flr(...) { if (!dev->has_flr) return -ENOTTY;if (!pci_wait_for_pending_transaction(dev)) ...
Quoted 9 lines
}
I think this should be enough that we could get rid of
pcie_has_flr()without having to expose the "probe" argument outsidedrivers/pci/.Procedural note: if we *do* have to expose the "probe" argument, can you arrange it to have the correct type before touching the drivers, so we only have to touch the drivers once?
Thanks for the details. I'll add dev->has_flr check in pcie_reset_flr. [...]
Thanks,
Amey
Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
In reply to Amey NarkhedeQuoted 19 lines · Amey Narkhede
Currently there is separate function
pcie_has_flr()to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new functionpcie_reset_flr()that follows the calling convention of reset methods.drivers/pci/pci.c +13
-0+/** +* pcie_reset_flr - initiate a PCIe function level reset +* @dev: device to reset +* @probe: If set, only check if the device can be reset this way. +* +* Initiate a function level reset on @dev. +*/ +int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) +return -ENOTTY ;
drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4664
Quoted 10 lines
drivers/pci/pci.c +10
-0+ +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev); +}
Tangent: I've been told before, but I can't remember why we need the "probe" interface. Since we're looking at this area again, can we add a comment to clarify this?
Every time I read this, I wonder why we can't just get rid of the probe and attempt a reset. If it fails because it's not supported, we could just try the next one in the list.
Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Quoted 35 lines · Bjorn Helgaas
Quoted 19 lines · Amey Narkhede
Currently there is separate function
pcie_has_flr()to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new functionpcie_reset_flr()that follows the calling convention of reset methods.drivers/pci/pci.c +13
-0+/** +* pcie_reset_flr - initiate a PCIe function level reset +* @dev: device to reset +* @probe: If set, only check if the device can be reset this way. +* +* Initiate a function level reset on @dev. +*/ +int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) +return -ENOTTY ;drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4664
Quoted 10 lines
drivers/pci/pci.c +10
-0+ +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev); +}Tangent: I've been told before, but I can't remember why we need the "probe" interface. Since we're looking at this area again, can we add a comment to clarify this?
Quoted 3 lines
Every time I read this, I wonder why we can't just get rid of the probe and attempt a reset. If it fails because it's not supported, we could just try the next one in the list.
Part of the reason is to have same calling convention as other reset methods and other reason is devices that run in VMs where various capabilities can be hidden or have quirks for avoiding known troublesome combination of device features as Alex explained here
https://lore.kernel.org/linux-pci/20210624151242.ybew2z5rseuusj7v@archlinux/T/#mb67c09a2ce08ce4787652e4c0e7b9e5adf1df57a
On the side note as you suggested earlier to cache flr capability earlier the PCI_EXP_DEVCAP reading code won't be there in next version so its just trivial check(dev->has_flr).
Thanks,
Amey
Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
[+to Alex]
Quoted 46 lines · Amey Narkhede
Quoted 35 lines · Bjorn Helgaas
Quoted 19 lines · Amey Narkhede
Currently there is separate function
pcie_has_flr()to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new functionpcie_reset_flr()that follows the calling convention of reset methods.drivers/pci/pci.c +13
-0+/** +* pcie_reset_flr - initiate a PCIe function level reset +* @dev: device to reset +* @probe: If set, only check if the device can be reset this way. +* +* Initiate a function level reset on @dev. +*/ +int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) +return -ENOTTY ;drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4664
Quoted 10 lines
drivers/pci/pci.c +10
-0+ +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev); +}Tangent: I've been told before, but I can't remember why we need the "probe" interface. Since we're looking at this area again, can we add a comment to clarify this?
Quoted 3 lines
Every time I read this, I wonder why we can't just get rid of the probe and attempt a reset. If it fails because it's not supported, we could just try the next one in the list.
Part of the reason is to have same calling convention as other reset methods and other reason is devices that run in VMs where various capabilities can be hidden or have quirks for avoiding known troublesome combination of device features as Alex explained here
https://lore.kernel.org/linux-pci/20210624151242.ybew2z5rseuusj7v@archlinux/T/#mb67c09a2ce08ce4787652e4c0e7b9e5adf1df57a
Quoted 3 lines
On the side note as you suggested earlier to cache flr capability earlier the
PCI_EXP_DEVCAPreading code won't be there in next version so its just trivialcheck(dev->has_flr).
Sorry, I didn't make my question clear. I'm not asking why we're adding a "probe" argument to pcie_reset_flr() to make it consistent with pci_af_flr(), pci_pm_reset(), pci_parent_bus_reset(), etc. I like making the interfaces consistent.
What I'm asking here is why the "probe" argument exists for *any* of these interfaces and why pci_probe_reset_function() exists.
This is really more a question for Alex since it's a historical question, not anything directly related to your series. I'm not proposing *removing* the "probe" argument; I know it exists for a reason because I've asked about it before. But I forgot the answer, which makes me think a hint in the code would be useful.
Bjorn
Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Quoted 58 lines · On Thu, 24 Jun 2021 11:15:59 -0500 Bjorn Helgaas <[email protected]> wrote:
[+to Alex]
Quoted 46 lines · Amey Narkhede
Quoted 35 lines · Bjorn Helgaas
Quoted 19 lines · Amey Narkhede
Currently there is separate function
pcie_has_flr()to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new functionpcie_reset_flr()that follows the calling convention of reset methods.drivers/pci/pci.c +13
-0+/** +* pcie_reset_flr - initiate a PCIe function level reset +* @dev: device to reset +* @probe: If set, only check if the device can be reset this way. +* +* Initiate a function level reset on @dev. +*/ +int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) +return -ENOTTY ;drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4664
Quoted 10 lines
drivers/pci/pci.c +10
-0+ +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev); +}Tangent: I've been told before, but I can't remember why we need the "probe" interface. Since we're looking at this area again, can we add a comment to clarify this?
Quoted 3 lines
Every time I read this, I wonder why we can't just get rid of the probe and attempt a reset. If it fails because it's not supported, we could just try the next one in the list.
Part of the reason is to have same calling convention as other reset methods and other reason is devices that run in VMs where various capabilities can be hidden or have quirks for avoiding known troublesome combination of device features as Alex explained here
https://lore.kernel.org/linux-pci/20210624151242.ybew2z5rseuusj7v@archlinux/T/#mb67c09a2ce08ce4787652e4c0e7b9e5adf1df57aQuoted 3 lines
On the side note as you suggested earlier to cache flr capability earlier the
PCI_EXP_DEVCAPreading code won't be there in next version so its just trivialcheck(dev->has_flr).Sorry, I didn't make my question clear. I'm not asking why we're adding a "probe" argument to
pcie_reset_flr()to make it consistent withpci_af_flr(),pci_pm_reset(),pci_parent_bus_reset(), etc. I like making the interfaces consistent.
Quoted 8 lines
What I'm asking here is why the "probe" argument exists for *any* of these interfaces and why
pci_probe_reset_function()exists.This is really more a question for Alex since it's a historical question, not anything directly related to your series. I'm not proposing *removing* the "probe" argument; I know it exists for a reason because I've asked about it before. But I forgot the answer, which makes me think a hint in the code would be useful.
Heh [1]
That might be what you're recalling, but in that case I was adding exported symbols that allowed probing bus vs slot reset because the scope of affected devices is different. My use case is testing whether the user owns all the affected devices, so it's really not a test-by-doing opportunity.
For these single-function scoped resets, as in the reply to [1] pci_probe_reset_function() isn't exported and the only caller is internal PCI code to determine whether to create the 'reset' sysfs attribute. Sure, as it exists today we could reset the device and test whether it worked to get that value, that's what vfio-pci does now before we give the device to the user, but the critical difference is that in the vfio case we always want to flush any state that might be leaked to the user and at device init time, doing so only invites issues.
This series obviously expands the scope of probing, we don't just want to know that there's at least one method available to us, but precisely which ones. It's rather impractical to try to reset a function a half dozen different ways on boot for the possibility that the admin might want to manipulate the reset order later. And oh gosh, if we don't cache the methods supported and re-test-by-doing when the attribute is written, let's just not go there. Thanks,
Alex
PATCH v7 2/8 PCI: Add new array for keeping track of ordering of reset methods
In reply to Amey NarkhedeIntroduce a new array reset_methods in struct pci_dev to keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.
--- drivers/pci/pci.c | 108 ++++++++++++++++++++++++++------------------ drivers/pci/pci.h | 8 +++- drivers/pci/probe.c | 5 +- include/linux/pci.h | 7 +++ 4 files changed, 81 insertions(+), 47 deletions(-)
drivers/pci/pci.c +65 -43
@@ -72,6 +72,14 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)msleep (delay); } +bool pci_reset_supported (struct pci_dev *dev) +{ +u8 null_reset_methods[PCI_RESET_METHODS_NUM ] = {0 }; + +return memcmp (null_reset_methods, + dev->reset_methods,PCI_RESET_METHODS_NUM ); +} +#ifdef CONFIG_PCI_DOMAINS int pci_domains_supported =1 ;#endif
@@ -5107,6 +5115,18 @@ static void pci_dev_restore(struct pci_dev *dev) err_handler->reset_done (dev); } +/* +* The ordering for functions in pci_reset_fn_methods is required for +* reset_methods byte array defined in struct pci_dev. +*/ +const struct pci_reset_fn_method pci_reset_fn_methods[] = { + { &pci_dev_specific_reset, .name ="device_specific" }, + { &pcie_reset_flr, .name ="flr" }, + { &pci_af_flr, .name ="af_flr" }, + { &pci_pm_reset, .name ="pm" }, + { &pci_reset_bus_function, .name ="bus" }, +}; +/** * __pci_reset_function_locked - reset a PCI device function while holding * the @dev mutex lock.
@@ -5129,65 +5149,67 @@ static void pci_dev_restore(struct pci_dev *dev)*/ int __pci_reset_function_locked (struct pci_dev *dev) { -int rc; +int i, rc = -ENOTTY ; +u8 prio;might_sleep (); -/* -* A reset method returns -ENOTTY if it doesn't support this device -* and we should try the next method. -* -* If it returns 0 (success), we're finished. If it returns any -* other error, we're also finished: this indicates that further -* reset mechanisms might be broken on the device. -*/ - rc =pci_dev_specific_reset (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pcie_reset_flr (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_af_flr (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_pm_reset (dev,0 ); -if (rc != -ENOTTY ) -return rc; -return pci_reset_bus_function (dev,0 ); +for (prio =PCI_RESET_METHODS_NUM ; prio; prio--) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (dev->reset_methods[i] == prio) { +/* +* A reset method returns -ENOTTY if it doesn't +* support this device and we should try the +* next method. +* +* If it returns 0 (success), we're finished. +* If it returns any other error, we're also +* finished: this indicates that further reset +* mechanisms might be broken on the device. +*/ + rc = pci_reset_fn_methods[i].reset_fn (dev,0 ); +if (rc != -ENOTTY ) +return rc; +break ; + } + } +if (i ==PCI_RESET_METHODS_NUM ) +break ; + } +return rc; }EXPORT_SYMBOL_GPL (__pci_reset_function_locked);/** -* pci_probe_reset_function - check whether the device can be safely reset -* @dev: PCI device to reset +* pci_init_reset_methods - check whether device can be safely reset +* and store supported reset mechanisms. +* @dev: PCI device to check for reset mechanisms * * Some devices allow an individual function to be reset without affecting * other functions in the same device. The PCI device must be responsive -* to PCI config space in order to use this function. +* to reads and writes to its PCI config space in order to use this function. * -* Returns 0 if the device function can be reset or negative if the -* device doesn't support resetting a single function. +* Stores reset mechanisms supported by device in reset_methods byte array +* which is a member of struct pci_dev. */ -int pci_probe_reset_function (struct pci_dev *dev) +void pci_init_reset_methods (struct pci_dev *dev) { -int rc; +int i, rc; +u8 prio =PCI_RESET_METHODS_NUM ; +u8 reset_methods[PCI_RESET_METHODS_NUM ] = {0 }; -might_sleep (); +BUILD_BUG_ON (ARRAY_SIZE (pci_reset_fn_methods) !=PCI_RESET_METHODS_NUM ); - rc =pci_dev_specific_reset (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pcie_reset_flr (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_af_flr (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_pm_reset (dev,1 ); -if (rc != -ENOTTY ) -return rc; +might_sleep (); -return pci_reset_bus_function (dev,1 ); +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { + rc = pci_reset_fn_methods[i].reset_fn (dev,1 ); +if (!rc) + reset_methods[i] = prio--; +else if (rc != -ENOTTY ) +break ; + } +memcpy (dev->reset_methods, reset_methods,sizeof (reset_methods)); }/**
drivers/pci/pci.h +7 -1
@@ -33,7 +33,7 @@ enum pci_mmap_api {int pci_mmap_fits (struct pci_dev *pdev,int resno,struct vm_area_struct *vmai,enum pci_mmap_api mmap_api); -int pci_probe_reset_function (struct pci_dev *dev); +void pci_init_reset_methods (struct pci_dev *dev);int pci_bridge_secondary_bus_reset (struct pci_dev *dev);int pci_bus_error_reset (struct pci_dev *dev);
@@ -606,6 +606,12 @@ struct pci_dev_reset_methods {int (*reset)(struct pci_dev *dev,int probe); }; +struct pci_reset_fn_method { +int (*reset_fn)(struct pci_dev *pdev,int probe); +char *name; +}; + +extern const struct pci_reset_fn_method pci_reset_fn_methods[];#ifdef CONFIG_PCI_QUIRKS int pci_dev_specific_reset (struct pci_dev *dev,int probe);#else
drivers/pci/probe.c +2 -3
@@ -2404,9 +2404,8 @@ static void pci_init_capabilities(struct pci_dev *dev)pci_rcec_init (dev);/* Root Complex Event Collector */ pcie_report_downtraining (dev); - -if (pci_probe_reset_function (dev) ==0 ) - dev->reset_fn =1 ; +pci_init_reset_methods (dev); + dev->reset_fn =pci_reset_supported (dev); }/*
include/linux/pci.h +7 -0
@@ -49,6 +49,8 @@ PCI_STATUS_SIG_TARGET_ABORT | \ PCI_STATUS_PARITY) +#define PCI_RESET_METHODS_NUM 5 +/* * The PCI interface treats multi-function devices as independent * devices. The slot/function address of each device is encoded
@@ -505,6 +507,10 @@ struct pci_dev {char *driver_override;/* Driver name to force a match */ unsigned long priv_flags;/* Private flags for the PCI driver */ +/* +* See pci_reset_fn_methods array in pci.c for ordering. +*/ +u8 reset_methods[PCI_RESET_METHODS_NUM ];/* Reset methods ordered by priority */ };static inline struct pci_dev *pci_physfn (struct pci_dev *dev)
@@ -1227,6 +1233,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,void pcie_print_link_status (struct pci_dev *dev);int pcie_reset_flr (struct pci_dev *dev,int probe);int pcie_flr (struct pci_dev *dev); +bool pci_reset_supported (struct pci_dev *dev);int __pci_reset_function_locked (struct pci_dev *dev);int pci_reset_function (struct pci_dev *dev);int pci_reset_function_locked (struct pci_dev *dev);
Signature · 1 line
2.31.1
Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods
Quoted 10 lines · Amey Narkhede
Introduce a new array
reset_methodsin structpci_devto keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.Reviewed-by Alex Williamson <[email protected]>Reviewed-by Raphael Norwitz <[email protected]>Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>
Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods
In reply to Amey Narkhede"Add new" in subject and below is slightly redundant.
Quoted 22 lines · Amey Narkhede
Introduce a new array
reset_methodsin structpci_devto keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.Reviewed-by Alex Williamson <[email protected]>Reviewed-by Raphael Norwitz <[email protected]>Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>--- drivers/pci/pci.c | 108 ++++++++++++++++++++++++++------------------ drivers/pci/pci.h | 8 +++- drivers/pci/probe.c | 5 +- include/linux/pci.h | 7 +++ 4 files changed, 81 insertions(+), 47 deletions(-)drivers/pci/pci.c +0
-0
drivers/pci/pci.c static void pci_dev_d3_sleep(struct pci_dev *dev) · line 80
Quoted 9 lines
drivers/pci/pci.c +6
-0@@ -72,6 +72,14 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)msleep (delay); } +bool pci_reset_supported (struct pci_dev *dev) +{ +u8 null_reset_methods[PCI_RESET_METHODS_NUM ] = {0 }; + +return memcmp (null_reset_methods, + dev->reset_methods,PCI_RESET_METHODS_NUM );
memcmp() doesn't actually return a bool. Either just return int and rely on the C "anything non-zero is true, zero is false" or convert the memcmp result to bool, i.e., something like:
if (memcmp(...) == 0)
return true;
return false;
drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5120
Quoted 12 lines
drivers/pci/pci.c +5
-0+} +#ifdef CONFIG_PCI_DOMAINS int pci_domains_supported =1 ;#endif @@ -5107,6 +5115,18 @@ static void pci_dev_restore(struct pci_dev *dev) err_handler->reset_done (dev); } +/* +* The ordering for functions in pci_reset_fn_methods is required for +* reset_methods byte array defined in struct pci_dev.
I'm not quite sure what this comment is telling me. What breaks if I change the order? If I add a new method, how do I know where to put it?
By reading the code, I infer that:
- Each dev has dev->reset_methods[PCI_RESET_METHODS_NUM]
- dev->reset_methods[i] corresponds to pci_reset_fn_methods[i]
- dev->reset_methods[i] == 0 means dev doesn't support that method
- Otherwise, dev->reset_methods[i] is a value in the range of [1, PCI_RESET_METHODS_NUM], and the higher the number, the higher the reset method priority
- The order in pci_reset_fn_methods[] determines the initial priority via pci_init_reset_methods(), but the priority can be changed via sysfs
Quoted 49 lines
drivers/pci/pci.c +17
-22+*/ +const struct pci_reset_fn_method pci_reset_fn_methods[] = { + { &pci_dev_specific_reset, .name ="device_specific" }, + { &pcie_reset_flr, .name ="flr" }, + { &pci_af_flr, .name ="af_flr" }, + { &pci_pm_reset, .name ="pm" }, + { &pci_reset_bus_function, .name ="bus" }, +}; +/** * __pci_reset_function_locked - reset a PCI device function while holding * the @dev mutex lock. @@ -5129,65 +5149,67 @@ static void pci_dev_restore(struct pci_dev *dev)*/ int __pci_reset_function_locked (struct pci_dev *dev) { -int rc; +int i, rc = -ENOTTY ; +u8 prio;might_sleep (); -/* -* A reset method returns -ENOTTY if it doesn't support this device -* and we should try the next method. -* -* If it returns 0 (success), we're finished. If it returns any -* other error, we're also finished: this indicates that further -* reset mechanisms might be broken on the device. -*/ - rc =pci_dev_specific_reset (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pcie_reset_flr (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_af_flr (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_pm_reset (dev,0 ); -if (rc != -ENOTTY ) -return rc; -return pci_reset_bus_function (dev,0 ); +for (prio =PCI_RESET_METHODS_NUM ; prio; prio--) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (dev->reset_methods[i] == prio) { +/* +* A reset method returns -ENOTTY if it doesn't +* support this device and we should try the
drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5172
Quoted 10 lines
drivers/pci/pci.c +10
-0+* next method. +* +* If it returns 0 (success), we're finished. +* If it returns any other error, we're also +* finished: this indicates that further reset +* mechanisms might be broken on the device. +*/ + rc = pci_reset_fn_methods[i].reset_fn (dev,0 ); +if (rc != -ENOTTY ) +return rc;
Maybe leave the comment outside the loop where it used to be so the text lines are longer and it's easier to read.
drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5179
Quoted 7 lines
drivers/pci/pci.c +7
-0+break ; + } + } +if (i ==PCI_RESET_METHODS_NUM ) +break ; + } +return rc;
I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:
const struct pci_reset_fn_method pci_reset_fn_methods[] = {
{ }, # 0
{ &pci_dev_specific_reset, .name = "device_specific" }, # 1
{ &pci_dev_acpi_reset, .name = "acpi" }, # 2
{ &pcie_reset_flr, .name = "flr" }, # 3
{ &pci_af_flr, .name = "af_flr" }, # 4
{ &pci_pm_reset, .name = "pm" }, # 5
{ &pci_reset_bus_function, .name = "bus" }, # 6
}; dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority order
dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means only pci_dev_specific_reset is supported
dev->reset_methods[] = [3, 5, 0, 0, 0, 0]
means pcie_reset_flr and pci_pm_reset are supported, in that priority order
Then we wouldn't need the nested loop and the return value would be easier to analyze:
for (i = 0; i < PCI_RESET_METHODS_NUM && (m = dev->reset_methods[i]); i++) {
rc = pci_reset_fn_methods[m].reset_fn(dev, 0);
if (rc == 0)
return 0;
if (rc != -ENOTTY)
return rc;
}
return -ENOTTY;pci_init_reset_methods() would be something like:
n = 0;
for (i = 1; i < PCI_RESET_METHODS_NUM; i++) {
rc = pci_reset_fn_methods[i].reset_fn(dev, 1);
if (!rc)
dev->reset_methods[n++] = i;
if (rc != -ENOTTY)
return;
}Quoted 104 lines
drivers/pci/pci.c +20
-21}EXPORT_SYMBOL_GPL (__pci_reset_function_locked);/** -* pci_probe_reset_function - check whether the device can be safely reset -* @dev: PCI device to reset +* pci_init_reset_methods - check whether device can be safely reset +* and store supported reset mechanisms. +* @dev: PCI device to check for reset mechanisms * * Some devices allow an individual function to be reset without affecting * other functions in the same device. The PCI device must be responsive -* to PCI config space in order to use this function. +* to reads and writes to its PCI config space in order to use this function. * -* Returns 0 if the device function can be reset or negative if the -* device doesn't support resetting a single function. +* Stores reset mechanisms supported by device in reset_methods byte array +* which is a member of struct pci_dev. */ -int pci_probe_reset_function (struct pci_dev *dev) +void pci_init_reset_methods (struct pci_dev *dev) { -int rc; +int i, rc; +u8 prio =PCI_RESET_METHODS_NUM ; +u8 reset_methods[PCI_RESET_METHODS_NUM ] = {0 }; -might_sleep (); +BUILD_BUG_ON (ARRAY_SIZE (pci_reset_fn_methods) !=PCI_RESET_METHODS_NUM ); - rc =pci_dev_specific_reset (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pcie_reset_flr (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_af_flr (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_pm_reset (dev,1 ); -if (rc != -ENOTTY ) -return rc; +might_sleep (); -return pci_reset_bus_function (dev,1 ); +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { + rc = pci_reset_fn_methods[i].reset_fn (dev,1 ); +if (!rc) + reset_methods[i] = prio--; +else if (rc != -ENOTTY ) +break ; + } +memcpy (dev->reset_methods, reset_methods,sizeof (reset_methods)); }/** drivers/pci/pci.h +7
-1@@ -33,7 +33,7 @@ enum pci_mmap_api {int pci_mmap_fits (struct pci_dev *pdev,int resno,struct vm_area_struct *vmai,enum pci_mmap_api mmap_api); -int pci_probe_reset_function (struct pci_dev *dev); +void pci_init_reset_methods (struct pci_dev *dev);int pci_bridge_secondary_bus_reset (struct pci_dev *dev);int pci_bus_error_reset (struct pci_dev *dev);@@ -606,6 +606,12 @@ struct pci_dev_reset_methods {int (*reset)(struct pci_dev *dev,int probe); }; +struct pci_reset_fn_method { +int (*reset_fn)(struct pci_dev *pdev,int probe); +char *name; +}; + +extern const struct pci_reset_fn_method pci_reset_fn_methods[];#ifdef CONFIG_PCI_QUIRKS int pci_dev_specific_reset (struct pci_dev *dev,int probe);#else drivers/pci/probe.c +2
-3@@ -2404,9 +2404,8 @@ static void pci_init_capabilities(struct pci_dev *dev)pci_rcec_init (dev);/* Root Complex Event Collector */ pcie_report_downtraining (dev); - -if (pci_probe_reset_function (dev) ==0 ) - dev->reset_fn =1 ; +pci_init_reset_methods (dev); + dev->reset_fn =pci_reset_supported (dev); }/* include/linux/pci.h +0
-0
include/linux/pci.h line 52
Quoted 4 lines
include/linux/pci.h +1
-0@@ -49,6 +49,8 @@ PCI_STATUS_SIG_TARGET_ABORT | \ PCI_STATUS_PARITY) +#define PCI_RESET_METHODS_NUM 5
I'm pretty sure this needs to be kept in sync with something, maybe ARRAY_SIZE(pci_reset_fn_methods)? We need some mechanism to enforce this, or at the very least, a comment. Oh, I see you have a BUILD_BUG_ON() in pci_init_reset_methods(). That's good, but a comment here would help, too.
This name should be something like "PCI_RESET_METHODS" or "PCI_NUM_RESET_METHODS". Putting "_NUM" at the end makes it sounds like we're referring to one specific method.
Quoted 15 lines
include/linux/pci.h +4
-0/* * The PCI interface treats multi-function devices as independent * devices. The slot/function address of each device is encoded @@ -505,6 +507,10 @@ struct pci_dev {char *driver_override;/* Driver name to force a match */ unsigned long priv_flags;/* Private flags for the PCI driver */ +/* +* See pci_reset_fn_methods array in pci.c for ordering. +*/ +u8 reset_methods[PCI_RESET_METHODS_NUM ];/* Reset methods ordered by priority */ };static inline struct pci_dev *pci_physfn (struct pci_dev *dev)
include/linux/pci.h u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev, · line 1236
Quoted 4 lines
include/linux/pci.h +1
-0@@ -1227,6 +1233,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,void pcie_print_link_status (struct pci_dev *dev);int pcie_reset_flr (struct pci_dev *dev,int probe);int pcie_flr (struct pci_dev *dev); +bool pci_reset_supported (struct pci_dev *dev);
This function isn't used outside drivers/pci/, so I'd rather have the prototype in drivers/pci/pci.h.
Quoted 6 lines
include/linux/pci.h +0
-0int __pci_reset_function_locked (struct pci_dev *dev);int pci_reset_function (struct pci_dev *dev);int pci_reset_function_locked (struct pci_dev *dev);Signature · 1 line
2.31.1
Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods
Quoted 68 lines · Bjorn Helgaas
"Add new" in subject and below is slightly redundant.
Quoted 22 lines · Amey Narkhede
Introduce a new array
reset_methodsin structpci_devto keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.Reviewed-by Alex Williamson <[email protected]>Reviewed-by Raphael Norwitz <[email protected]>Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>--- drivers/pci/pci.c | 108 ++++++++++++++++++++++++++------------------ drivers/pci/pci.h | 8 +++- drivers/pci/probe.c | 5 +- include/linux/pci.h | 7 +++ 4 files changed, 81 insertions(+), 47 deletions(-)drivers/pci/pci.c +0
-0drivers/pci/pci.c static void pci_dev_d3_sleep(struct pci_dev *dev) · line 80
Quoted 9 lines
drivers/pci/pci.c +6
-0@@ -72,6 +72,14 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)msleep (delay); } +bool pci_reset_supported (struct pci_dev *dev) +{ +u8 null_reset_methods[PCI_RESET_METHODS_NUM ] = {0 }; + +return memcmp (null_reset_methods, + dev->reset_methods,PCI_RESET_METHODS_NUM );
memcmp()doesn't actually return a bool. Either just return int and rely on the C "anything non-zero is true, zero is false" or convert the memcmp result to bool, i.e., something like:if (memcmp(...) == 0)
return true;return false;
drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5120
Quoted 12 lines
drivers/pci/pci.c +5
-0+} +#ifdef CONFIG_PCI_DOMAINS int pci_domains_supported =1 ;#endif @@ -5107,6 +5115,18 @@ static void pci_dev_restore(struct pci_dev *dev) err_handler->reset_done (dev); } +/* +* The ordering for functions in pci_reset_fn_methods is required for +* reset_methods byte array defined in struct pci_dev. I'm not quite sure what this comment is telling me. What breaks if I change the order? If I add a new method, how do I know where to put it?
By reading the code, I infer that:
- Each dev has dev->reset_methods[PCI_RESET_METHODS_NUM]
- dev->reset_methods[i] corresponds to pci_reset_fn_methods[i]
- dev->reset_methods[i] == 0 means dev doesn't support that method
Quoted 8 lines
- Otherwise, dev->reset_methods[i] is a value in the range of [1,
PCI_RESET_METHODS_NUM], and the higher the number, the higher the reset method priority- The order in pci_reset_fn_methods[] determines the initial priority via
pci_init_reset_methods(), but the priority can be changed via sysfs
Correct. I agree the comment is not clear. Adding new reset method won't break anything unless default order is changed and user has some assumptions from previous versions of kernel.
Quoted 111 lines
Quoted 49 lines
drivers/pci/pci.c +17
-22+*/ +const struct pci_reset_fn_method pci_reset_fn_methods[] = { + { &pci_dev_specific_reset, .name ="device_specific" }, + { &pcie_reset_flr, .name ="flr" }, + { &pci_af_flr, .name ="af_flr" }, + { &pci_pm_reset, .name ="pm" }, + { &pci_reset_bus_function, .name ="bus" }, +}; +/** * __pci_reset_function_locked - reset a PCI device function while holding * the @dev mutex lock. @@ -5129,65 +5149,67 @@ static void pci_dev_restore(struct pci_dev *dev)*/ int __pci_reset_function_locked (struct pci_dev *dev) { -int rc; +int i, rc = -ENOTTY ; +u8 prio;might_sleep (); -/* -* A reset method returns -ENOTTY if it doesn't support this device -* and we should try the next method. -* -* If it returns 0 (success), we're finished. If it returns any -* other error, we're also finished: this indicates that further -* reset mechanisms might be broken on the device. -*/ - rc =pci_dev_specific_reset (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pcie_reset_flr (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_af_flr (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_pm_reset (dev,0 ); -if (rc != -ENOTTY ) -return rc; -return pci_reset_bus_function (dev,0 ); +for (prio =PCI_RESET_METHODS_NUM ; prio; prio--) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (dev->reset_methods[i] == prio) { +/* +* A reset method returns -ENOTTY if it doesn't +* support this device and we should try the drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5172
Quoted 10 lines
drivers/pci/pci.c +10
-0+* next method. +* +* If it returns 0 (success), we're finished. +* If it returns any other error, we're also +* finished: this indicates that further reset +* mechanisms might be broken on the device. +*/ + rc = pci_reset_fn_methods[i].reset_fn (dev,0 ); +if (rc != -ENOTTY ) +return rc;Maybe leave the comment outside the loop where it used to be so the text lines are longer and it's easier to read.
drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5179
Quoted 7 lines
drivers/pci/pci.c +7
-0+break ; + } + } +if (i ==PCI_RESET_METHODS_NUM ) +break ; + } +return rc;I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:
const struct pci_reset_fn_method pci_reset_fn_methods[] = { { }, # 0 { &pci_dev_specific_reset, .name = "device_specific" }, # 1 { &pci_dev_acpi_reset, .name = "acpi" }, # 2 { &pcie_reset_flr, .name = "flr" }, # 3 { &pci_af_flr, .name = "af_flr" }, # 4 { &pci_pm_reset, .name = "pm" }, # 5 { &pci_reset_bus_function, .name = "bus" }, # 6 };dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority orderdev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means onlypci_dev_specific_resetis supporteddev->reset_methods[] = [3, 5, 0, 0, 0, 0]
meanspcie_reset_flrandpci_pm_resetare supported, in that priority orderThen we wouldn't need the nested loop and the return value would be easier to analyze:
for (i = 0; i < PCI_RESET_METHODS_NUM && (m = dev->reset_methods[i]); i++) { rc = pci_reset_fn_methods[m].reset_fn(dev, 0); if (rc == 0) return 0; if (rc != -ENOTTY) return rc; } return -ENOTTY;
pci_init_reset_methods()would be something like:
Quoted 9 lines
n = 0;
for (i = 1; i < PCI_RESET_METHODS_NUM; i++) {
rc = pci_reset_fn_methods[i].reset_fn(dev, 1);
if (!rc)
dev->reset_methods[n++] = i;
if (rc != -ENOTTY)
return;
}I had similar idea initially but couldn't put it in words nicely thanks for this. I'll update this.
[...]
Thanks,
Amey
Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods
Hi Bjorn,
Quoted 19 lines · Amey Narkhede
I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:
const struct pci_reset_fn_method pci_reset_fn_methods[] = { { }, # 0 { &pci_dev_specific_reset, .name = "device_specific" }, # 1 { &pci_dev_acpi_reset, .name = "acpi" }, # 2 { &pcie_reset_flr, .name = "flr" }, # 3 { &pci_af_flr, .name = "af_flr" }, # 4 { &pci_pm_reset, .name = "pm" }, # 5 { &pci_reset_bus_function, .name = "bus" }, # 6 };dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority order
Quoted 6 lines
dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means onlypci_dev_specific_resetis supporteddev->reset_methods[] = [3, 5, 0, 0, 0, 0]
meanspcie_reset_flrandpci_pm_resetare supported, in that priority order
What about keeping two bitmap fields 'resets_supported' and 'resets_enabled' in pci_dev object and mange it through sysfs and probe helper function. We can avoid two loops multiple paces and takes only 2Bytes of memory to keep track resets.
resets_supported ---> initialized during pci_dev setupresets_enabled ---> Exposed to userspace through sysfs by default set to resets_supported
include/linux/pci.h:
------------------------
/* Different types of PCI resets possible, lower number is higher priority */ #define PCI_RESET_METHOD_DEVSPEC 0
#define PCI_RESET_METHOD_ACPI 1
#define PCI_RESET_METHOD_FLR 2
#define PCI_RESET_METHOD_Af_FLR 3
#define PCI_RESET_METHOD_PM 4
#define PCI_RESET_METHOD_BUS 5
#define PCI_RESET_METHOD_MAX 6
struct pci_dev {
...
u8 resets_supported;
u8 resets_enabled;
};
static inline bool pci_reset_supported(struct pci_dev *dev) {
return !!(dev->resets_supported);
}
drivers/pci/pci.c:
--------------------
const struct pci_reset_fn_method pci_reset_fn_methods[PCI_RESET_METHOD_MAX] = { [PCI_RESET_METHOD_DEVSPEC] = { &pci_dev_specific_reset,
.name = "device_specific" }, [PCI_RESET_METHOD_ACPI] = { &pci_dev_acpi_reset, .name = "acpi" }, [PCI_RESET_METHOD_FLR] = { &pcie_reset_flr, .name = "flr" }, [PCI_RESET_METHOD_Af_FLR] = { &pci_af_flr, .name = "af_flr" }, [PCI_RESET_METHOD_PM] = { &pci_pm_reset, .name = "pm" }, [PCI_RESET_METHOD_BUS] = { &pci_reset_bus_function, .name = "bus" }
};
void pci_init_reset_methods(struct pci_dev *dev)
{
int i, rc;
BUILD_BUG_ON(ARRAY_SIZE(pci_reset_fn_methods) != PCI_RESET_METHOD_MAX); might_sleep();
for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
rc = pci_reset_fn_methods[i].reset_fn(dev, PCI_RESET_PROBE); if (!rc)
dev->resets_supported |= BIT(i);
else if (rc != -ENOTTY)
break;
}
dev->resets_enabled = dev->resets_supported;
}
int __pci_reset_function_locked(struct pci_dev *dev) {
int i, rc = -ENOTTY;
might_sleep();
for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
if (dev->resets_enabled & BIT(i)) {
rc = pci_reset_fn_methods[i].reset_fn(dev, 0); if (rc != -ENOTTY)
return rc;
}
}
if (rc == -ENOTTY)
pci_warn(dev, "No reset happened reason %s\n", !!dev->resets_supported ? "disabled by user" : "not supported");
return rc;
}
drivers/pci/pci-sysfs.c
----------------------------
static ssize_t reset_method_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) {
struct pci_dev *pdev = to_pci_dev(dev);
u8 resets_enabled = 0;
...
if (sysfs_streq(options, "default")) { pdev->resets_enabled = pdev->resets_supported; goto set_reset_methods;
}
while ((name = strsep(&options, ",")) != NULL) { if (sysfs_streq(name, "")) continue;
name = strim(name);
for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
if ((pdev->resets_supported & BIT(i)) && sysfs_streq(name, pci_reset_fn_methods[i].name)) { resets_enabled |= BIT(i);
break;
}
}
...
}
set_reset_methods:
kfree(options);
pdev->resets_enabled = resets_enabled;
return count;
}
static ssize_t reset_method_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct pci_dev *pdev = to_pci_dev(dev); ssize_t len = 0;
int i;
for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
if (pdev->resets_enabled & BIT(i))
len += sysfs_emit_at(buf, len, "%s%s", len ? "," : "", pci_reset_fn_methods[i].name); }
len += sysfs_emit_at(buf, len, len ? "\n" : "");
return len;
}
Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods
Quoted 161 lines · Shanker R Donthineni
Hi Bjorn,
Quoted 19 lines · Amey Narkhede
I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:
const struct pci_reset_fn_method pci_reset_fn_methods[] = { { }, # 0 { &pci_dev_specific_reset, .name = "device_specific" }, # 1 { &pci_dev_acpi_reset, .name = "acpi" }, # 2 { &pcie_reset_flr, .name = "flr" }, # 3 { &pci_af_flr, .name = "af_flr" }, # 4 { &pci_pm_reset, .name = "pm" }, # 5 { &pci_reset_bus_function, .name = "bus" }, # 6 };dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority orderQuoted 6 lines
dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means onlypci_dev_specific_resetis supporteddev->reset_methods[] = [3, 5, 0, 0, 0, 0]
meanspcie_reset_flrandpci_pm_resetare supported, in that priority orderWhat about keeping two bitmap fields '
resets_supported' and 'resets_enabled' inpci_devobject and mange it through sysfs and probe helper function. We can avoid two loops multiple paces and takes only 2Bytes of memory to keep track resets.resets_supported ---> initialized during
pci_devsetupresets_enabled---> Exposed to userspace through sysfs by default set toresets_supported
include/linux/pci.h:
------------------------
/* Different types of PCI resets possible, lower number is higher priority */ #definePCI_RESET_METHOD_DEVSPEC0
#define PCI_RESET_METHOD_ACPI 1
#define PCI_RESET_METHOD_FLR 2
#define PCI_RESET_METHOD_Af_FLR 3
#define PCI_RESET_METHOD_PM 4
#define PCI_RESET_METHOD_BUS 5
#define PCI_RESET_METHOD_MAX 6struct
pci_dev{
...
u8resets_supported;
u8resets_enabled;
};static inline bool
pci_reset_supported(struct pci_dev *dev){
return !!(dev->resets_supported);
}
drivers/pci/pci.c:
--------------------
const structpci_reset_fn_methodpci_reset_fn_methods[PCI_RESET_METHOD_MAX] = { [PCI_RESET_METHOD_DEVSPEC] = {&pci_dev_specific_reset,
.name = "device_specific" }, [PCI_RESET_METHOD_ACPI] = {&pci_dev_acpi_reset, .name = "acpi" }, [PCI_RESET_METHOD_FLR] = {&pcie_reset_flr, .name = "flr" }, [PCI_RESET_METHOD_Af_FLR] = {&pci_af_flr, .name = "af_flr" }, [PCI_RESET_METHOD_PM] = {&pci_pm_reset, .name = "pm" }, [PCI_RESET_METHOD_BUS] = {&pci_reset_bus_function, .name = "bus" }
};void
pci_init_reset_methods(struct pci_dev *dev)
{
int i, rc;
BUILD_BUG_ON(ARRAY_SIZE(pci_reset_fn_methods) != PCI_RESET_METHOD_MAX);might_sleep();for (i = 0; i <
PCI_RESET_METHOD_MAX; i++) {
rc = pci_reset_fn_methods[i].reset_fn(dev,PCI_RESET_PROBE); if (!rc)
dev->resets_supported|= BIT(i);
else if (rc !=-ENOTTY)
break;
}
dev->resets_enabled=dev->resets_supported;
}int
__pci_reset_function_locked(struct pci_dev *dev){
int i, rc =-ENOTTY;
might_sleep();for (i = 0; i <
PCI_RESET_METHOD_MAX; i++) {
if (dev->resets_enabled& BIT(i)) {
rc = pci_reset_fn_methods[i].reset_fn(dev, 0); if (rc !=-ENOTTY)
return rc;
}
}if (rc ==
-ENOTTY)
pci_warn(dev, "No reset happened reason %s\n", !!dev->resets_supported ? "disabled by user" : "not supported");return rc;
}
drivers/pci/pci-sysfs.c
----------------------------
staticssize_treset_method_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count){
structpci_dev*pdev =to_pci_dev(dev);
u8resets_enabled= 0;
...
if (sysfs_streq(options, "default")) {pdev->resets_enabled=pdev->resets_supported; gotoset_reset_methods;
}while ((name =
strsep(&options, ",")) != NULL) { if (sysfs_streq(name, "")) continue;
name = strim(name);for (i = 0; i <
PCI_RESET_METHOD_MAX; i++) {
if ((pdev->resets_supported& BIT(i)) &&sysfs_streq(name, pci_reset_fn_methods[i].name)) {resets_enabled|= BIT(i);
break;
}
}
...
}
set_reset_methods:
kfree(options);
pdev->resets_enabled=resets_enabled;
return count;
}static
ssize_treset_method_show(struct device *dev, struct device_attribute *attr, char *buf)
{
structpci_dev*pdev =to_pci_dev(dev);ssize_tlen = 0;
int i;for (i = 0; i <
PCI_RESET_METHOD_MAX; i++) {
if (pdev->resets_enabled& BIT(i))
len +=sysfs_emit_at(buf, len, "%s%s", len ? "," : "", pci_reset_fn_methods[i].name); }
len +=sysfs_emit_at(buf, len, len ? "\n" : "");
Quoted 3 lines
return len;
}
Thank you for the idea.
Actually that would be coming full circle because Alex, Raphael and I tried similar approach earlier while prototyping for v2 but this implementation does look better than what I had at that time.
Thanks,
Amey
Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods
Quoted 162 lines · Amey Narkhede
Quoted 161 lines · Shanker R Donthineni
Quoted 19 lines · Amey Narkhede
I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:
const struct pci_reset_fn_method pci_reset_fn_methods[] = { { }, # 0 { &pci_dev_specific_reset, .name = "device_specific" }, # 1 { &pci_dev_acpi_reset, .name = "acpi" }, # 2 { &pcie_reset_flr, .name = "flr" }, # 3 { &pci_af_flr, .name = "af_flr" }, # 4 { &pci_pm_reset, .name = "pm" }, # 5 { &pci_reset_bus_function, .name = "bus" }, # 6 };dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority orderQuoted 6 lines
dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means onlypci_dev_specific_resetis supporteddev->reset_methods[] = [3, 5, 0, 0, 0, 0]
meanspcie_reset_flrandpci_pm_resetare supported, in that priority orderWhat about keeping two bitmap fields '
resets_supported' and 'resets_enabled' inpci_devobject and mange it through sysfs and probe helper function. We can avoid two loops multiple paces and takes only 2Bytes of memory to keep track resets.resets_supported ---> initialized during
pci_devsetupresets_enabled---> Exposed to userspace through sysfs by default set toresets_supported
include/linux/pci.h:
------------------------
/* Different types of PCI resets possible, lower number is higher priority */ #definePCI_RESET_METHOD_DEVSPEC0
#define PCI_RESET_METHOD_ACPI 1
#define PCI_RESET_METHOD_FLR 2
#define PCI_RESET_METHOD_Af_FLR 3
#define PCI_RESET_METHOD_PM 4
#define PCI_RESET_METHOD_BUS 5
#define PCI_RESET_METHOD_MAX 6struct
pci_dev{
...
u8resets_supported;
u8resets_enabled;
};static inline bool
pci_reset_supported(struct pci_dev *dev){
return !!(dev->resets_supported);
}
drivers/pci/pci.c:
--------------------
const structpci_reset_fn_methodpci_reset_fn_methods[PCI_RESET_METHOD_MAX] = { [PCI_RESET_METHOD_DEVSPEC] = {&pci_dev_specific_reset,
.name = "device_specific" }, [PCI_RESET_METHOD_ACPI] = {&pci_dev_acpi_reset, .name = "acpi" }, [PCI_RESET_METHOD_FLR] = {&pcie_reset_flr, .name = "flr" }, [PCI_RESET_METHOD_Af_FLR] = {&pci_af_flr, .name = "af_flr" }, [PCI_RESET_METHOD_PM] = {&pci_pm_reset, .name = "pm" }, [PCI_RESET_METHOD_BUS] = {&pci_reset_bus_function, .name = "bus" }
};void
pci_init_reset_methods(struct pci_dev *dev)
{
int i, rc;
BUILD_BUG_ON(ARRAY_SIZE(pci_reset_fn_methods) != PCI_RESET_METHOD_MAX);might_sleep();for (i = 0; i <
PCI_RESET_METHOD_MAX; i++) {
rc = pci_reset_fn_methods[i].reset_fn(dev,PCI_RESET_PROBE); if (!rc)
dev->resets_supported|= BIT(i);
else if (rc !=-ENOTTY)
break;
}
dev->resets_enabled=dev->resets_supported;
}int
__pci_reset_function_locked(struct pci_dev *dev){
int i, rc =-ENOTTY;
might_sleep();for (i = 0; i <
PCI_RESET_METHOD_MAX; i++) {
if (dev->resets_enabled& BIT(i)) {
rc = pci_reset_fn_methods[i].reset_fn(dev, 0); if (rc !=-ENOTTY)
return rc;
}
}if (rc ==
-ENOTTY)
pci_warn(dev, "No reset happened reason %s\n", !!dev->resets_supported ? "disabled by user" : "not supported");return rc;
}
drivers/pci/pci-sysfs.c
----------------------------
staticssize_treset_method_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count){
structpci_dev*pdev =to_pci_dev(dev);
u8resets_enabled= 0;
...
if (sysfs_streq(options, "default")) {pdev->resets_enabled=pdev->resets_supported; gotoset_reset_methods;
}while ((name =
strsep(&options, ",")) != NULL) { if (sysfs_streq(name, "")) continue;
name = strim(name);for (i = 0; i <
PCI_RESET_METHOD_MAX; i++) {
if ((pdev->resets_supported& BIT(i)) &&sysfs_streq(name, pci_reset_fn_methods[i].name)) {resets_enabled|= BIT(i);
break;
}
}
...
}
set_reset_methods:
kfree(options);
pdev->resets_enabled=resets_enabled;
return count;
}static
ssize_treset_method_show(struct device *dev, struct device_attribute *attr, char *buf)
{
structpci_dev*pdev =to_pci_dev(dev);ssize_tlen = 0;
int i;for (i = 0; i <
PCI_RESET_METHOD_MAX; i++) {
if (pdev->resets_enabled& BIT(i))
len +=sysfs_emit_at(buf, len, "%s%s", len ? "," : "", pci_reset_fn_methods[i].name); }
len +=sysfs_emit_at(buf, len, len ? "\n" : "");
Quoted 7 lines
Quoted 3 lines
return len;
}Thank you for the idea.
Actually that would be coming full circle because Alex, Raphael and I tried similar approach earlier while prototyping for v2 but this implementation does look better than what I had at that time.
I thought part of the point of this series was to allow the user to change the *order* of reset types. I don't think we can control the ordering if we only keep a bit (or even two) per reset type.
PATCH v7 3/8 PCI: Remove reset_fn field from pci_dev
In reply to Amey Narkhedereset_fn field is used to indicate whether the device supports any reset mechanism or not. Remove the use of reset_fn in favor of new reset_methods array which can be used to keep track of all supported reset mechanisms of a device and their ordering.
The octeon driver is incorrectly usingreset_fn field to detect if the device supports FLR or not. Use pcie_reset_flr() to probe whether it supports FLR or not.
--- drivers/net/ethernet/cavium/liquidio/lio_vf_main.c | 2 +- drivers/pci/pci-sysfs.c | 2 +- drivers/pci/pci.c | 6 +++--- drivers/pci/probe.c | 1 - drivers/pci/quirks.c | 2 +- drivers/pci/remove.c | 1 - include/linux/pci.h | 1 - 7 files changed, 6 insertions(+), 9 deletions(-)
drivers/net/ethernet/cavium/liquidio/lio_vf_main.c +1 -1
@@ -526,7 +526,7 @@ static void octeon_destroy_resources(struct octeon_device *oct) oct->irq_name_storage =NULL ; }/* Soft reset the octeon device before exiting */ -if (oct->pci_dev->reset_fn) +if (!pcie_reset_flr (oct->pci_dev,1 ))octeon_pci_flr (oct);else cn23xx_vf_ask_pf_to_do_flr (oct);
drivers/pci/pci-sysfs.c +1 -1
@@ -1367,7 +1367,7 @@ static umode_t pci_dev_reset_attr_is_visible(struct kobject *kobj, {struct pci_dev *pdev =to_pci_dev (kobj_to_dev (kobj)); -if (!pdev->reset_fn) +if (!pci_reset_supported (pdev))return 0 ;return a->mode;
drivers/pci/pci.c +3 -3
@@ -5232,7 +5232,7 @@ int pci_reset_function(struct pci_dev *dev) {int rc; -if (!dev->reset_fn) +if (!pci_reset_supported (dev))return -ENOTTY ;pci_dev_lock (dev);
@@ -5268,7 +5268,7 @@ int pci_reset_function_locked(struct pci_dev *dev) {int rc; -if (!dev->reset_fn) +if (!pci_reset_supported (dev))return -ENOTTY ;pci_dev_save_and_disable (dev);
@@ -5291,7 +5291,7 @@ int pci_try_reset_function(struct pci_dev *dev) {int rc; -if (!dev->reset_fn) +if (!pci_reset_supported (dev))return -ENOTTY ;if (!pci_dev_trylock (dev))
drivers/pci/probe.c +0 -1
@@ -2405,7 +2405,6 @@ static void pci_init_capabilities(struct pci_dev *dev)pcie_report_downtraining (dev);pci_init_reset_methods (dev); - dev->reset_fn =pci_reset_supported (dev); }/*
drivers/pci/quirks.c +1 -1
@@ -5589,7 +5589,7 @@ static void quirk_reset_lenovo_thinkpad_p50_nvgpu(struct pci_dev *pdev)if (pdev->subsystem_vendor !=PCI_VENDOR_ID_LENOVO || pdev->subsystem_device !=0x222e || - !pdev->reset_fn) + !pci_reset_supported (pdev))return ;if (pci_enable_device_mem (pdev))
drivers/pci/remove.c +0 -1
@@ -19,7 +19,6 @@ static void pci_stop_dev(struct pci_dev *dev)pci_pme_active (dev, false);if (pci_dev_is_added (dev)) { - dev->reset_fn =0 ;device_release_driver (&dev->dev);pci_proc_detach_device (dev);
include/linux/pci.h +0 -1
@@ -429,7 +429,6 @@ struct pci_dev { unsigned int state_saved:1; unsigned int is_physfn:1; unsigned int is_virtfn:1; - unsigned int reset_fn:1; unsigned int is_hotplug_bridge:1; unsigned int shpc_managed:1; /* SHPC owned by shpchp */ unsigned int is_thunderbolt:1; /* Thunderbolt controller */
Signature · 1 line
2.31.1
Re: [PATCH v7 3/8] PCI: Remove reset_fn field from pci_dev
Quoted 9 lines · Amey Narkhede
reset_fnfield is used to indicate whether the device supports any reset mechanism or not. Remove the use ofreset_fnin favor of newreset_methodsarray which can be used to keep track of all supported reset mechanisms of a device and their ordering.The octeon driver is incorrectly using
reset_fnfield to detect if the device supports FLR or not. Usepcie_reset_flr()to probe whether it supports FLR or not.
Quoted 5 lines
Reviewed-by Alex Williamson <[email protected]>Reviewed-by Raphael Norwitz <[email protected]>Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>
PATCH v7 4/8 PCI/sysfs: Allow userspace to query and set device reset mechanism
In reply to Amey NarkhedeAdd reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.
--- Documentation/ABI/testing/sysfs-bus-pci | 16 ++++ drivers/pci/pci-sysfs.c | 118 ++++++++++++++++++++++++ 2 files changed, 134 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +16 -0
@@ -121,6 +121,22 @@ Description: child buses, and re-discover devices removed earlier from this part of the device tree. +What: /sys/bus/pci/devices/.../reset_method +Date: March 2021 +Contact: Amey Narkhede <[email protected]> +Description: + Some devices allow an individual function to be reset + without affecting other functions in the same slot. + For devices that have this support, a file named reset_method + will be present in sysfs. Reading this file will give names + of the device supported reset methods and their ordering. + Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. Writing empty string to this file will disable + ability to reset the device and writing "default" will return + to the original value. + What: /sys/bus/pci/devices/.../reset Date: July 2009 Contact: Michael S. Tsirkin <[email protected]>
drivers/pci/pci-sysfs.c +118 -0
130 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 7 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>
Quoted 52 lines
--- Documentation/ABI/testing/sysfs-bus-pci | 16 ++++ drivers/pci/pci-sysfs.c | 118 ++++++++++++++++++++++++ 2 files changed, 134 insertions(+)Documentation/ABI/testing/sysfs-bus-pci +16
-0@@ -121,6 +121,22 @@ Description: child buses, and re-discover devices removed earlier from this part of the device tree. +What: /sys/bus/pci/devices/.../reset_method +Date: March 2021 +Contact: Amey Narkhede <[email protected]> +Description: + Some devices allow an individual function to be reset + without affecting other functions in the same slot. + For devices that have this support, a file named reset_method + will be present in sysfs. Reading this file will give names + of the device supported reset methods and their ordering. + Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. Writing empty string to this file will disable + ability to reset the device and writing "default" will return + to the original value. + What: /sys/bus/pci/devices/.../reset Date: July 2009 Contact: Michael S. Tsirkin <[email protected]>drivers/pci/pci-sysfs.c +12
-0@@ -1334,6 +1334,123 @@ static const struct attribute_group pci_dev_rom_attr_group = { .is_bin_visible = pci_dev_rom_attr_is_visible, }; +static ssize_t reset_method_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +ssize_t len =0 ; +int i, prio; + +for (prio =PCI_RESET_METHODS_NUM ; prio; prio--) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (prio == pdev->reset_methods[i]) { + len += sysfs_emit_at(buf, len, "%s%s",
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1358
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ len ?"," :"" , + pci_reset_fn_methods[i].name); +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) +break ; + } +
Don't you still need to ensure you add the newline even if there are no reset methods set? If the len is zero why don't we need the newline?
Otherwise looks good.
Quoted 19 lines
drivers/pci/pci-sysfs.c +19
-0+if (len) + len +=sysfs_emit_at (buf, len,"\n" ); + +return len; +} + +static ssize_t reset_method_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +u8 reset_methods[PCI_RESET_METHODS_NUM ]; +struct pci_dev *pdev =to_pci_dev (dev); +u8 prio =PCI_RESET_METHODS_NUM ; +char *name, *options; +int i; + +if (count >= (PAGE_SIZE -1 )) +return -EINVAL ; +
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1387
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ options =kstrndup (buf, count,GFP_KERNEL ); +if (!options) +return -ENOMEM ; + +/* +* Initialize reset_method such that 0xff indicates +* supported but not currently enabled reset methods +* as we only use priority values which are within +* the range of PCI_RESET_FN_METHODS array size +*/
NIT: missing period in above comment.
Quoted 81 lines
drivers/pci/pci-sysfs.c +67
-0+for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = pdev->reset_methods[i] ?0xff :0 ; + +if (sysfs_streq (options,"" )) { +pci_warn (pdev,"All device reset methods disabled by user" ); + goto set_reset_methods; + } + +if (sysfs_streq (options,"default" )) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = reset_methods[i] ? prio-- :0 ; + goto set_reset_methods; + } + +while ((name =strsep (&options,"," )) !=NULL ) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) { + reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + } + +if (reset_methods[0 ] && + reset_methods[0 ] !=PCI_RESET_METHODS_NUM ) +pci_warn (pdev,"Device specific reset disabled/de-prioritized by user" ); + +set_reset_methods: +kfree (options); +memcpy (pdev->reset_methods, reset_methods,sizeof (reset_methods)); +return count; +} +static DEVICE_ATTR_RW (reset_method); + +static struct attribute *pci_dev_reset_method_attrs[] = { + &dev_attr_reset_method.attr, +NULL , +}; + +static umode_t pci_dev_reset_method_attr_is_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +struct pci_dev *pdev =to_pci_dev (kobj_to_dev (kobj)); + +if (!pci_reset_supported (pdev)) +return 0 ; + +return a->mode; +} + +static const struct attribute_group pci_dev_reset_method_attr_group = { + .attrs = pci_dev_reset_method_attrs, + .is_visible = pci_dev_reset_method_attr_is_visible, +}; +static ssize_t reset_store (struct device *dev,struct device_attribute *attr,const char *buf,size_t count) {@@ -1491,6 +1608,7 @@ const struct attribute_group *pci_dev_groups[] = { &pci_dev_config_attr_group, &pci_dev_rom_attr_group, &pci_dev_reset_attr_group, + &pci_dev_reset_method_attr_group, &pci_dev_vpd_attr_group, #ifdef CONFIG_DMI &pci_dev_smbios_attr_group,Signature · 1 line
2.31.1
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Hi Raphael,
Quoted 25 lines · Raphael Norwitz
Quoted 12 lines
drivers/pci/pci-sysfs.c +12
-0+static ssize_t reset_method_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +ssize_t len =0 ; +int i, prio; + +for (prio =PCI_RESET_METHODS_NUM ; prio; prio--) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (prio == pdev->reset_methods[i]) { + len += sysfs_emit_at(buf, len, "%s%s",drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1358
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ len ?"," :"" , + pci_reset_fn_methods[i].name); +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) +break ; + } +Don't you still need to ensure you add the newline even if there are no reset methods set? If the len is zero why don't we need the newline?
Quoted 2 lines
Otherwise looks good.
sysfs entry 'reset_method' will not be visible if there are no reset methods.
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Yes - I got this one wrong.
Nevermind, looks good. Just the punctuation NIT.
Quoted 34 lines · Shanker R Donthineni
Hi Raphael,
Quoted 25 lines · Raphael Norwitz
Quoted 12 lines
drivers/pci/pci-sysfs.c +12
-0+static ssize_t reset_method_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +ssize_t len =0 ; +int i, prio; + +for (prio =PCI_RESET_METHODS_NUM ; prio; prio--) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (prio == pdev->reset_methods[i]) { + len += sysfs_emit_at(buf, len, "%s%s",drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1358
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ len ?"," :"" , + pci_reset_fn_methods[i].name); +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) +break ; + } +Don't you still need to ensure you add the newline even if there are no reset methods set? If the len is zero why don't we need the newline?
Quoted 2 lines
Otherwise looks good.
sysfs entry '
reset_method' will not be visible if there are no reset methods.
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
In reply to Amey NarkhedeQuoted 7 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
In reply to Amey NarkhedeQuoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.
Rewrap to fill 75 columns (also apply to other patches if applicable, e.g., 3/8 looks like it could use it).
2/8 looks like it's missing a blank line between paragraphs.
Quoted 21 lines
Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 16 ++++ drivers/pci/pci-sysfs.c | 118 ++++++++++++++++++++++++ 2 files changed, 134 insertions(+)Documentation/ABI/testing/sysfs-bus-pci +5
-0@@ -121,6 +121,22 @@ Description: child buses, and re-discover devices removed earlier from this part of the device tree. +What: /sys/bus/pci/devices/.../reset_method +Date: March 2021 +Contact: Amey Narkhede <[email protected]> +Description: + Some devices allow an individual function to be reset
Documentation/ABI/testing/sysfs-bus-pci Description: · line 138
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +10
-0+ without affecting other functions in the same slot. + For devices that have this support, a file named reset_method + will be present in sysfs. Reading this file will give names + of the device supported reset methods and their ordering. + Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. Writing empty string to this file will disable + ability to reset the device and writing "default" will return + to the original value.
Rewrap to fill or add a blank line if "For devices ..." is supposed to start a new paragraph.
My guess is you intend reading to show the *currently enabled* reset methods, not the entire "supported" set? So if a user has disabled one of them, it no longer appears when you read the file?
Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-pci +1
-0+ What: /sys/bus/pci/devices/.../reset Date: July 2009 Contact: Michael S. Tsirkin <[email protected]>drivers/pci/pci-sysfs.c +11
-0@@ -1334,6 +1334,123 @@ static const struct attribute_group pci_dev_rom_attr_group = { .is_bin_visible = pci_dev_rom_attr_is_visible, }; +static ssize_t reset_method_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +ssize_t len =0 ; +int i, prio; + +for (prio =PCI_RESET_METHODS_NUM ; prio; prio--) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (prio == pdev->reset_methods[i]) {
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1357
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ len +=sysfs_emit_at (buf, len,"%s%s" , + len ?"," :"" , + pci_reset_fn_methods[i].name); +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) +break ; + }
I'm guessing that if you adopt the alternate reset_methods[] encoding, this nested loop becomes a single loop and "prio" goes away?
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (len) + len +=sysfs_emit_at (buf, len,"\n" ); + +return len; +}
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1373
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ +static ssize_t reset_method_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +u8 reset_methods[PCI_RESET_METHODS_NUM ]; +struct pci_dev *pdev =to_pci_dev (dev); +u8 prio =PCI_RESET_METHODS_NUM ; +char *name, *options; +int i;
Reorder decls with to_pci_dev(dev) first, then in order of use.
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (count >= (PAGE_SIZE -1 )) +return -EINVAL ; + + options =kstrndup (buf, count,GFP_KERNEL ); +if (!options)
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1389
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+return -ENOMEM ; + +/* +* Initialize reset_method such that 0xff indicates +* supported but not currently enabled reset methods +* as we only use priority values which are within +* the range of PCI_RESET_FN_METHODS array size +*/ +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = pdev->reset_methods[i] ?0xff :0 ;
I'm hoping the 0xff trick goes away with the alternate encoding?
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1394
Quoted 4 lines
drivers/pci/pci-sysfs.c +4
-0+if (sysfs_streq (options,"" )) { +pci_warn (pdev,"All device reset methods disabled by user" ); + goto set_reset_methods; + }
I think you can get this case out of the way early with no kstrndup(), no goto, etc.
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (sysfs_streq (options,"default" )) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = reset_methods[i] ? prio-- :0 ; + goto set_reset_methods; + }
If you use pci_init_reset_methods() here, you can also get this case out of the way early.
Quoted 13 lines
drivers/pci/pci-sysfs.c +13
-0+while ((name =strsep (&options,"," )) !=NULL ) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) { + reset_methods[i] = prio--; +break ; + } + }
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + } + +if (reset_methods[0 ] && + reset_methods[0 ] !=PCI_RESET_METHODS_NUM ) +pci_warn (pdev,"Device specific reset disabled/de-prioritized by user" );
Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.
Quoted 9 lines
drivers/pci/pci-sysfs.c +9
-0+set_reset_methods: +kfree (options); +memcpy (pdev->reset_methods, reset_methods,sizeof (reset_methods)); +return count; +} +static DEVICE_ATTR_RW (reset_method); + +static struct attribute *pci_dev_reset_method_attrs[] = { + &dev_attr_reset_method.attr,
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1444
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ NULL, +}; + +static umode_t pci_dev_reset_method_attr_is_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +struct pci_dev *pdev =to_pci_dev (kobj_to_dev (kobj)); + +if (!pci_reset_supported (pdev)) +return 0 ;
I think this _is_visible method is executed only once, at device_add()-time. That means if a device doesn't support any resets at that time, "reset_method" will not be visible, and there will be no way to ever enable a reset method at run-time. I assume that's OK; just double-checking.
Quoted 23 lines
drivers/pci/pci-sysfs.c +10
-0+ +return a->mode; +} + +static const struct attribute_group pci_dev_reset_method_attr_group = { + .attrs = pci_dev_reset_method_attrs, + .is_visible = pci_dev_reset_method_attr_is_visible, +}; +static ssize_t reset_store (struct device *dev,struct device_attribute *attr,const char *buf,size_t count) {@@ -1491,6 +1608,7 @@ const struct attribute_group *pci_dev_groups[] = { &pci_dev_config_attr_group, &pci_dev_rom_attr_group, &pci_dev_reset_attr_group, + &pci_dev_reset_method_attr_group, &pci_dev_vpd_attr_group, #ifdef CONFIG_DMI &pci_dev_smbios_attr_group,Signature · 1 line
2.31.1
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 130 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Rewrap to fill 75 columns (also apply to other patches if applicable, e.g., 3/8 looks like it could use it).
2/8 looks like it's missing a blank line between paragraphs.
Quoted 21 lines
Co-developed-by Alex Williamson <[email protected]>Signed-off-by Alex Williamson <[email protected]>Signed-off-by Amey Narkhede <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 16 ++++ drivers/pci/pci-sysfs.c | 118 ++++++++++++++++++++++++ 2 files changed, 134 insertions(+)Documentation/ABI/testing/sysfs-bus-pci +5
-0@@ -121,6 +121,22 @@ Description: child buses, and re-discover devices removed earlier from this part of the device tree. +What: /sys/bus/pci/devices/.../reset_method +Date: March 2021 +Contact: Amey Narkhede <[email protected]> +Description: + Some devices allow an individual function to be resetDocumentation/ABI/testing/sysfs-bus-pci Description: · line 138
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +10
-0+ without affecting other functions in the same slot. + For devices that have this support, a file named reset_method + will be present in sysfs. Reading this file will give names + of the device supported reset methods and their ordering. + Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. Writing empty string to this file will disable + ability to reset the device and writing "default" will return + to the original value.Rewrap to fill or add a blank line if "For devices ..." is supposed to start a new paragraph.
My guess is you intend reading to show the *currently enabled* reset methods, not the entire "supported" set? So if a user has disabled one of them, it no longer appears when you read the file?
Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-pci +1
-0+ What: /sys/bus/pci/devices/.../reset Date: July 2009 Contact: Michael S. Tsirkin <[email protected]>drivers/pci/pci-sysfs.c +11
-0@@ -1334,6 +1334,123 @@ static const struct attribute_group pci_dev_rom_attr_group = { .is_bin_visible = pci_dev_rom_attr_is_visible, }; +static ssize_t reset_method_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +ssize_t len =0 ; +int i, prio; + +for (prio =PCI_RESET_METHODS_NUM ; prio; prio--) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (prio == pdev->reset_methods[i]) {drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1357
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ len +=sysfs_emit_at (buf, len,"%s%s" , + len ?"," :"" , + pci_reset_fn_methods[i].name); +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) +break ; + }I'm guessing that if you adopt the alternate reset_methods[] encoding, this nested loop becomes a single loop and "prio" goes away?
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (len) + len +=sysfs_emit_at (buf, len,"\n" ); + +return len; +}drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1373
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ +static ssize_t reset_method_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +u8 reset_methods[PCI_RESET_METHODS_NUM ]; +struct pci_dev *pdev =to_pci_dev (dev); +u8 prio =PCI_RESET_METHODS_NUM ; +char *name, *options; +int i;Reorder decls with
to_pci_dev(dev)first, then in order of use.Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (count >= (PAGE_SIZE -1 )) +return -EINVAL ; + + options =kstrndup (buf, count,GFP_KERNEL ); +if (!options)drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1389
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+return -ENOMEM ; + +/* +* Initialize reset_method such that 0xff indicates +* supported but not currently enabled reset methods +* as we only use priority values which are within +* the range of PCI_RESET_FN_METHODS array size +*/ +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = pdev->reset_methods[i] ?0xff :0 ;I'm hoping the
0xfftrick goes away with the alternate encoding?drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1394
Quoted 4 lines
drivers/pci/pci-sysfs.c +4
-0+if (sysfs_streq (options,"" )) { +pci_warn (pdev,"All device reset methods disabled by user" ); + goto set_reset_methods; + }I think you can get this case out of the way early with no
kstrndup(), no goto, etc.
Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (sysfs_streq (options,"default" )) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = reset_methods[i] ? prio-- :0 ; + goto set_reset_methods; + }If you use
pci_init_reset_methods()here, you can also get this case out of the way early.
The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,
# cat reset_methods flr,bus # dev->reset_methods = [3, 5, 0, ...] # echo bus > reset_methods # dev->reset_methods = [5, 0, 0, ...] # cat reset_methods bus
Now if an user wants to enable flr
# echo flr > reset_methods # dev->reset_methods = [3, 0, 0, ...] OR # echo bus,flr > reset_methods # dev->reset_methods = [5, 3, 0, ...]
either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.
Quoted 19 lines
Quoted 19 lines
drivers/pci/pci-sysfs.c +19
-0+while ((name =strsep (&options,"," )) !=NULL ) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) { + reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }
Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424
Quoted 4 lines
drivers/pci/pci-sysfs.c +4
-0+ +if (reset_methods[0 ] && + reset_methods[0 ] !=PCI_RESET_METHODS_NUM ) +pci_warn (pdev,"Device specific reset disabled/de-prioritized by user" );Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.
I think generally presence of device specific reset method means other methods are potentially broken. Is it okay to skip this?
Quoted 17 lines
Quoted 17 lines
drivers/pci/pci-sysfs.c +17
-0+set_reset_methods: +kfree (options); +memcpy (pdev->reset_methods, reset_methods,sizeof (reset_methods)); +return count; +} +static DEVICE_ATTR_RW (reset_method); + +static struct attribute *pci_dev_reset_method_attrs[] = { + &dev_attr_reset_method.attr, +NULL , +}; + +static umode_t pci_dev_reset_method_attr_is_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +struct pci_dev *pdev =to_pci_dev (kobj_to_dev (kobj)); +
Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1444
Quoted 2 lines
drivers/pci/pci-sysfs.c +2
-0+if (!pci_reset_supported (pdev)) +return 0 ;I think this
_is_visiblemethod is executed only once, at device_add()-time. That means if a device doesn't support any resets at that time, "reset_method" will not be visible, and there will be no way to ever enable a reset method at run-time. I assume that's OK; just double-checking.
Correct. Its similar to exisitng reset_fn bitfield which is removed in this patch series.
[...]
Thanks,
Amey
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (sysfs_streq (options,"default" )) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = reset_methods[i] ? prio-- :0 ; + goto set_reset_methods; + }If you use
pci_init_reset_methods()here, you can also get this case out of the way early.The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,
# cat reset_methods flr,bus # dev->reset_methods = [3, 5, 0, ...] # echo bus > reset_methods # dev->reset_methods = [5, 0, 0, ...] # cat reset_methods bus
Quoted 8 lines
Now if an user wants to enable flr
# echo flr > reset_methods # dev->reset_methods = [3, 0, 0, ...] OR # echo bus,flr > reset_methods # dev->reset_methods = [5, 3, 0, ...]either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to
reset_methodattribute.
Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.
If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with -ENOTTY? Is that a problem?
Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
drivers/pci/pci-sysfs.c +19
-0+while ((name =strsep (&options,"," )) !=NULL ) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) { + reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424
Quoted 4 lines
drivers/pci/pci-sysfs.c +4
-0+ +if (reset_methods[0 ] && + reset_methods[0 ] !=PCI_RESET_METHODS_NUM ) +pci_warn (pdev,"Device specific reset disabled/de-prioritized by user" );Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.
Quoted 2 lines
I think generally presence of device specific reset method means other methods are potentially broken. Is it okay to skip this?
We might want a warning at reset-time if all the methods failed, because that means we may leak state between users. Maybe we also want one here, if *all* reset methods are disabled. I don't really like special treatment of device-specific methods here because it depends on the assumption that "device-specific means all other resets are broken." That's hard to maintain.
Bjorn
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (sysfs_streq (options,"default" )) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = reset_methods[i] ? prio-- :0 ; + goto set_reset_methods; + }If you use
pci_init_reset_methods()here, you can also get this case out of the way early.The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,
# cat reset_methods flr,bus # dev->reset_methods = [3, 5, 0, ...] # echo bus > reset_methods # dev->reset_methods = [5, 0, 0, ...] # cat reset_methods busQuoted 8 lines
Now if an user wants to enable flr
# echo flr > reset_methods # dev->reset_methods = [3, 0, 0, ...] OR # echo bus,flr > reset_methods # dev->reset_methods = [5, 3, 0, ...]either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to
reset_methodattribute.
Quoted 8 lines
Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.
If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with
-ENOTTY? Is that a problem?
I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.
Alternate way of doing this is using 2 bitmaps as outlined here by Shanker https://marc.info/?l=linux-kernel&m=162428773101702&w=2
Quoted 31 lines
Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
drivers/pci/pci-sysfs.c +19
-0+while ((name =strsep (&options,"," )) !=NULL ) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) { + reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424
Quoted 4 lines
drivers/pci/pci-sysfs.c +4
-0+ +if (reset_methods[0 ] && + reset_methods[0 ] !=PCI_RESET_METHODS_NUM ) +pci_warn (pdev,"Device specific reset disabled/de-prioritized by user" );Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.
Quoted 2 lines
I think generally presence of device specific reset method means other methods are potentially broken. Is it okay to skip this?
Quoted 8 lines
We might want a warning at reset-time if all the methods failed, because that means we may leak state between users. Maybe we also want one here, if *all* reset methods are disabled. I don't really like special treatment of device-specific methods here because it depends on the assumption that "device-specific means all other resets are broken." That's hard to maintain.
Bjorn
Makes sense. I'll update this.
Thanks,
Amey
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (sysfs_streq (options,"default" )) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = reset_methods[i] ? prio-- :0 ; + goto set_reset_methods; + }If you use
pci_init_reset_methods()here, you can also get this case out of the way early.The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,
# cat reset_methods flr,bus # dev->reset_methods = [3, 5, 0, ...] # echo bus > reset_methods # dev->reset_methods = [5, 0, 0, ...] # cat reset_methods busQuoted 8 lines
Now if an user wants to enable flr
# echo flr > reset_methods # dev->reset_methods = [3, 0, 0, ...] OR # echo bus,flr > reset_methods # dev->reset_methods = [5, 3, 0, ...]either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to
reset_methodattribute.Quoted 8 lines
Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.
If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with
-ENOTTY? Is that a problem?
Quoted 8 lines
I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.
I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "reset_method" sets the "reset methods and their ordering."
It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "reset_method" to enable reset X, can we just probe reset X to see if it's supported?
Bjorn
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 58 lines · Bjorn Helgaas
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400
Quoted 5 lines
drivers/pci/pci-sysfs.c +5
-0+if (sysfs_streq (options,"default" )) { +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) + reset_methods[i] = reset_methods[i] ? prio-- :0 ; + goto set_reset_methods; + }If you use
pci_init_reset_methods()here, you can also get this case out of the way early.The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,
# cat reset_methods flr,bus # dev->reset_methods = [3, 5, 0, ...] # echo bus > reset_methods # dev->reset_methods = [5, 0, 0, ...] # cat reset_methods busQuoted 8 lines
Now if an user wants to enable flr
# echo flr > reset_methods # dev->reset_methods = [3, 0, 0, ...] OR # echo bus,flr > reset_methods # dev->reset_methods = [5, 3, 0, ...]either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to
reset_methodattribute.Quoted 8 lines
Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.
If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with
-ENOTTY? Is that a problem?Quoted 8 lines
I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.
I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "
reset_method" sets the "reset methods and their ordering."
Quoted 5 lines
It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "
reset_method" to enable reset X, can we just probe reset X to see if it's supported?Bjorn
Although final result is same whether user writes a supported reset method or their ordering that is,
# echo bus > reset_methods and # echo bus,flr > reset_methods
are the same but in the first version, users don't have to explicitly set the ordering if they just want to perform bus reset. Current implementation allows the flexibility for switching between first and second option.
Does this address your doubt?
Thanks,
Amey
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 65 lines · Amey Narkhede
Quoted 53 lines · Bjorn Helgaas
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Quoted 9 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (sysfs_streq(options, "default")) { + for (i = 0; i < PCI_RESET_METHODS_NUM; i++) + reset_methods[i] = reset_methods[i] ? prio-- : 0; + goto set_reset_methods; + }If you use
pci_init_reset_methods()here, you can also get this case out of the way early.The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,
# cat reset_methods flr,bus # dev->reset_methods = [3, 5, 0, ...] # echo bus > reset_methods # dev->reset_methods = [5, 0, 0, ...] # cat reset_methods busQuoted 8 lines
Now if an user wants to enable flr
# echo flr > reset_methods # dev->reset_methods = [3, 0, 0, ...] OR # echo bus,flr > reset_methods # dev->reset_methods = [5, 3, 0, ...]either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to
reset_methodattribute.Quoted 8 lines
Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.
If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with
-ENOTTY? Is that a problem?Quoted 8 lines
I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.
Quoted 8 lines
I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "
reset_method" sets the "reset methods and their ordering."It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "
reset_method" to enable reset X, can we just probe reset X to see if it's supported?Although final result is same whether user writes a supported reset method or their ordering that is,
Quoted 8 lines
# echo bus > reset_methods and # echo bus,flr > reset_methodsare the same but in the first version, users don't have to explicitly set the ordering if they just want to perform bus reset. Current implementation allows the flexibility for switching between first and second option.
Sorry, I can't quite make sense of the above.
Your doc implies the following are different:
# echo bus,flr > reset_methods # echo flr,bus > reset_methods
Are they? If you don't need to provide control over the order of trying resets, this can all be simplified quite a bit.
Bjorn
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 79 lines · Bjorn Helgaas
Quoted 65 lines · Amey Narkhede
Quoted 53 lines · Bjorn Helgaas
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Quoted 9 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (sysfs_streq(options, "default")) { + for (i = 0; i < PCI_RESET_METHODS_NUM; i++) + reset_methods[i] = reset_methods[i] ? prio-- : 0; + goto set_reset_methods; + }If you use
pci_init_reset_methods()here, you can also get this case out of the way early.The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,
# cat reset_methods flr,bus # dev->reset_methods = [3, 5, 0, ...] # echo bus > reset_methods # dev->reset_methods = [5, 0, 0, ...] # cat reset_methods busQuoted 8 lines
Now if an user wants to enable flr
# echo flr > reset_methods # dev->reset_methods = [3, 0, 0, ...] OR # echo bus,flr > reset_methods # dev->reset_methods = [5, 3, 0, ...]either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to
reset_methodattribute.Quoted 8 lines
Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.
If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with
-ENOTTY? Is that a problem?Quoted 8 lines
I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.
Quoted 8 lines
I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "
reset_method" sets the "reset methods and their ordering."It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "
reset_method" to enable reset X, can we just probe reset X to see if it's supported?Although final result is same whether user writes a supported reset method or their ordering that is,
Quoted 8 lines
# echo bus > reset_methods and # echo bus,flr > reset_methodsare the same but in the first version, users don't have to explicitly set the ordering if they just want to perform bus reset. Current implementation allows the flexibility for switching between first and second option.
Sorry, I can't quite make sense of the above.
Your doc implies the following are different:
Quoted 7 lines
# echo bus,flr > reset_methods # echo flr,bus > reset_methodsAre they? If you don't need to provide control over the order of trying resets, this can all be simplified quite a bit.
Bjorn
The v1 of the patch series allowed only single reset method to be written instead of ordering of all supported reset methods. With your example, current implementation allows both writing single value and list of supported reset methods.
# echo bus > reset_methods and # echo bus,flr > reset_methods
OR
# echo flr > reset_methods and echo flr,bus > reset_methods
Its more of a preference than a functional point. Ultimately the __pci_reset_function_locked() function will only perform 'bus' reset in this example because we make sure 'reset_methods' file only contains valid device supported reset methods all the time so __pci_reset_function_locked() won't go into -ENOTTY case but with new encoding theres no way to make sure reset_methodssysfs attribute will contain valid supported reset methods all the time because of the reset methods can be masked implicitly if user uses first option of writing only single value.
My point is current implementation allows more flexibility for the user.
Thanks,
Amey
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 96 lines · Amey Narkhede
Quoted 79 lines · Bjorn Helgaas
Quoted 65 lines · Amey Narkhede
Quoted 53 lines · Bjorn Helgaas
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.Quoted 9 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (sysfs_streq(options, "default")) { + for (i = 0; i < PCI_RESET_METHODS_NUM; i++) + reset_methods[i] = reset_methods[i] ? prio-- : 0; + goto set_reset_methods; + }If you use
pci_init_reset_methods()here, you can also get this case out of the way early.The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,
# cat reset_methods flr,bus # dev->reset_methods = [3, 5, 0, ...] # echo bus > reset_methods # dev->reset_methods = [5, 0, 0, ...] # cat reset_methods busQuoted 8 lines
Now if an user wants to enable flr
# echo flr > reset_methods # dev->reset_methods = [3, 0, 0, ...] OR # echo bus,flr > reset_methods # dev->reset_methods = [5, 3, 0, ...]either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to
reset_methodattribute.Quoted 8 lines
Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.
If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with
-ENOTTY? Is that a problem?Quoted 8 lines
I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.
Quoted 8 lines
I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "
reset_method" sets the "reset methods and their ordering."It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "
reset_method" to enable reset X, can we just probe reset X to see if it's supported?Although final result is same whether user writes a supported reset method or their ordering that is,
Quoted 8 lines
# echo bus > reset_methods and # echo bus,flr > reset_methodsare the same but in the first version, users don't have to explicitly set the ordering if they just want to perform bus reset. Current implementation allows the flexibility for switching between first and second option.
Sorry, I can't quite make sense of the above.
Your doc implies the following are different:
Quoted 7 lines
# echo bus,flr > reset_methods # echo flr,bus > reset_methodsAre they? If you don't need to provide control over the order of trying resets, this can all be simplified quite a bit.
Bjorn
The v1 of the patch series allowed only single reset method to be written instead of ordering of all supported reset methods. With your example, current implementation allows both writing single value and list of supported reset methods.
# echo bus > reset_methods and # echo bus,flr > reset_methods
Quoted 6 lines
OR
# echo flr > reset_methods and echo flr,bus > reset_methods
# echo flr,bus and echo bus,flr are different.
Quoted 5 lines
Its more of a preference than a functional point. Ultimately the
__pci_reset_function_locked()function will only perform 'bus' reset in this example because we make sure 'reset_methods' file only contains valid device supported reset methods all the time so__pci_reset_function_locked()won't go into-ENOTTYcase but with new
Oops I'm wrong here. __pci_reset_function_locked() can return -ENOTTY and follow through if a reset fails.
Rest of the point should hold.
Quoted 9 lines
encoding theres no way to make sure
reset_methodssysfs attribute will contain valid supported reset methods all the time because of the reset methods can be masked implicitly if user uses first option of writing only single value.My point is current implementation allows more flexibility for the user.
Thanks,
Amey
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
In reply to Bjorn HelgaasQuoted 34 lines · On Mon, 21 Jun 2021 08:01:35 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 30 lines · Amey Narkhede
Quoted 20 lines · Bjorn Helgaas
Quoted 19 lines · Amey Narkhede
drivers/pci/pci-sysfs.c +19
-0+while ((name =strsep (&options,"," )) !=NULL ) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) { + reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }Quoted 9 lines
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424
Quoted 4 lines
drivers/pci/pci-sysfs.c +4
-0+ +if (reset_methods[0 ] && + reset_methods[0 ] !=PCI_RESET_METHODS_NUM ) +pci_warn (pdev,"Device specific reset disabled/de-prioritized by user" );Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.
Quoted 2 lines
I think generally presence of device specific reset method means other methods are potentially broken. Is it okay to skip this?
Quoted 6 lines
We might want a warning at reset-time if all the methods failed, because that means we may leak state between users. Maybe we also want one here, if *all* reset methods are disabled. I don't really like special treatment of device-specific methods here because it depends on the assumption that "device-specific means all other resets are broken." That's hard to maintain.
I'd say the device specific reset is special. The device itself can support a number of resets and they're theoretically all equivalent, it's a policy decision which to use. But the device specific reset is a software provided reset. Someone has specifically gone to the trouble to create a reset mechanism that is in some way better than the other methods. Not using that one by default sure feels like something worthy of leaving a breadcrumb in dmesg for debugging. Thanks,
Alex
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
In reply to Amey NarkhedeQuoted 18 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.drivers/pci/pci-sysfs.c +13
-0+ Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. + while ((name = strsep(&options, ",")) != NULL) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) {
drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }
Asking again since we didn't get this clarified before. The above tells me that "reset_methods" allows the user to control the *order* in which we try reset methods.
Consider the following two uses:
(1) # echo bus,flr > reset_methods
(2) # echo flr,bus > reset_methods
Do these have the same effect or not?
If "reset_methods" allows control over the order, I expect them to be
different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 34 lines · Bjorn Helgaas
Quoted 18 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.drivers/pci/pci-sysfs.c +13
-0+ Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. + while ((name = strsep(&options, ",")) != NULL) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) {drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }Asking again since we didn't get this clarified before. The above tells me that "
reset_methods" allows the user to control the *order* in which we try reset methods.
Quoted 8 lines
Consider the following two uses:
(1) # echo bus,flr >
reset_methods(2) # echo flr,bus >
reset_methodsDo these have the same effect or not?
They have different effect.
Quoted 3 lines
If "
reset_methods" allows control over the order, I expect them to be
different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.
Exactly you are right.
Now the point I was presenting was with new encoding we have to write list of *all of the supported reset methods* in order for example, in above example flr,bus or bus,flr. We can't just write 'flr' or 'bus' then switch back to writing flr,bus/bus,flr(these have different effect as mentioned earlier).
Basically with new encoding an user can't write subset of reset methods they have to write list of *all* supported methods everytime.
Thanks,
Amey
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 37 lines · Amey Narkhede
Quoted 36 lines · Bjorn Helgaas
Quoted 18 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.drivers/pci/pci-sysfs.c +13
-0+ Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. + while ((name = strsep(&options, ",")) != NULL) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) {drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }Asking again since we didn't get this clarified before. The above tells me that "
reset_methods" allows the user to control the *order* in which we try reset methods.Consider the following two uses:
Quoted 7 lines
Quoted 6 lines
(1) # echo bus,flr >
reset_methods(2) # echo flr,bus >
reset_methodsDo these have the same effect or not?
They have different effect.
I asked about this because Shanker's idea [1] of using two bitmaps only keeps track of which resets are *enabled*. It does not keep track of the *ordering*. Since you want to control the ordering, I think we need more state than just the supported/enabled bitmaps.
Quoted 5 lines
Quoted 4 lines
If "
reset_methods" allows control over the order, I expect them to be different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.
Quoted 7 lines
Exactly you are right.
Now the point I was presenting was with new encoding we have to write list of *all of the supported reset methods* in order for example, in above example flr,bus or bus,flr. We can't just write 'flr' or 'bus' then switch back to writing flr,bus/bus,flr (these have different effect as mentioned earlier).
It sounds like you're saying this sequence can't work:
# echo flr > reset_methods # echo bus,flr > reset_methods
But I'm afraid you'll have to walk me through the reasons why this can't be made to work.
Quoted 3 lines
Basically with new encoding an user can't write subset of reset methods they have to write list of *all* supported methods everytime.
Why does the user have to write all supported methods? Is that to preserve the fact that "cat reset_methods" always shows all the supported methods so the user knows what's available?
I'm wondering why we can't do something like this (pidgin code):
if (option == "default") {
pci_init_reset_methods(dev);
return;
}
n = 0;
foreach method in option {
i = lookup_reset_method(method);
if (pci_reset_methods[i].reset_fn(dev, PROBE) == 0)
dev->reset_methods[n++] = i; # method i supported
}
dev->reset_methods[n++] = 0; # end of supported methodsIf we did something like the above, the user could always find the list of all methods supported by a device by doing this:
# echo default > reset_methods # cat reset_methods
Yes, this does call the "probe" methods several times. I don't think that's necessarily a problem.
Bjorn
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 19 lines · Bjorn Helgaas
Why does the user have to write all supported methods? Is that to preserve the fact that "cat
reset_methods" always shows all the supported methods so the user knows what's available?I'm wondering why we can't do something like this (pidgin code):
if (option == "default") { pci_init_reset_methods(dev); return; } n = 0; foreach method in option { i = lookup_reset_method(method); if (pci_reset_methods[i].reset_fn(dev, PROBE) == 0) dev->reset_methods[n++] = i; # method i supported } dev->reset_methods[n++] = 0; # end of supported methods
Quoted 8 lines
If we did something like the above, the user could always find the list of all methods supported by a device by doing this:
# echo default > reset_methods # cat reset_methodsYes, this does call the "probe" methods several times. I don't think that's necessarily a problem.
Agree, I don't think admin/user will change reset methods frequently and no side effects or performance impacts on probing multiple times.
We should support enabling partially ordered reset methods and restore default methods either by re-probing resets or remember supported resets in pci_dev.
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
In reply to Bjorn HelgaasQuoted 64 lines · Bjorn Helgaas
Quoted 37 lines · Amey Narkhede
Quoted 36 lines · Bjorn Helgaas
Quoted 18 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.drivers/pci/pci-sysfs.c +13
-0+ Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. + while ((name = strsep(&options, ",")) != NULL) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) {drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }Asking again since we didn't get this clarified before. The above tells me that "
reset_methods" allows the user to control the *order* in which we try reset methods.Consider the following two uses:
Quoted 7 lines
Quoted 6 lines
(1) # echo bus,flr >
reset_methods(2) # echo flr,bus >
reset_methodsDo these have the same effect or not?
They have different effect.
I asked about this because Shanker's idea [1] of using two bitmaps only keeps track of which resets are *enabled*. It does not keep track of the *ordering*. Since you want to control the ordering, I think we need more state than just the supported/enabled bitmaps.
Quoted 5 lines
Quoted 4 lines
If "
reset_methods" allows control over the order, I expect them to be different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.Quoted 7 lines
Exactly you are right.
Now the point I was presenting was with new encoding we have to write list of *all of the supported reset methods* in order for example, in above example flr,bus or bus,flr. We can't just write 'flr' or 'bus' then switch back to writing flr,bus/bus,flr (these have different effect as mentioned earlier).
Quoted 3 lines
It sounds like you're saying this sequence can't work:
# echo flr > reset_methods
# dev->reset_methods = [3, 0, 0, ..]
Quoted 1 line
# echo bus,flr > reset_methods
# to get dev->reset_methods = [6, 3, 0, ...] we'll need to probe reset methods here.
Quoted 3 lines
But I'm afraid you'll have to walk me through the reasons why this can't be made to work.
I wrote incomplete description. It can work but we'll need to probe everytime which involves reading different capabilities(PCI_CAP_ID_AF, PCI_PM_CTRL etc) from device. With current encoding we just have to probe at the begining.
Quoted 11 lines
Quoted 4 lines
Basically with new encoding an user can't write subset of reset methods they have to write list of *all* supported methods everytime.
Why does the user have to write all supported methods? Is that to preserve the fact that "cat
reset_methods" always shows all the supported methods so the user knows what's available?I'm wondering why we can't do something like this (pidgin code):
Quoted 9 lines
if (option == "default") {
pci_init_reset_methods(dev);
return;
}
n = 0;
foreach method in option {
i = lookup_reset_method(method);
if (pci_reset_methods[i].reset_fn(dev, PROBE) == 0)Repeatedly calling probe might have some impact as it involves reading device registers as explained earlier.
Quoted 10 lines
dev->reset_methods[n++] = i; # method i supported}
dev->reset_methods[n++] = 0; # end of supported methodsIf we did something like the above, the user could always find the list of all methods supported by a device by doing this:
# echo default > reset_methods # cat reset_methods
This is one solution for current problem with new encoding.
Quoted 2 lines
Yes, this does call the "probe" methods several times. I don't think that's necessarily a problem.
I thought this would be a problem because of your earlier suggestion of caching flr capability to avoid probing multiple times. In this case we'll need to read different device registers multiple times. With current encoding we don't have to do that multiple times.
Thanks,
Amey
Quoted 4 lines
Bjorn
Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism
Quoted 119 lines · Amey Narkhede
Quoted 64 lines · Bjorn Helgaas
Quoted 37 lines · Amey Narkhede
Quoted 36 lines · Bjorn Helgaas
Quoted 18 lines · Amey Narkhede
Add
reset_methodsysfs attribute to enable user to query and set user preferred device reset methods and their ordering.drivers/pci/pci-sysfs.c +13
-0+ Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. + while ((name = strsep(&options, ",")) != NULL) { +if (sysfs_streq (name,"" )) +continue ; + + name =strim (name); + +for (i =0 ; i <PCI_RESET_METHODS_NUM ; i++) { +if (reset_methods[i] && +sysfs_streq (name, pci_reset_fn_methods[i].name)) {drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420
Quoted 10 lines
drivers/pci/pci-sysfs.c +10
-0+ reset_methods[i] = prio--; +break ; + } + } + +if (i ==PCI_RESET_METHODS_NUM ) { +kfree (options); +return -EINVAL ; + } + }Asking again since we didn't get this clarified before. The above tells me that "
reset_methods" allows the user to control the *order* in which we try reset methods.Consider the following two uses:
Quoted 7 lines
Quoted 6 lines
(1) # echo bus,flr >
reset_methods(2) # echo flr,bus >
reset_methodsDo these have the same effect or not?
They have different effect.
I asked about this because Shanker's idea [1] of using two bitmaps only keeps track of which resets are *enabled*. It does not keep track of the *ordering*. Since you want to control the ordering, I think we need more state than just the supported/enabled bitmaps.
Quoted 5 lines
Quoted 4 lines
If "
reset_methods" allows control over the order, I expect them to be different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.Quoted 7 lines
Exactly you are right.
Now the point I was presenting was with new encoding we have to write list of *all of the supported reset methods* in order for example, in above example flr,bus or bus,flr. We can't just write 'flr' or 'bus' then switch back to writing flr,bus/bus,flr (these have different effect as mentioned earlier).
Quoted 3 lines
It sounds like you're saying this sequence can't work:
# echo flr > reset_methods#
dev->reset_methods= [3, 0, 0, ..]Quoted 1 line
# echo bus,flr > reset_methods# to get dev->reset_methods = [6, 3, 0, ...] we'll need to probe reset methods here.Quoted 2 lines
But I'm afraid you'll have to walk me through the reasons why this can't be made to work.
I wrote incomplete description. It can work but we'll need to probe everytime which involves reading different
capabilities(PCI_CAP_ID_AF, PCI_PM_CTRL etc)from device. With current encoding we just have to probe at the begining.Quoted 10 lines
Quoted 3 lines
Basically with new encoding an user can't write subset of reset methods they have to write list of *all* supported methods everytime.
Why does the user have to write all supported methods? Is that to preserve the fact that "cat
reset_methods" always shows all the supported methods so the user knows what's available?I'm wondering why we can't do something like this (pidgin code):
Quoted 9 lines
if (option == "default") { pci_init_reset_methods(dev); return; } n = 0; foreach method in option { i = lookup_reset_method(method); if (pci_reset_methods[i].reset_fn(dev, PROBE) == 0)Repeatedly calling probe might have some impact as it involves reading device registers as explained earlier.
Quoted 9 lines
dev->reset_methods[n++] = i; # method i supported}
dev->reset_methods[n++] = 0; # end of supported methodsIf we did something like the above, the user could always find the list of all methods supported by a device by doing this:
# echo default > reset_methods # cat reset_methodsThis is one solution for current problem with new encoding.
Quoted 7 lines
Quoted 2 lines
Yes, this does call the "probe" methods several times. I don't think that's necessarily a problem.
I thought this would be a problem because of your earlier suggestion of caching flr capability to avoid probing multiple times. In this case we'll need to read different device registers multiple times. With current encoding we don't have to do that multiple times.
I don't think it's a problem to run "probe" methods when we're setting the enabled reset methods (either at enumeration-time or when we write to "reset_methods"). These are low-frequency events and I don't think there's any performance issue.
I don't think we should have to run "probe" methods every time we call pci_reset_function().
I suggested a dev->has_flr bit for two reasons:
1) It avoids reading PCI_EXP_DEVCAP every time a driver calls pcie_reset_flr(), and
2) It gives a nice opportunity for quirks to disable FLR for devices where it's broken.
Quoted 1 line
Quoted 1 line
PATCH v7 5/8 PCI: Setup ACPI_COMPANION early
In reply to Amey NarkhedeFrom: Shanker Donthineni <[email protected]>
Currently, the ACPI_COMPANION is not available until device_add(). The software features which have dependency on ACPI fwnode properties and needs to be handled before device_add() will not work. One use case, software has to check the existence of _RST method to support ACPI based reset mechanism.
This patch adds a new function pci_set_acpi_fwnode() for setting the ACPI_COMPANION, same code which is available in acpi_pci_bridge_d3().
Call pci_set_acpi_fwnode() from pci_scan_device() to fix the issue.
--- drivers/pci/pci-acpi.c | 12 ++++++++---- drivers/pci/pci.h | 2 ++ drivers/pci/probe.c | 2 ++ 3 files changed, 12 insertions(+), 4 deletions(-)
drivers/pci/pci-acpi.c +8 -4
@@ -934,6 +934,13 @@ static pci_power_t acpi_pci_choose_state(struct pci_dev *pdev)static struct acpi_device *acpi_pci_find_companion (struct device *dev); +void pci_set_acpi_fwnode (struct pci_dev *dev) +{ +if (!ACPI_COMPANION (&dev->dev) && !pci_dev_is_added (dev)) +ACPI_COMPANION_SET (&dev->dev, +acpi_pci_find_companion (&dev->dev)); +} +static bool acpi_pci_bridge_d3 (struct pci_dev *dev) {const struct fwnode_handle *fwnode;
@@ -945,11 +952,8 @@ static bool acpi_pci_bridge_d3(struct pci_dev *dev)return false;/* Assume D3 support if the bridge is power-manageable by ACPI. */ +pci_set_acpi_fwnode (dev); adev =ACPI_COMPANION (&dev->dev); -if (!adev && !pci_dev_is_added (dev)) { - adev =acpi_pci_find_companion (&dev->dev); -ACPI_COMPANION_SET (&dev->dev, adev); - }if (adev &&acpi_device_power_manageable (adev))return true;
drivers/pci/pci.h +2 -0
@@ -703,7 +703,9 @@ static inline int pci_aer_raw_clear_status(struct pci_dev *dev) { return -EINVAL#ifdef CONFIG_ACPI int pci_acpi_program_hp_params (struct pci_dev *dev);extern const struct attribute_group pci_dev_acpi_attr_group; +void pci_set_acpi_fwnode (struct pci_dev *dev);#else +static inline void pci_set_acpi_fwnode (struct pci_dev *dev) {}static inline int pci_acpi_program_hp_params (struct pci_dev *dev) {return -ENODEV ;
drivers/pci/probe.c +2 -0
@@ -2359,6 +2359,8 @@ static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn)return NULL ; } +pci_set_acpi_fwnode (dev); +return dev; }
Signature · 1 line
2.31.1
PATCH v7 6/8 PCI: Add support for ACPI _RST reset method
In reply to Amey NarkhedeFrom: Shanker Donthineni <[email protected]>
The _RST is a standard method specified in the ACPI specification. It provides a function level reset when it is described in the acpi_device context associated with PCI-device. Implement a new reset function pci_dev_acpi_reset() for probing RST method and execute if it is defined in the firmware.
The default priority of the ACPI reset is set to below device-specific and above hardware resets.
--- drivers/pci/pci-acpi.c | 23 +++++++++++++++++++++++ drivers/pci/pci.c | 1 + drivers/pci/pci.h | 6 ++++++ include/linux/pci.h | 2 +- 4 files changed, 31 insertions(+), 1 deletion(-)
drivers/pci/pci-acpi.c +23 -0
@@ -941,6 +941,29 @@ void pci_set_acpi_fwnode(struct pci_dev *dev) acpi_pci_find_companion(&dev->dev)); } +/** +* pci_dev_acpi_reset - do a function level reset using _RST method +* @dev: device to reset +* @probe: check if _RST method is included in the acpi_device context. +*/ +int pci_dev_acpi_reset (struct pci_dev *dev,int probe) +{ +acpi_handle handle =ACPI_HANDLE (&dev->dev); + +if (!handle || !acpi_has_method (handle,"_RST" )) +return -ENOTTY ; + +if (probe) +return 0 ; + +if (ACPI_FAILURE (acpi_evaluate_object (handle,"_RST" ,NULL ,NULL ))) { +pci_warn (dev,"ACPI _RST failed\n" ); +return -EINVAL ; + } + +return 0 ; +} +static bool acpi_pci_bridge_d3 (struct pci_dev *dev) {const struct fwnode_handle *fwnode;
drivers/pci/pci.c +1 -0
@@ -5121,6 +5121,7 @@ static void pci_dev_restore(struct pci_dev *dev)*/ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { &pci_dev_specific_reset, .name ="device_specific" }, + { &pci_dev_acpi_reset, .name ="acpi" }, { &pcie_reset_flr, .name ="flr" }, { &pci_af_flr, .name ="af_flr" }, { &pci_pm_reset, .name ="pm" },
drivers/pci/pci.h +6 -0
@@ -704,7 +704,13 @@ static inline int pci_aer_raw_clear_status(struct pci_dev *dev) { return -EINVALint pci_acpi_program_hp_params (struct pci_dev *dev);extern const struct attribute_group pci_dev_acpi_attr_group;void pci_set_acpi_fwnode (struct pci_dev *dev); +int pci_dev_acpi_reset (struct pci_dev *dev,int probe);#else +static inline int pci_dev_acpi_reset (struct pci_dev *dev,int probe) +{ +return -ENOTTY ; +} +static inline void pci_set_acpi_fwnode (struct pci_dev *dev) {}static inline int pci_acpi_program_hp_params (struct pci_dev *dev) {
include/linux/pci.h +1 -1
@@ -49,7 +49,7 @@ PCI_STATUS_SIG_TARGET_ABORT | \ PCI_STATUS_PARITY) -#define PCI_RESET_METHODS_NUM 5 +#define PCI_RESET_METHODS_NUM 6/* * The PCI interface treats multi-function devices as independent
Signature · 1 line
2.31.1
PATCH v7 7/8 PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
In reply to Amey NarkhedeFrom: Shanker Donthineni <[email protected]>
On select platforms, some Nvidia GPU devices do not work with SBR. Triggering SBR would leave the device inoperable for the current system boot. It requires a system hard-reboot to get the GPU device back to normal operating condition post-SBR. For the affected devices, enable NO_BUS_RESET quirk to fix the issue.
This issue will be fixed in the next generation of hardware.
--- drivers/pci/quirks.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
drivers/pci/quirks.c +12 -0
@@ -3546,6 +3546,18 @@ static void quirk_no_bus_reset(struct pci_dev *dev) dev->dev_flags |=PCI_DEV_FLAGS_NO_BUS_RESET ; } +/* +* Some Nvidia GPU devices do not work with bus reset, SBR needs to be +* prevented for those affected devices. +*/ +static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) +{ +if ((dev->device &0xffc0 ) ==0x2340 ) +quirk_no_bus_reset (dev); +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , + quirk_nvidia_no_bus_reset); +/* * Some Atheros AR9xxx and QCA988x chips do not behave after a bus reset. * The device will throw a Link Down error on AER-capable systems and
Signature · 1 line
2.31.1
Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
Quoted 3 lines · Amey Narkhede
From: Shanker Donthineni <[email protected]>
On select platforms, some Nvidia GPU devices do not work with SBR.
Interesting that you say "on select platforms." Apparently SBR does work for some of these GPUs, but not on all platforms? If you have any clarification here, I can still update the commit log.
Quoted 9 lines
Triggering SBR would leave the device inoperable for the current system boot. It requires a system hard-reboot to get the GPU device back to normal operating condition post-SBR. For the affected devices, enable
NO_BUS_RESETquirk to fix the issue.This issue will be fixed in the next generation of hardware.
Signed-off-by Shanker Donthineni <[email protected]>Reviewed-by Sinan Kaya <[email protected]>
This patch doesn't seem to have any dependencies or particular connection to the rest of the reset series, so I applied this patch by itself to for-linus for v5.13 and marked it for stable.
If that's not right, let me know.
Quoted 30 lines
--- drivers/pci/quirks.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)drivers/pci/quirks.c +12
-0@@ -3546,6 +3546,18 @@ static void quirk_no_bus_reset(struct pci_dev *dev) dev->dev_flags |=PCI_DEV_FLAGS_NO_BUS_RESET ; } +/* +* Some Nvidia GPU devices do not work with bus reset, SBR needs to be +* prevented for those affected devices. +*/ +static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) +{ +if ((dev->device &0xffc0 ) ==0x2340 ) +quirk_no_bus_reset (dev); +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , + quirk_nvidia_no_bus_reset); +/* * Some Atheros AR9xxx and QCA988x chips do not behave after a bus reset. * The device will throw a Link Down error on AER-capable systems and Signature · 1 line
2.31.1
Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
Hi Bjorn,
Quoted 7 lines · Bjorn Helgaas
Quoted 7 lines
Triggering SBR would leave the device inoperable for the current system boot. It requires a system hard-reboot to get the GPU device back to normal operating condition post-SBR. For the affected devices, enable
NO_BUS_RESETquirk to fix the issue.This issue will be fixed in the next generation of hardware.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Shanker Donthineni <[email protected]>Reviewed-by Sinan Kaya <[email protected]>This patch doesn't seem to have any dependencies or particular connection to the rest of the reset series, so I applied this patch by itself to for-linus for v5.13 and marked it for stable.
If that's not right, let me know.
Yes, you're right this patch no dependency on reset method series.
Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
In reply to Bjorn HelgaasHi Bjorn,
Quoted 7 lines · Bjorn Helgaas
Quoted 3 lines
From: Shanker Donthineni <[email protected]>
On select platforms, some Nvidia GPU devices do not work with SBR.
Interesting that you say "on select platforms." Apparently SBR does work for some of these GPUs, but not on all platforms? If you have any clarification here, I can still update the commit log.
Yes, SBR works for some GPUs but GPUs which are listed in this quirk will not work and these GPUs are available only on selected server platforms. I believe commit text reflects the issue but please update if needed.
-
Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
Quoted 11 lines · Shanker R Donthineni
Quoted 7 lines · Bjorn Helgaas
Quoted 3 lines
From: Shanker Donthineni <[email protected]>
On select platforms, some Nvidia GPU devices do not work with SBR.
Interesting that you say "on select platforms." Apparently SBR does work for some of these GPUs, but not on all platforms? If you have any clarification here, I can still update the commit log.
Yes, SBR works for some GPUs but GPUs which are listed in this quirk will not work and these GPUs are available only on selected server platforms. I believe commit text reflects the issue but please update if needed.
It sounds like there is no actual dependency on the platform. So even though these GPUs are only available on certain platforms, if one were to move one of them to a different, non-supported platform, SBR would still not work.
So I think I'll remove the reference to "select platforms" since it doesn't add any useful information and might suggest that SBR should work on some platforms, if you could only find the right ones.
Bjorn
Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
Hi Bjorn,
Quoted 8 lines · Bjorn Helgaas
It sounds like there is no actual dependency on the platform. So even though these GPUs are only available on certain platforms, if one were to move one of them to a different, non-supported platform, SBR would still not work.
So I think I'll remove the reference to "select platforms" since it doesn't add any useful information and might suggest that SBR should work on some platforms, if you could only find the right ones.
Appreciate your time on code review, providing better text, and picking patch for v5.14. Please let us know if any code improvements or suggestions for the remaining reset patch series to be considered for v5.14
PATCH v7 8/8 PCI: Change the type of probe argument in reset functions
In reply to Amey NarkhedeIntroduce a new enum pci_reset_mode_t to make the context of probe argument in reset functions clear and the code easier to read.
Change the type of probe argument in functions which implement reset methods from int to pci_reset_mode_t to make the intent clear.
Add a new line in return statement of pci_reset_bus_function().
--- drivers/crypto/cavium/nitrox/nitrox_main.c | 2 +- .../ethernet/cavium/liquidio/lio_vf_main.c | 2 +- drivers/pci/hotplug/pciehp.h | 2 +- drivers/pci/hotplug/pciehp_hpc.c | 4 +- drivers/pci/pci-acpi.c | 10 ++- drivers/pci/pci.c | 85 ++++++++++++------- drivers/pci/pci.h | 12 +-- drivers/pci/pcie/aer.c | 2 +- drivers/pci/quirks.c | 37 ++++---- include/linux/pci.h | 8 +- include/linux/pci_hotplug.h | 2 +- 11 files changed, 101 insertions(+), 65 deletions(-)
drivers/crypto/cavium/nitrox/nitrox_main.c +1 -1
@@ -306,7 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)return -ENOMEM ; } -pcie_reset_flr (pdev,0 ); +pcie_reset_flr (pdev,PCI_RESET_DO_RESET );pci_restore_state (pdev);
drivers/net/ethernet/cavium/liquidio/lio_vf_main.c +1 -1
@@ -526,7 +526,7 @@ static void octeon_destroy_resources(struct octeon_device *oct) oct->irq_name_storage =NULL ; }/* Soft reset the octeon device before exiting */ -if (!pcie_reset_flr (oct->pci_dev,1 )) +if (!pcie_reset_flr (oct->pci_dev,PCI_RESET_PROBE ))octeon_pci_flr (oct);else cn23xx_vf_ask_pf_to_do_flr (oct);
drivers/pci/hotplug/pciehp.h +1 -1
@@ -181,7 +181,7 @@ void pciehp_release_ctrl(struct controller *ctrl);int pciehp_sysfs_enable_slot (struct hotplug_slot *hotplug_slot);int pciehp_sysfs_disable_slot (struct hotplug_slot *hotplug_slot); -int pciehp_reset_slot (struct hotplug_slot *hotplug_slot,int probe); +int pciehp_reset_slot (struct hotplug_slot *hotplug_slot,pci_reset_mode_t mode);int pciehp_get_attention_status (struct hotplug_slot *hotplug_slot,u8 *status);int pciehp_set_raw_indicator_status (struct hotplug_slot *h_slot,u8 status);int pciehp_get_raw_indicator_status (struct hotplug_slot *h_slot,u8 *status);
drivers/pci/hotplug/pciehp_hpc.c +2 -2
@@ -834,14 +834,14 @@ void pcie_disable_interrupt(struct controller *ctrl)* momentarily, if we see that they could interfere. Also, clear any spurious * events after. */ -int pciehp_reset_slot (struct hotplug_slot *hotplug_slot,int probe) +int pciehp_reset_slot (struct hotplug_slot *hotplug_slot,pci_reset_mode_t mode) {struct controller *ctrl =to_ctrl (hotplug_slot);struct pci_dev *pdev =ctrl_dev (ctrl);u16 stat_mask =0 , ctrl_mask =0 ;int rc; -if (probe) +if (mode ==PCI_RESET_PROBE )return 0 ;down_write (&ctrl->reset_lock);
drivers/pci/pci-acpi.c +7 -3
@@ -944,16 +944,20 @@ void pci_set_acpi_fwnode(struct pci_dev *dev)/** * pci_dev_acpi_reset - do a function level reset using _RST method * @dev: device to reset -* @probe: check if _RST method is included in the acpi_device context. +* @probe: If PCI_RESET_PROBE, check whether _RST method is included +* in the acpi_device context. */ -int pci_dev_acpi_reset (struct pci_dev *dev,int probe) +int pci_dev_acpi_reset (struct pci_dev *dev,pci_reset_mode_t mode) {acpi_handle handle =ACPI_HANDLE (&dev->dev); +if (mode >=PCI_RESET_MODE_MAX ) +return -EINVAL ; +if (!handle || !acpi_has_method (handle,"_RST" ))return -ENOTTY ; -if (probe) +if (mode ==PCI_RESET_PROBE )return 0 ;if (ACPI_FAILURE (acpi_evaluate_object (handle,"_RST" ,NULL ,NULL ))) {
drivers/pci/pci.c +53 -32
268 lines not on this page · raw message
drivers/pci/pci.h +6 -6
@@ -603,19 +603,19 @@ static inline int pci_enable_ptm(struct pci_dev *dev, u8 *granularity)struct pci_dev_reset_methods {u16 vendor;u16 device; -int (*reset)(struct pci_dev *dev,int probe); +int (*reset)(struct pci_dev *dev,pci_reset_mode_t mode); };struct pci_reset_fn_method { -int (*reset_fn)(struct pci_dev *pdev,int probe); +int (*reset_fn)(struct pci_dev *pdev,pci_reset_mode_t mode);char *name; };extern const struct pci_reset_fn_method pci_reset_fn_methods[];#ifdef CONFIG_PCI_QUIRKS -int pci_dev_specific_reset (struct pci_dev *dev,int probe); +int pci_dev_specific_reset (struct pci_dev *dev,pci_reset_mode_t mode);#else -static inline int pci_dev_specific_reset (struct pci_dev *dev,int probe) +static inline int pci_dev_specific_reset (struct pci_dev *dev,pci_reset_mode_t mode) {return -ENOTTY ; }
@@ -704,9 +704,9 @@ static inline int pci_aer_raw_clear_status(struct pci_dev *dev) { return -EINVALint pci_acpi_program_hp_params (struct pci_dev *dev);extern const struct attribute_group pci_dev_acpi_attr_group;void pci_set_acpi_fwnode (struct pci_dev *dev); -int pci_dev_acpi_reset (struct pci_dev *dev,int probe); +int pci_dev_acpi_reset (struct pci_dev *dev,pci_reset_mode_t mode);#else -static inline int pci_dev_acpi_reset (struct pci_dev *dev,int probe) +static inline int pci_dev_acpi_reset (struct pci_dev *dev,pci_reset_mode_t mode) {return -ENOTTY ; }
drivers/pci/pcie/aer.c +1 -1
@@ -1405,7 +1405,7 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev) }if (type ==PCI_EXP_TYPE_RC_EC || type ==PCI_EXP_TYPE_RC_END ) { - rc =pcie_reset_flr (dev,0 ); + rc =pcie_reset_flr (dev,PCI_RESET_DO_RESET );if (!rc)pci_info (dev,"has been reset\n" );else
drivers/pci/quirks.c +21 -16
@@ -3681,7 +3681,7 @@ DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL,* reset a single function if other methods (e.g. FLR, PM D0->D3) are * not available. */ -static int reset_intel_82599_sfp_virtfn (struct pci_dev *dev,int probe) +static int reset_intel_82599_sfp_virtfn (struct pci_dev *dev,pci_reset_mode_t mode) {/* * http://www.intel.com/content/dam/doc/datasheet/82599-10-gbe-controller-datasheet.pdf
@@ -3691,7 +3691,7 @@ static int reset_intel_82599_sfp_virtfn(struct pci_dev *dev, int probe)* Thus we must call pcie_flr() directly without first checking if it is * supported. */ -if (!probe) +if (mode ==PCI_RESET_DO_RESET )pcie_flr (dev);return 0 ; }
@@ -3703,13 +3703,13 @@ static int reset_intel_82599_sfp_virtfn(struct pci_dev *dev, int probe)#define NSDE_PWR_STATE 0xd0100#define IGD_OPERATION_TIMEOUT 10000/* set timeout 10 seconds */ -static int reset_ivb_igd (struct pci_dev *dev,int probe) +static int reset_ivb_igd (struct pci_dev *dev,pci_reset_mode_t mode) {void __iomem *mmio_base;unsigned long timeout;u32 val; -if (probe) +if (mode ==PCI_RESET_PROBE )return 0 ; mmio_base =pci_iomap (dev,0 ,0 );
@@ -3746,7 +3746,7 @@ static int reset_ivb_igd(struct pci_dev *dev, int probe) }/* Device-specific reset method for Chelsio T4-based adapters */ -static int reset_chelsio_generic_dev (struct pci_dev *dev,int probe) +static int reset_chelsio_generic_dev (struct pci_dev *dev,pci_reset_mode_t mode) {u16 old_command;u16 msix_flags;
@@ -3762,7 +3762,7 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, int probe)* If this is the "probe" phase, return 0 indicating that we can * reset this device. */ -if (probe) +if (mode ==PCI_RESET_PROBE )return 0 ;/*
@@ -3824,17 +3824,17 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, int probe)* Chapter 3: NVMe control registers * Chapter 7.3: Reset behavior */ -static int nvme_disable_and_flr (struct pci_dev *dev,int probe) +static int nvme_disable_and_flr (struct pci_dev *dev,pci_reset_mode_t mode) {void __iomem *bar;u16 cmd;u32 cfg;if (dev->class !=PCI_CLASS_STORAGE_EXPRESS || -pcie_reset_flr (dev,1 ) || !pci_resource_start (dev,0 )) +pcie_reset_flr (dev,PCI_RESET_PROBE ) || !pci_resource_start (dev,0 ))return -ENOTTY ; -if (probe) +if (mode ==PCI_RESET_PROBE )return 0 ; bar =pci_iomap (dev,0 ,NVME_REG_CC +sizeof (cfg));
@@ -3898,11 +3898,13 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)* device too soon after FLR. A 250ms delay after FLR has heuristically * proven to produce reliably working results for device assignment cases. */ -static int delay_250ms_after_flr (struct pci_dev *dev,int probe) +static int delay_250ms_after_flr (struct pci_dev *dev,pci_reset_mode_t mode) { -int ret =pcie_reset_flr (dev, probe); +int ret; + + ret =pcie_reset_flr (dev, mode); -if (probe) +if (ret || mode ==PCI_RESET_PROBE )return ret;msleep (250 );
@@ -3918,13 +3920,13 @@ static int delay_250ms_after_flr(struct pci_dev *dev, int probe)#define HINIC_OPERATION_TIMEOUT 15000/* 15 seconds */ /* Device-specific reset method for Huawei Intelligent NIC virtual functions */ -static int reset_hinic_vf_dev (struct pci_dev *pdev,int probe) +static int reset_hinic_vf_dev (struct pci_dev *pdev,pci_reset_mode_t mode) {unsigned long timeout;void __iomem *bar;u32 val; -if (probe) +if (mode ==PCI_RESET_PROBE )return 0 ; bar =pci_iomap (pdev,0 ,0 );
@@ -3995,16 +3997,19 @@ static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {* because when a host assigns a device to a guest VM, the host may need * to reset the device but probably doesn't have a driver for it. */ -int pci_dev_specific_reset (struct pci_dev *dev,int probe) +int pci_dev_specific_reset (struct pci_dev *dev,pci_reset_mode_t mode) {const struct pci_dev_reset_methods *i; +if (mode >=PCI_RESET_MODE_MAX ) +return -EINVAL ; +for (i = pci_dev_reset_methods; i->reset; i++) {if ((i->vendor == dev->vendor || i->vendor == (u16 )PCI_ANY_ID ) && (i->device == dev->device || i->device == (u16 )PCI_ANY_ID )) -return i->reset (dev, probe); +return i->reset (dev, mode); }return -ENOTTY ;
include/linux/pci.h +7 -1
@@ -51,6 +51,12 @@#define PCI_RESET_METHODS_NUM 6 +typedef enum pci_reset_mode { +PCI_RESET_DO_RESET , +PCI_RESET_PROBE , +PCI_RESET_MODE_MAX , +}pci_reset_mode_t ; +/* * The PCI interface treats multi-function devices as independent * devices. The slot/function address of each device is encoded
@@ -1230,7 +1236,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,enum pci_bus_speed *speed, enum pcie_link_width *width);void pcie_print_link_status (struct pci_dev *dev); -int pcie_reset_flr (struct pci_dev *dev,int probe); +int pcie_reset_flr (struct pci_dev *dev,pci_reset_mode_t mode);int pcie_flr (struct pci_dev *dev);bool pci_reset_supported (struct pci_dev *dev);int __pci_reset_function_locked (struct pci_dev *dev);
include/linux/pci_hotplug.h +1 -1
@@ -44,7 +44,7 @@ struct hotplug_slot_ops {int (*get_attention_status) (struct hotplug_slot *slot,u8 *value);int (*get_latch_status) (struct hotplug_slot *slot,u8 *value);int (*get_adapter_status) (struct hotplug_slot *slot,u8 *value); -int (*reset_slot) (struct hotplug_slot *slot,int probe); +int (*reset_slot) (struct hotplug_slot *slot,pci_reset_mode_t mode); };/**
--
2.31.1
Re: [PATCH v7 8/8] PCI: Change the type of probe argument in reset functions
Quoted 10 lines · Amey Narkhede
Introduce a new enum
pci_reset_mode_tto make the context of probe argument in reset functions clear and the code easier to read.
Change the type of probe argument in functions which implement reset methods from int topci_reset_mode_tto make the intent clear.Add a new line in return statement of
pci_reset_bus_function().Suggested-by Alex Williamson <[email protected]>Suggested-by Krzysztof Wilczyński <[email protected]>Signed-off-by Amey Narkhede <[email protected]>
Quoted 13 lines
--- drivers/crypto/cavium/nitrox/nitrox_main.c | 2 +- .../ethernet/cavium/liquidio/lio_vf_main.c | 2 +- drivers/pci/hotplug/pciehp.h | 2 +- drivers/pci/hotplug/pciehp_hpc.c | 4 +- drivers/pci/pci-acpi.c | 10 ++- drivers/pci/pci.c | 85 ++++++++++++------- drivers/pci/pci.h | 12 +-- drivers/pci/pcie/aer.c | 2 +- drivers/pci/quirks.c | 37 ++++---- include/linux/pci.h | 8 +- include/linux/pci_hotplug.h | 2 +- 11 files changed, 101 insertions(+), 65 deletions(-)
On 08.06.21 07:48, Amey Narkhede wrote:
Hi,
Quoted 5 lines
PCI and PCIe devices may support a number of possible reset mechanisms for example Function Level Reset (FLR) provided via Advanced Feature or PCIe capabilities, Power Management reset, bus reset, or device specific reset. Currently the PCI subsystem creates a policy prioritizing these reset methods which provides neither visibility nor control to userspace.
Since I've got a current use case for that - could you perhaps tell more about the whole pci device reset mechanisms ?
In my case I've got a board that wires reset lines to the soc's gpios. Not sure how exactly to qualify this, but I guess it would count as a bus wide reset.
Now the big question for me is how to implement that in a board specific platform driver (which already does setup of gpios and other attached devices), so we can reset the card in slot X in a generic way.
Any help highly appreciated.
--mtx
Signature · 8 lines
--- Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren GPG/PGP-Schlüssel zu. --- Enrico Weigelt, metux IT consult Free software and Linux embedded engineering [email protected] -- +49-151-27565287
Quoted 17 lines · metux IT consult
On 08.06.21 07:48, Amey Narkhede wrote:
Hi,
Quoted 5 lines
PCI and PCIe devices may support a number of possible reset mechanisms for example Function Level Reset (FLR) provided via Advanced Feature or PCIe capabilities, Power Management reset, bus reset, or device specific reset. Currently the PCI subsystem creates a policy prioritizing these reset methods which provides neither visibility nor control to userspace.
Since I've got a current use case for that - could you perhaps tell more about the whole pci device reset mechanisms ?
In my case I've got a board that wires reset lines to the soc's gpios. Not sure how exactly to qualify this, but I guess it would count as a bus wide reset.
Quoted 9 lines
Now the big question for me is how to implement that in a board specific platform driver (which already does setup of gpios and other attached devices), so we can reset the card in slot X in a generic way.
Any help highly appreciated.
--mtx
In case of bus reset(pci_reset_secondary_bus()), it uses bridge control register to assert reset on bus so I think it should out of the box but not 100% sure about it.
Thanks,
Amey