Skip to contentSkip to conversation

Expose and manage PCI device reset

PATCH Superseded Amey Narkhede 6 participants 52 messages Jun 8, 2021 – Jun 24, 2021

Subsystems (6)
Network drivers ACPI PCI Crypto API +2 more subsystems

v7 of 16 Superseded Latest is v16 →

lore
Share & tools
lore
#0 lore raw

v7 cover Expose and manage PCI device reset

PCI and PCIe devices may support a number of possible reset mechanisms for example Function Level Reset (FLR) provided via Advanced Feature or PCIe capabilities, Power Management reset, bus reset, or device specific reset. Currently the PCI subsystem creates a policy prioritizing these reset methods which provides neither visibility nor control to userspace.

Expose the reset methods available per device to userspace, via sysfs and allow an administrative user or device owner to have ability to manage per device reset method priorities or exclusions. This feature aims to allow greater control of a device for use cases as device assignment, where specific device or platform issues may interact poorly with a given reset method, and for which device specific quirks have not been developed.

Changes in v7:
- Fix the pci_dev_acpi_reset() prototype mismatch in case of CONFIG_ACPI=n

Changes in v6:
- Address Bjorn's and Krzysztof's review comments
- Add Shanker's updated patches along with new "PCI: Setup ACPI_COMPANION early" patch

Changes in v5:
- Rebase the series over pci/reset branch of Bjorn's pci tree to avoid merge conflicts caused by recent changes in existing reset sysfs attribute

Changes in v4:
- Change the order or strlen and strim in reset_method_store function to avoid extra strlen call.
- Use consistent terminology in new pci_reset_mode enum and rename the probe argument of reset functions.

Changes in v3:
- Dropped "PCI: merge slot and bus reset implementations" which was already accepted separately
- Grammar fixes
- Added Shanker's patches which were rebased on v2 of this series
- Added "PCI: Change the type of probe argument in reset functions" and additional user input sanitization code in reset_method_store function per review feedback from Krzysztof

Changes in v2:
- Use byte array instead of bitmap to keep track of ordering of reset methods
- Fix incorrect use of reset_fn field in octeon driver
- Allow writing comma separated list of names of supported reset methods to reset_method sysfs attribute
- Writing empty string instead of "none" to reset_method attribute disables ability of reset the device

Amey Narkhede (5):
PCI: Add pcie_reset_flr to follow calling convention of other reset
methods
PCI: Add new array for keeping track of ordering of reset methods
PCI: Remove reset_fn field from pci_dev
PCI/sysfs: Allow userspace to query and set device reset mechanism
PCI: Change the type of probe argument in reset functions

Shanker Donthineni (3):
PCI: Setup ACPI_COMPANION early
PCI: Add support for ACPI _RST reset method
PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs

 Documentation/ABI/testing/sysfs-bus-pci       |  16 ++
 drivers/crypto/cavium/nitrox/nitrox_main.c    |   4 +-
 .../ethernet/cavium/liquidio/lio_vf_main.c    |   2 +-
 drivers/pci/hotplug/pciehp.h                  |   2 +-
 drivers/pci/hotplug/pciehp_hpc.c              |   4 +-
 drivers/pci/pci-acpi.c                        |  39 ++-
 drivers/pci/pci-sysfs.c                       | 120 ++++++++-
 drivers/pci/pci.c                             | 238 +++++++++++-------
 drivers/pci/pci.h                             |  22 +-
 drivers/pci/pcie/aer.c                        |  12 +-
 drivers/pci/probe.c                           |   6 +-
 drivers/pci/quirks.c                          |  54 ++--
 drivers/pci/remove.c                          |   1 -
 include/linux/pci.h                           |  16 +-
 include/linux/pci_hotplug.h                   |   2 +-
 15 files changed, 393 insertions(+), 145 deletions(-)

--
2.31.1

#1 lore raw

PATCH v7 1/8 PCI: Add pcie_reset_flr to follow calling convention of other reset methods

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

---
 drivers/crypto/cavium/nitrox/nitrox_main.c |  4 +-
 drivers/pci/pci.c                          | 62 ++++++++++++----------
 drivers/pci/pcie/aer.c                     | 12 ++---
 drivers/pci/quirks.c                       |  9 ++--
 include/linux/pci.h                        |  2 +-
 5 files changed, 43 insertions(+), 46 deletions(-)
drivers/crypto/cavium/nitrox/nitrox_main.c +1 -3
index facc8e6bc..15d6c8452 100644
@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)
 		return -ENOMEM;
 	}
 
-	/* check flr support */
-	if (pcie_has_flr(pdev))
-		pcie_flr(pdev);
+	pcie_reset_flr(pdev, 0);
 
 	pci_restore_state(pdev);
 
drivers/pci/pci.c +33 -29

90 lines not on this page · raw message

drivers/pci/pcie/aer.c +5 -7
index ec943cee5..98077595a 100644
@@ -1405,13 +1405,11 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)
 	}
 
 	if (type == PCI_EXP_TYPE_RC_EC || type == PCI_EXP_TYPE_RC_END) {
-		if (pcie_has_flr(dev)) {
-			rc = pcie_flr(dev);
-			pci_info(dev, "has been reset (%d)\n", rc);
-		} else {
-			pci_info(dev, "not reset (no FLR support)\n");
-			rc = -ENOTTY;
-		}
+		rc = pcie_reset_flr(dev, 0);
+		if (!rc)
+			pci_info(dev, "has been reset\n");
+		else
+			pci_info(dev, "not reset (no FLR support: %d)\n", rc);
 	} else {
 		rc = pci_bus_error_reset(dev);
 		pci_info(dev, "%s Port link has been reset (%d)\n",
drivers/pci/quirks.c +3 -6
index d85914afe..f977ba79a 100644
@@ -3819,7 +3819,7 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)
 	u32 cfg;
 
 	if (dev->class != PCI_CLASS_STORAGE_EXPRESS ||
-	    !pcie_has_flr(dev) || !pci_resource_start(dev, 0))
+	    pcie_reset_flr(dev, 1) || !pci_resource_start(dev, 0))
 		return -ENOTTY;
 
 	if (probe)
@@ -3888,13 +3888,10 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)
  */
 static int delay_250ms_after_flr(struct pci_dev *dev, int probe)
 {
-	if (!pcie_has_flr(dev))
-		return -ENOTTY;
+	int ret = pcie_reset_flr(dev, probe);
 
 	if (probe)
-		return 0;
-
-	pcie_flr(dev);
+		return ret;
 
 	msleep(250);
 
include/linux/pci.h +1 -1
index c20211e59..20b90c205 100644
@@ -1225,7 +1225,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
 			     enum pci_bus_speed *speed,
 			     enum pcie_link_width *width);
 void pcie_print_link_status(struct pci_dev *dev);
-bool pcie_has_flr(struct pci_dev *dev);
+int pcie_reset_flr(struct pci_dev *dev, int probe);
 int pcie_flr(struct pci_dev *dev);
 int __pci_reset_function_locked(struct pci_dev *dev);
 int pci_reset_function(struct pci_dev *dev);
Signature · 1 line
2.31.1
Tested-by on v7 1/8
#2 lore raw

Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods

Quoted 11 lines · Amey Narkhede

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

#3 lore raw

Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods

In reply to Amey Narkhede

[+cc Christoph, since he added pcie_flr()]

Quoted 5 lines · Amey Narkhede

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

I don't like the fact that we handle FLR differently from other types of reset, so I do like the fact that this makes them more consistent.

Quoted 83 lines
---
 drivers/crypto/cavium/nitrox/nitrox_main.c |  4 +-
 drivers/pci/pci.c                          | 62 ++++++++++++----------
 drivers/pci/pcie/aer.c                     | 12 ++---
 drivers/pci/quirks.c                       |  9 ++--
 include/linux/pci.h                        |  2 +-
 5 files changed, 43 insertions(+), 46 deletions(-)
drivers/crypto/cavium/nitrox/nitrox_main.c +1 -3
index facc8e6bc..15d6c8452 100644
@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)
 		return -ENOMEM;
 	}
 
-	/* check flr support */
-	if (pcie_has_flr(pdev))
-		pcie_flr(pdev);
+	pcie_reset_flr(pdev, 0);
 
 	pci_restore_state(pdev);
 
drivers/pci/pci.c +14 -22
index 452351025..3bf36924c 100644
@@ -4611,32 +4611,12 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
-/**
- * pcie_has_flr - check if a device supports function level resets
- * @dev: device to check
- *
- * Returns true if the device advertises support for PCIe function level
- * resets.
- */
-bool pcie_has_flr(struct pci_dev *dev)
-{
-	u32 cap;
-
-	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
-		return false;
-
-	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
-	return cap & PCI_EXP_DEVCAP_FLR;
-}
-EXPORT_SYMBOL_GPL(pcie_has_flr);
-
 /**
  * pcie_flr - initiate a PCIe function level reset
  * @dev: device to reset
  *
- * Initiate a function level reset on @dev.  The caller should ensure the
- * device supports FLR before calling this function, e.g. by using the
- * pcie_has_flr() helper.
+ * Initiate a function level reset unconditionally on @dev without
+ * checking any flags and DEVCAP
  */
 int pcie_flr(struct pci_dev *dev)
 {
@@ -4659,6 +4639,31 @@ int pcie_flr(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
+/**
+ * pcie_reset_flr - initiate a PCIe function level reset
+ * @dev: device to reset
+ * @probe: If set, only check if the device can be reset this way.
+ *
+ * Initiate a function level reset on @dev.
+ */
+int pcie_reset_flr(struct pci_dev *dev, int probe)
+{
+	u32 cap;
+
+	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)

drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4663

Quoted 10 lines
drivers/pci/pci.c +10 -0
+		return -ENOTTY;
+
+	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
+	if (!(cap & PCI_EXP_DEVCAP_FLR))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	return pcie_flr(dev);

Christoph added pcie_flr() with a60a2b73ba69 ("PCI: Export pcie_flr()"), where the commit log says he split out the probing because "non-core callers already know their hardware."

It *is* reasonable to expect that drivers know whether their device supports FLR so they don't need to probe.

But we don't expose the "probe" argument outside the PCI core for any other reset methods, and I would like to avoid that here.

It seems excessive to have to read PCI_EXP_DEVCAP every time. PCI_EXP_DEVCAP_FLR is a read-only bit, and we should only need to look at it once.

What I would really like here is a single bit in the pci_dev that we could set at enumeration-time, e.g., something like this:

  struct pci_dev {
    ...
    unsigned int has_flr:1;
  };

  void set_pcie_port_type(...)    # during enumeration
  {
    pci_read_config_word(dev, pos + PCI_EXP_DEVCAP, &reg16);
    if (reg16 & PCI_EXP_DEVCAP_FLR)
      dev->has_flr = 1;
  }

  static void quirk_no_flr(...)
  {
    dev->has_flr = 0;             # get rid of PCI_DEV_FLAGS_NO_FLR_RESET
  }

  int pcie_flr(...)
  {
    if (!dev->has_flr)
      return -ENOTTY;
    if (!pci_wait_for_pending_transaction(dev))
      ...

}

I think this should be enough that we could get rid of pcie_has_flr() without having to expose the "probe" argument outside drivers/pci/.

Procedural note: if we *do* have to expose the "probe" argument, can you arrange it to have the correct type before touching the drivers, so we only have to touch the drivers once?

Quoted 101 lines
drivers/pci/pci.c +9 -7
+}
+EXPORT_SYMBOL_GPL(pcie_reset_flr);
+
 static int pci_af_flr(struct pci_dev *dev, int probe)
 {
 	int pos;
@@ -5139,11 +5144,9 @@ int __pci_reset_function_locked(struct pci_dev *dev)
 	rc = pci_dev_specific_reset(dev, 0);
 	if (rc != -ENOTTY)
 		return rc;
-	if (pcie_has_flr(dev)) {
-		rc = pcie_flr(dev);
-		if (rc != -ENOTTY)
-			return rc;
-	}
+	rc = pcie_reset_flr(dev, 0);
+	if (rc != -ENOTTY)
+		return rc;
 	rc = pci_af_flr(dev, 0);
 	if (rc != -ENOTTY)
 		return rc;
@@ -5174,8 +5177,9 @@ int pci_probe_reset_function(struct pci_dev *dev)
 	rc = pci_dev_specific_reset(dev, 1);
 	if (rc != -ENOTTY)
 		return rc;
-	if (pcie_has_flr(dev))
-		return 0;
+	rc = pcie_reset_flr(dev, 1);
+	if (rc != -ENOTTY)
+		return rc;
 	rc = pci_af_flr(dev, 1);
 	if (rc != -ENOTTY)
 		return rc;
drivers/pci/pcie/aer.c +5 -7
index ec943cee5..98077595a 100644
@@ -1405,13 +1405,11 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)
 	}
 
 	if (type == PCI_EXP_TYPE_RC_EC || type == PCI_EXP_TYPE_RC_END) {
-		if (pcie_has_flr(dev)) {
-			rc = pcie_flr(dev);
-			pci_info(dev, "has been reset (%d)\n", rc);
-		} else {
-			pci_info(dev, "not reset (no FLR support)\n");
-			rc = -ENOTTY;
-		}
+		rc = pcie_reset_flr(dev, 0);
+		if (!rc)
+			pci_info(dev, "has been reset\n");
+		else
+			pci_info(dev, "not reset (no FLR support: %d)\n", rc);
 	} else {
 		rc = pci_bus_error_reset(dev);
 		pci_info(dev, "%s Port link has been reset (%d)\n",
drivers/pci/quirks.c +3 -6
index d85914afe..f977ba79a 100644
@@ -3819,7 +3819,7 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)
 	u32 cfg;
 
 	if (dev->class != PCI_CLASS_STORAGE_EXPRESS ||
-	    !pcie_has_flr(dev) || !pci_resource_start(dev, 0))
+	    pcie_reset_flr(dev, 1) || !pci_resource_start(dev, 0))
 		return -ENOTTY;
 
 	if (probe)
@@ -3888,13 +3888,10 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)
  */
 static int delay_250ms_after_flr(struct pci_dev *dev, int probe)
 {
-	if (!pcie_has_flr(dev))
-		return -ENOTTY;
+	int ret = pcie_reset_flr(dev, probe);
 
 	if (probe)
-		return 0;
-
-	pcie_flr(dev);
+		return ret;
 
 	msleep(250);
 
include/linux/pci.h +1 -1
index c20211e59..20b90c205 100644
@@ -1225,7 +1225,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
 			     enum pci_bus_speed *speed,
 			     enum pcie_link_width *width);
 void pcie_print_link_status(struct pci_dev *dev);
-bool pcie_has_flr(struct pci_dev *dev);
+int pcie_reset_flr(struct pci_dev *dev, int probe);
 int pcie_flr(struct pci_dev *dev);
 int __pci_reset_function_locked(struct pci_dev *dev);
 int pci_reset_function(struct pci_dev *dev);
Signature · 1 line
2.31.1
#4 lore raw

Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods

Quoted 110 lines · On Thu, 17 Jun 2021 16:57:34 -0500 Bjorn Helgaas <[email protected]> wrote:

[+cc Christoph, since he added pcie_flr()]

Quoted 5 lines · Amey Narkhede

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

I don't like the fact that we handle FLR differently from other types of reset, so I do like the fact that this makes them more consistent.

Quoted 83 lines
---
 drivers/crypto/cavium/nitrox/nitrox_main.c |  4 +-
 drivers/pci/pci.c                          | 62 ++++++++++++----------
 drivers/pci/pcie/aer.c                     | 12 ++---
 drivers/pci/quirks.c                       |  9 ++--
 include/linux/pci.h                        |  2 +-
 5 files changed, 43 insertions(+), 46 deletions(-)
drivers/crypto/cavium/nitrox/nitrox_main.c +1 -3
index facc8e6bc..15d6c8452 100644
@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)
 		return -ENOMEM;
 	}
 
-	/* check flr support */
-	if (pcie_has_flr(pdev))
-		pcie_flr(pdev);
+	pcie_reset_flr(pdev, 0);
 
 	pci_restore_state(pdev);
 
drivers/pci/pci.c +14 -22
index 452351025..3bf36924c 100644
@@ -4611,32 +4611,12 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
-/**
- * pcie_has_flr - check if a device supports function level resets
- * @dev: device to check
- *
- * Returns true if the device advertises support for PCIe function level
- * resets.
- */
-bool pcie_has_flr(struct pci_dev *dev)
-{
-	u32 cap;
-
-	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
-		return false;
-
-	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
-	return cap & PCI_EXP_DEVCAP_FLR;
-}
-EXPORT_SYMBOL_GPL(pcie_has_flr);
-
 /**
  * pcie_flr - initiate a PCIe function level reset
  * @dev: device to reset
  *
- * Initiate a function level reset on @dev.  The caller should ensure the
- * device supports FLR before calling this function, e.g. by using the
- * pcie_has_flr() helper.
+ * Initiate a function level reset unconditionally on @dev without
+ * checking any flags and DEVCAP
  */
 int pcie_flr(struct pci_dev *dev)
 {
@@ -4659,6 +4639,31 @@ int pcie_flr(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
+/**
+ * pcie_reset_flr - initiate a PCIe function level reset
+ * @dev: device to reset
+ * @probe: If set, only check if the device can be reset this way.
+ *
+ * Initiate a function level reset on @dev.
+ */
+int pcie_reset_flr(struct pci_dev *dev, int probe)
+{
+	u32 cap;
+
+	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)

drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4663

Quoted 10 lines
drivers/pci/pci.c +10 -0
+		return -ENOTTY;
+
+	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
+	if (!(cap & PCI_EXP_DEVCAP_FLR))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	return pcie_flr(dev);  

Christoph added pcie_flr() with a60a2b73ba69 ("PCI: Export pcie_flr()"), where the commit log says he split out the probing because "non-core callers already know their hardware."

Quoted 2 lines

It *is* reasonable to expect that drivers know whether their device supports FLR so they don't need to probe.

I don't think it changes your suggestion below, but this statement is a little troublesome when we look at devices running in VMs where we've been known to hide various capabilities, or simply quirks where some combination of a known device feature might be otherwise avoided. A more robust driver should try to make fewer assumptions for these cases, if not simply the inability to predict future changes to the hardware.

FLR should be a relatively rare event, but caching for driver API purposes seems reasonable. Thanks,

Alex

Quoted 145 lines

But we don't expose the "probe" argument outside the PCI core for any other reset methods, and I would like to avoid that here.

It seems excessive to have to read PCI_EXP_DEVCAP every time. PCI_EXP_DEVCAP_FLR is a read-only bit, and we should only need to look at it once.

What I would really like here is a single bit in the pci_dev that we could set at enumeration-time, e.g., something like this:

  struct pci_dev {
    ...
    unsigned int has_flr:1;
  };

  void set_pcie_port_type(...)    # during enumeration
  {
    pci_read_config_word(dev, pos + PCI_EXP_DEVCAP, &reg16);
    if (reg16 & PCI_EXP_DEVCAP_FLR)
      dev->has_flr = 1;
  }

  static void quirk_no_flr(...)
  {
    dev->has_flr = 0;             # get rid of PCI_DEV_FLAGS_NO_FLR_RESET
  }

  int pcie_flr(...)
  {
    if (!dev->has_flr)
      return -ENOTTY;
    if (!pci_wait_for_pending_transaction(dev))
      ...

}

I think this should be enough that we could get rid of pcie_has_flr() without having to expose the "probe" argument outside drivers/pci/.

Procedural note: if we *do* have to expose the "probe" argument, can you arrange it to have the correct type before touching the drivers, so we only have to touch the drivers once?

Quoted 101 lines
drivers/pci/pci.c +9 -7
+}
+EXPORT_SYMBOL_GPL(pcie_reset_flr);
+
 static int pci_af_flr(struct pci_dev *dev, int probe)
 {
 	int pos;
@@ -5139,11 +5144,9 @@ int __pci_reset_function_locked(struct pci_dev *dev)
 	rc = pci_dev_specific_reset(dev, 0);
 	if (rc != -ENOTTY)
 		return rc;
-	if (pcie_has_flr(dev)) {
-		rc = pcie_flr(dev);
-		if (rc != -ENOTTY)
-			return rc;
-	}
+	rc = pcie_reset_flr(dev, 0);
+	if (rc != -ENOTTY)
+		return rc;
 	rc = pci_af_flr(dev, 0);
 	if (rc != -ENOTTY)
 		return rc;
@@ -5174,8 +5177,9 @@ int pci_probe_reset_function(struct pci_dev *dev)
 	rc = pci_dev_specific_reset(dev, 1);
 	if (rc != -ENOTTY)
 		return rc;
-	if (pcie_has_flr(dev))
-		return 0;
+	rc = pcie_reset_flr(dev, 1);
+	if (rc != -ENOTTY)
+		return rc;
 	rc = pci_af_flr(dev, 1);
 	if (rc != -ENOTTY)
 		return rc;
drivers/pci/pcie/aer.c +5 -7
index ec943cee5..98077595a 100644
@@ -1405,13 +1405,11 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)
 	}
 
 	if (type == PCI_EXP_TYPE_RC_EC || type == PCI_EXP_TYPE_RC_END) {
-		if (pcie_has_flr(dev)) {
-			rc = pcie_flr(dev);
-			pci_info(dev, "has been reset (%d)\n", rc);
-		} else {
-			pci_info(dev, "not reset (no FLR support)\n");
-			rc = -ENOTTY;
-		}
+		rc = pcie_reset_flr(dev, 0);
+		if (!rc)
+			pci_info(dev, "has been reset\n");
+		else
+			pci_info(dev, "not reset (no FLR support: %d)\n", rc);
 	} else {
 		rc = pci_bus_error_reset(dev);
 		pci_info(dev, "%s Port link has been reset (%d)\n",
drivers/pci/quirks.c +3 -6
index d85914afe..f977ba79a 100644
@@ -3819,7 +3819,7 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)
 	u32 cfg;
 
 	if (dev->class != PCI_CLASS_STORAGE_EXPRESS ||
-	    !pcie_has_flr(dev) || !pci_resource_start(dev, 0))
+	    pcie_reset_flr(dev, 1) || !pci_resource_start(dev, 0))
 		return -ENOTTY;
 
 	if (probe)
@@ -3888,13 +3888,10 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)
  */
 static int delay_250ms_after_flr(struct pci_dev *dev, int probe)
 {
-	if (!pcie_has_flr(dev))
-		return -ENOTTY;
+	int ret = pcie_reset_flr(dev, probe);
 
 	if (probe)
-		return 0;
-
-	pcie_flr(dev);
+		return ret;
 
 	msleep(250);
 
include/linux/pci.h +1 -1
index c20211e59..20b90c205 100644
@@ -1225,7 +1225,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
 			     enum pci_bus_speed *speed,
 			     enum pcie_link_width *width);
 void pcie_print_link_status(struct pci_dev *dev);
-bool pcie_has_flr(struct pci_dev *dev);
+int pcie_reset_flr(struct pci_dev *dev, int probe);
 int pcie_flr(struct pci_dev *dev);
 int __pci_reset_function_locked(struct pci_dev *dev);
 int pci_reset_function(struct pci_dev *dev);
Signature · 1 line
2.31.1
#5 lore raw

Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods

In reply to Bjorn Helgaas
Quoted 147 lines · Bjorn Helgaas

[+cc Christoph, since he added pcie_flr()]

Quoted 5 lines · Amey Narkhede

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

I don't like the fact that we handle FLR differently from other types of reset, so I do like the fact that this makes them more consistent.

Quoted 83 lines
---
 drivers/crypto/cavium/nitrox/nitrox_main.c |  4 +-
 drivers/pci/pci.c                          | 62 ++++++++++++----------
 drivers/pci/pcie/aer.c                     | 12 ++---
 drivers/pci/quirks.c                       |  9 ++--
 include/linux/pci.h                        |  2 +-
 5 files changed, 43 insertions(+), 46 deletions(-)
drivers/crypto/cavium/nitrox/nitrox_main.c +1 -3
index facc8e6bc..15d6c8452 100644
@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)
 		return -ENOMEM;
 	}
 
-	/* check flr support */
-	if (pcie_has_flr(pdev))
-		pcie_flr(pdev);
+	pcie_reset_flr(pdev, 0);
 
 	pci_restore_state(pdev);
 
drivers/pci/pci.c +14 -22
index 452351025..3bf36924c 100644
@@ -4611,32 +4611,12 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
-/**
- * pcie_has_flr - check if a device supports function level resets
- * @dev: device to check
- *
- * Returns true if the device advertises support for PCIe function level
- * resets.
- */
-bool pcie_has_flr(struct pci_dev *dev)
-{
-	u32 cap;
-
-	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
-		return false;
-
-	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
-	return cap & PCI_EXP_DEVCAP_FLR;
-}
-EXPORT_SYMBOL_GPL(pcie_has_flr);
-
 /**
  * pcie_flr - initiate a PCIe function level reset
  * @dev: device to reset
  *
- * Initiate a function level reset on @dev.  The caller should ensure the
- * device supports FLR before calling this function, e.g. by using the
- * pcie_has_flr() helper.
+ * Initiate a function level reset unconditionally on @dev without
+ * checking any flags and DEVCAP
  */
 int pcie_flr(struct pci_dev *dev)
 {
@@ -4659,6 +4639,31 @@ int pcie_flr(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
+/**
+ * pcie_reset_flr - initiate a PCIe function level reset
+ * @dev: device to reset
+ * @probe: If set, only check if the device can be reset this way.
+ *
+ * Initiate a function level reset on @dev.
+ */
+int pcie_reset_flr(struct pci_dev *dev, int probe)
+{
+	u32 cap;
+
+	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)

drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4663

Quoted 10 lines
drivers/pci/pci.c +10 -0
+		return -ENOTTY;
+
+	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
+	if (!(cap & PCI_EXP_DEVCAP_FLR))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	return pcie_flr(dev);

Christoph added pcie_flr() with a60a2b73ba69 ("PCI: Export pcie_flr()"), where the commit log says he split out the probing because "non-core callers already know their hardware."

It *is* reasonable to expect that drivers know whether their device supports FLR so they don't need to probe.

But we don't expose the "probe" argument outside the PCI core for any other reset methods, and I would like to avoid that here.

It seems excessive to have to read PCI_EXP_DEVCAP every time. PCI_EXP_DEVCAP_FLR is a read-only bit, and we should only need to look at it once.

What I would really like here is a single bit in the pci_dev that we could set at enumeration-time, e.g., something like this:

  struct pci_dev {
    ...
    unsigned int has_flr:1;
  };

  void set_pcie_port_type(...)    # during enumeration
  {
    pci_read_config_word(dev, pos + PCI_EXP_DEVCAP, &reg16);
    if (reg16 & PCI_EXP_DEVCAP_FLR)
      dev->has_flr = 1;
  }

  static void quirk_no_flr(...)
  {
    dev->has_flr = 0;             # get rid of PCI_DEV_FLAGS_NO_FLR_RESET
  }

  int pcie_flr(...)
  {
    if (!dev->has_flr)
      return -ENOTTY;
    if (!pci_wait_for_pending_transaction(dev))
      ...
Quoted 9 lines

}

I think this should be enough that we could get rid of pcie_has_flr() without having to expose the "probe" argument outside drivers/pci/.

Procedural note: if we *do* have to expose the "probe" argument, can you arrange it to have the correct type before touching the drivers, so we only have to touch the drivers once?

Thanks for the details. I'll add dev->has_flr check in pcie_reset_flr. [...]

Thanks,
Amey

#6 lore raw

Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods

In reply to Amey Narkhede
Quoted 19 lines · Amey Narkhede

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

drivers/pci/pci.c +13 -0
+/**
+ * pcie_reset_flr - initiate a PCIe function level reset
+ * @dev: device to reset
+ * @probe: If set, only check if the device can be reset this way.
+ *
+ * Initiate a function level reset on @dev.
+ */
+int pcie_reset_flr(struct pci_dev *dev, int probe)
+{
+	u32 cap;
+
+	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
+		return -ENOTTY;

drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4664

Quoted 10 lines
drivers/pci/pci.c +10 -0
+
+	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
+	if (!(cap & PCI_EXP_DEVCAP_FLR))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	return pcie_flr(dev);
+}

Tangent: I've been told before, but I can't remember why we need the "probe" interface. Since we're looking at this area again, can we add a comment to clarify this?

Every time I read this, I wonder why we can't just get rid of the probe and attempt a reset. If it fails because it's not supported, we could just try the next one in the list.

#7 lore raw

Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods

Quoted 35 lines · Bjorn Helgaas
Quoted 19 lines · Amey Narkhede

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

drivers/pci/pci.c +13 -0
+/**
+ * pcie_reset_flr - initiate a PCIe function level reset
+ * @dev: device to reset
+ * @probe: If set, only check if the device can be reset this way.
+ *
+ * Initiate a function level reset on @dev.
+ */
+int pcie_reset_flr(struct pci_dev *dev, int probe)
+{
+	u32 cap;
+
+	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
+		return -ENOTTY;

drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4664

Quoted 10 lines
drivers/pci/pci.c +10 -0
+
+	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
+	if (!(cap & PCI_EXP_DEVCAP_FLR))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	return pcie_flr(dev);
+}

Tangent: I've been told before, but I can't remember why we need the "probe" interface. Since we're looking at this area again, can we add a comment to clarify this?

Quoted 3 lines

Every time I read this, I wonder why we can't just get rid of the probe and attempt a reset. If it fails because it's not supported, we could just try the next one in the list.

Part of the reason is to have same calling convention as other reset methods and other reason is devices that run in VMs where various capabilities can be hidden or have quirks for avoiding known troublesome combination of device features as Alex explained here
https://lore.kernel.org/linux-pci/20210624151242.ybew2z5rseuusj7v@archlinux/T/#mb67c09a2ce08ce4787652e4c0e7b9e5adf1df57a

On the side note as you suggested earlier to cache flr capability earlier the PCI_EXP_DEVCAP reading code won't be there in next version so its just trivial check(dev->has_flr).

Thanks,
Amey

#8 lore raw

Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods

[+to Alex]

Quoted 46 lines · Amey Narkhede
Quoted 35 lines · Bjorn Helgaas
Quoted 19 lines · Amey Narkhede

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

drivers/pci/pci.c +13 -0
+/**
+ * pcie_reset_flr - initiate a PCIe function level reset
+ * @dev: device to reset
+ * @probe: If set, only check if the device can be reset this way.
+ *
+ * Initiate a function level reset on @dev.
+ */
+int pcie_reset_flr(struct pci_dev *dev, int probe)
+{
+	u32 cap;
+
+	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
+		return -ENOTTY;

drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4664

Quoted 10 lines
drivers/pci/pci.c +10 -0
+
+	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
+	if (!(cap & PCI_EXP_DEVCAP_FLR))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	return pcie_flr(dev);
+}

Tangent: I've been told before, but I can't remember why we need the "probe" interface. Since we're looking at this area again, can we add a comment to clarify this?

Quoted 3 lines

Every time I read this, I wonder why we can't just get rid of the probe and attempt a reset. If it fails because it's not supported, we could just try the next one in the list.

Part of the reason is to have same calling convention as other reset methods and other reason is devices that run in VMs where various capabilities can be hidden or have quirks for avoiding known troublesome combination of device features as Alex explained here
https://lore.kernel.org/linux-pci/20210624151242.ybew2z5rseuusj7v@archlinux/T/#mb67c09a2ce08ce4787652e4c0e7b9e5adf1df57a

Quoted 3 lines

On the side note as you suggested earlier to cache flr capability earlier the PCI_EXP_DEVCAP reading code won't be there in next version so its just trivial check(dev->has_flr).

Sorry, I didn't make my question clear. I'm not asking why we're adding a "probe" argument to pcie_reset_flr() to make it consistent with pci_af_flr(), pci_pm_reset(), pci_parent_bus_reset(), etc. I like making the interfaces consistent.

What I'm asking here is why the "probe" argument exists for *any* of these interfaces and why pci_probe_reset_function() exists.

This is really more a question for Alex since it's a historical question, not anything directly related to your series. I'm not proposing *removing* the "probe" argument; I know it exists for a reason because I've asked about it before. But I forgot the answer, which makes me think a hint in the code would be useful.

Bjorn

#9 lore raw

Re: [PATCH v7 1/8] PCI: Add pcie_reset_flr to follow calling convention of other reset methods

Quoted 58 lines · On Thu, 24 Jun 2021 11:15:59 -0500 Bjorn Helgaas <[email protected]> wrote:

[+to Alex]

Quoted 46 lines · Amey Narkhede
Quoted 35 lines · Bjorn Helgaas
Quoted 19 lines · Amey Narkhede

Currently there is separate function pcie_has_flr() to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr() that follows the calling convention of reset methods.

drivers/pci/pci.c +13 -0
+/**
+ * pcie_reset_flr - initiate a PCIe function level reset
+ * @dev: device to reset
+ * @probe: If set, only check if the device can be reset this way.
+ *
+ * Initiate a function level reset on @dev.
+ */
+int pcie_reset_flr(struct pci_dev *dev, int probe)
+{
+	u32 cap;
+
+	if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
+		return -ENOTTY;

drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4664

Quoted 10 lines
drivers/pci/pci.c +10 -0
+
+	pcie_capability_read_dword(dev, PCI_EXP_DEVCAP, &cap);
+	if (!(cap & PCI_EXP_DEVCAP_FLR))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	return pcie_flr(dev);
+}  

Tangent: I've been told before, but I can't remember why we need the "probe" interface. Since we're looking at this area again, can we add a comment to clarify this?

Quoted 3 lines

Every time I read this, I wonder why we can't just get rid of the probe and attempt a reset. If it fails because it's not supported, we could just try the next one in the list.

Part of the reason is to have same calling convention as other reset methods and other reason is devices that run in VMs where various capabilities can be hidden or have quirks for avoiding known troublesome combination of device features as Alex explained here
https://lore.kernel.org/linux-pci/20210624151242.ybew2z5rseuusj7v@archlinux/T/#mb67c09a2ce08ce4787652e4c0e7b9e5adf1df57a

Quoted 3 lines

On the side note as you suggested earlier to cache flr capability earlier the PCI_EXP_DEVCAP reading code won't be there in next version so its just trivial check(dev->has_flr).

Sorry, I didn't make my question clear. I'm not asking why we're adding a "probe" argument to pcie_reset_flr() to make it consistent with pci_af_flr(), pci_pm_reset(), pci_parent_bus_reset(), etc. I like making the interfaces consistent.

Quoted 8 lines

What I'm asking here is why the "probe" argument exists for *any* of these interfaces and why pci_probe_reset_function() exists.

This is really more a question for Alex since it's a historical question, not anything directly related to your series. I'm not proposing *removing* the "probe" argument; I know it exists for a reason because I've asked about it before. But I forgot the answer, which makes me think a hint in the code would be useful.

Heh [1]

That might be what you're recalling, but in that case I was adding exported symbols that allowed probing bus vs slot reset because the scope of affected devices is different. My use case is testing whether the user owns all the affected devices, so it's really not a test-by-doing opportunity.

For these single-function scoped resets, as in the reply to [1] pci_probe_reset_function() isn't exported and the only caller is internal PCI code to determine whether to create the 'reset' sysfs attribute. Sure, as it exists today we could reset the device and test whether it worked to get that value, that's what vfio-pci does now before we give the device to the user, but the critical difference is that in the vfio case we always want to flush any state that might be leaked to the user and at device init time, doing so only invites issues.

This series obviously expands the scope of probing, we don't just want to know that there's at least one method available to us, but precisely which ones. It's rather impractical to try to reset a function a half dozen different ways on boot for the possibility that the admin might want to manipulate the reset order later. And oh gosh, if we don't cache the methods supported and re-test-by-doing when the attribute is written, let's just not go there. Thanks,

Alex

[1]https://lore.kernel.org/linux-pci/CAErSpo625CTnxZvy-gmy8VzxT4favF4s=_giU6nGey_N=VwK5A@mail.gmail.com/

#10 lore raw

PATCH v7 2/8 PCI: Add new array for keeping track of ordering of reset methods

In reply to Amey Narkhede

Introduce a new array reset_methods in struct pci_dev to keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.

---
 drivers/pci/pci.c   | 108 ++++++++++++++++++++++++++------------------
 drivers/pci/pci.h   |   8 +++-
 drivers/pci/probe.c |   5 +-
 include/linux/pci.h |   7 +++
 4 files changed, 81 insertions(+), 47 deletions(-)
drivers/pci/pci.c +65 -43
index 3bf36924c..39a9ea8bb 100644
@@ -72,6 +72,14 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)
 		msleep(delay);
 }
 
+bool pci_reset_supported(struct pci_dev *dev)
+{
+	u8 null_reset_methods[PCI_RESET_METHODS_NUM] = { 0 };
+
+	return memcmp(null_reset_methods,
+		      dev->reset_methods, PCI_RESET_METHODS_NUM);
+}
+
 #ifdef CONFIG_PCI_DOMAINS
 int pci_domains_supported = 1;
 #endif
@@ -5107,6 +5115,18 @@ static void pci_dev_restore(struct pci_dev *dev)
 		err_handler->reset_done(dev);
 }
 
+/*
+ * The ordering for functions in pci_reset_fn_methods is required for
+ * reset_methods byte array defined in struct pci_dev.
+ */
+const struct pci_reset_fn_method pci_reset_fn_methods[] = {
+	{ &pci_dev_specific_reset, .name = "device_specific" },
+	{ &pcie_reset_flr, .name = "flr" },
+	{ &pci_af_flr, .name = "af_flr" },
+	{ &pci_pm_reset, .name = "pm" },
+	{ &pci_reset_bus_function, .name = "bus" },
+};
+
 /**
  * __pci_reset_function_locked - reset a PCI device function while holding
  * the @dev mutex lock.
@@ -5129,65 +5149,67 @@ static void pci_dev_restore(struct pci_dev *dev)
  */
 int __pci_reset_function_locked(struct pci_dev *dev)
 {
-	int rc;
+	int i, rc = -ENOTTY;
+	u8 prio;
 
 	might_sleep();
 
-	/*
-	 * A reset method returns -ENOTTY if it doesn't support this device
-	 * and we should try the next method.
-	 *
-	 * If it returns 0 (success), we're finished.  If it returns any
-	 * other error, we're also finished: this indicates that further
-	 * reset mechanisms might be broken on the device.
-	 */
-	rc = pci_dev_specific_reset(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pcie_reset_flr(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_af_flr(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_pm_reset(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	return pci_reset_bus_function(dev, 0);
+	for (prio = PCI_RESET_METHODS_NUM; prio; prio--) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (dev->reset_methods[i] == prio) {
+				/*
+				 * A reset method returns -ENOTTY if it doesn't
+				 * support this device and we should try the
+				 * next method.
+				 *
+				 * If it returns 0 (success), we're finished.
+				 * If it returns any other error, we're also
+				 * finished: this indicates that further reset
+				 * mechanisms might be broken on the device.
+				 */
+				rc = pci_reset_fn_methods[i].reset_fn(dev, 0);
+				if (rc != -ENOTTY)
+					return rc;
+				break;
+			}
+		}
+		if (i == PCI_RESET_METHODS_NUM)
+			break;
+	}
+	return rc;
 }
 EXPORT_SYMBOL_GPL(__pci_reset_function_locked);
 
 /**
- * pci_probe_reset_function - check whether the device can be safely reset
- * @dev: PCI device to reset
+ * pci_init_reset_methods - check whether device can be safely reset
+ * and store supported reset mechanisms.
+ * @dev: PCI device to check for reset mechanisms
  *
  * Some devices allow an individual function to be reset without affecting
  * other functions in the same device.  The PCI device must be responsive
- * to PCI config space in order to use this function.
+ * to reads and writes to its PCI config space in order to use this function.
  *
- * Returns 0 if the device function can be reset or negative if the
- * device doesn't support resetting a single function.
+ * Stores reset mechanisms supported by device in reset_methods byte array
+ * which is a member of struct pci_dev.
  */
-int pci_probe_reset_function(struct pci_dev *dev)
+void pci_init_reset_methods(struct pci_dev *dev)
 {
-	int rc;
+	int i, rc;
+	u8 prio = PCI_RESET_METHODS_NUM;
+	u8 reset_methods[PCI_RESET_METHODS_NUM] = { 0 };
 
-	might_sleep();
+	BUILD_BUG_ON(ARRAY_SIZE(pci_reset_fn_methods) != PCI_RESET_METHODS_NUM);
 
-	rc = pci_dev_specific_reset(dev, 1);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pcie_reset_flr(dev, 1);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_af_flr(dev, 1);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_pm_reset(dev, 1);
-	if (rc != -ENOTTY)
-		return rc;
+	might_sleep();
 
-	return pci_reset_bus_function(dev, 1);
+	for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+		rc = pci_reset_fn_methods[i].reset_fn(dev, 1);
+		if (!rc)
+			reset_methods[i] = prio--;
+		else if (rc != -ENOTTY)
+			break;
+	}
+	memcpy(dev->reset_methods, reset_methods, sizeof(reset_methods));
 }
 
 /**
drivers/pci/pci.h +7 -1
index 37c913bbc..13ec6bd6f 100644
@@ -33,7 +33,7 @@ enum pci_mmap_api {
 int pci_mmap_fits(struct pci_dev *pdev, int resno, struct vm_area_struct *vmai,
 		  enum pci_mmap_api mmap_api);
 
-int pci_probe_reset_function(struct pci_dev *dev);
+void pci_init_reset_methods(struct pci_dev *dev);
 int pci_bridge_secondary_bus_reset(struct pci_dev *dev);
 int pci_bus_error_reset(struct pci_dev *dev);
 
@@ -606,6 +606,12 @@ struct pci_dev_reset_methods {
 	int (*reset)(struct pci_dev *dev, int probe);
 };
 
+struct pci_reset_fn_method {
+	int (*reset_fn)(struct pci_dev *pdev, int probe);
+	char *name;
+};
+
+extern const struct pci_reset_fn_method pci_reset_fn_methods[];
 #ifdef CONFIG_PCI_QUIRKS
 int pci_dev_specific_reset(struct pci_dev *dev, int probe);
 #else
drivers/pci/probe.c +2 -3
index 3a62d09b8..8cf532681 100644
@@ -2404,9 +2404,8 @@ static void pci_init_capabilities(struct pci_dev *dev)
 	pci_rcec_init(dev);		/* Root Complex Event Collector */
 
 	pcie_report_downtraining(dev);
-
-	if (pci_probe_reset_function(dev) == 0)
-		dev->reset_fn = 1;
+	pci_init_reset_methods(dev);
+	dev->reset_fn = pci_reset_supported(dev);
 }
 
 /*
include/linux/pci.h +7 -0
index 20b90c205..0955246f8 100644
@@ -49,6 +49,8 @@
 			       PCI_STATUS_SIG_TARGET_ABORT | \
 			       PCI_STATUS_PARITY)
 
+#define PCI_RESET_METHODS_NUM 5
+
 /*
  * The PCI interface treats multi-function devices as independent
  * devices.  The slot/function address of each device is encoded
@@ -505,6 +507,10 @@ struct pci_dev {
 	char		*driver_override; /* Driver name to force a match */
 
 	unsigned long	priv_flags;	/* Private flags for the PCI driver */
+	/*
+	 * See pci_reset_fn_methods array in pci.c for ordering.
+	 */
+	u8 reset_methods[PCI_RESET_METHODS_NUM];	/* Reset methods ordered by priority */
 };
 
 static inline struct pci_dev *pci_physfn(struct pci_dev *dev)
@@ -1227,6 +1233,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
 void pcie_print_link_status(struct pci_dev *dev);
 int pcie_reset_flr(struct pci_dev *dev, int probe);
 int pcie_flr(struct pci_dev *dev);
+bool pci_reset_supported(struct pci_dev *dev);
 int __pci_reset_function_locked(struct pci_dev *dev);
 int pci_reset_function(struct pci_dev *dev);
 int pci_reset_function_locked(struct pci_dev *dev);
Signature · 1 line
2.31.1
Tested-by on v7 2/8
#11 lore raw

Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods

Quoted 10 lines · Amey Narkhede

Introduce a new array reset_methods in struct pci_dev to keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.

#12 lore raw

Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods

In reply to Amey Narkhede

"Add new" in subject and below is slightly redundant.

Quoted 22 lines · Amey Narkhede

Introduce a new array reset_methods in struct pci_dev to keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.

---
 drivers/pci/pci.c   | 108 ++++++++++++++++++++++++++------------------
 drivers/pci/pci.h   |   8 +++-
 drivers/pci/probe.c |   5 +-
 include/linux/pci.h |   7 +++
 4 files changed, 81 insertions(+), 47 deletions(-)
drivers/pci/pci.c +0 -0
index 3bf36924c..39a9ea8bb 100644

drivers/pci/pci.c static void pci_dev_d3_sleep(struct pci_dev *dev) · line 80

Quoted 9 lines
drivers/pci/pci.c +6 -0
@@ -72,6 +72,14 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)
 		msleep(delay);
 }
 
+bool pci_reset_supported(struct pci_dev *dev)
+{
+	u8 null_reset_methods[PCI_RESET_METHODS_NUM] = { 0 };
+
+	return memcmp(null_reset_methods,
+		      dev->reset_methods, PCI_RESET_METHODS_NUM);

memcmp() doesn't actually return a bool. Either just return int and rely on the C "anything non-zero is true, zero is false" or convert the memcmp result to bool, i.e., something like:

if (memcmp(...) == 0)

    return true;

return false;

drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5120

Quoted 12 lines
drivers/pci/pci.c +5 -0
+}
+
 #ifdef CONFIG_PCI_DOMAINS
 int pci_domains_supported = 1;
 #endif
@@ -5107,6 +5115,18 @@ static void pci_dev_restore(struct pci_dev *dev)
 		err_handler->reset_done(dev);
 }
 
+/*
+ * The ordering for functions in pci_reset_fn_methods is required for
+ * reset_methods byte array defined in struct pci_dev.

I'm not quite sure what this comment is telling me. What breaks if I change the order? If I add a new method, how do I know where to put it?

By reading the code, I infer that:

- Each dev has dev->reset_methods[PCI_RESET_METHODS_NUM]

- dev->reset_methods[i] corresponds to pci_reset_fn_methods[i]

- dev->reset_methods[i] == 0 means dev doesn't support that method

- Otherwise, dev->reset_methods[i] is a value in the range of [1, PCI_RESET_METHODS_NUM], and the higher the number, the higher the reset method priority

- The order in pci_reset_fn_methods[] determines the initial priority via pci_init_reset_methods(), but the priority can be changed via sysfs

Quoted 49 lines
drivers/pci/pci.c +17 -22
+ */
+const struct pci_reset_fn_method pci_reset_fn_methods[] = {
+	{ &pci_dev_specific_reset, .name = "device_specific" },
+	{ &pcie_reset_flr, .name = "flr" },
+	{ &pci_af_flr, .name = "af_flr" },
+	{ &pci_pm_reset, .name = "pm" },
+	{ &pci_reset_bus_function, .name = "bus" },
+};
+
 /**
  * __pci_reset_function_locked - reset a PCI device function while holding
  * the @dev mutex lock.
@@ -5129,65 +5149,67 @@ static void pci_dev_restore(struct pci_dev *dev)
  */
 int __pci_reset_function_locked(struct pci_dev *dev)
 {
-	int rc;
+	int i, rc = -ENOTTY;
+	u8 prio;
 
 	might_sleep();
 
-	/*
-	 * A reset method returns -ENOTTY if it doesn't support this device
-	 * and we should try the next method.
-	 *
-	 * If it returns 0 (success), we're finished.  If it returns any
-	 * other error, we're also finished: this indicates that further
-	 * reset mechanisms might be broken on the device.
-	 */
-	rc = pci_dev_specific_reset(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pcie_reset_flr(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_af_flr(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_pm_reset(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	return pci_reset_bus_function(dev, 0);
+	for (prio = PCI_RESET_METHODS_NUM; prio; prio--) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (dev->reset_methods[i] == prio) {
+				/*
+				 * A reset method returns -ENOTTY if it doesn't
+				 * support this device and we should try the

drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5172

Quoted 10 lines
drivers/pci/pci.c +10 -0
+				 * next method.
+				 *
+				 * If it returns 0 (success), we're finished.
+				 * If it returns any other error, we're also
+				 * finished: this indicates that further reset
+				 * mechanisms might be broken on the device.
+				 */
+				rc = pci_reset_fn_methods[i].reset_fn(dev, 0);
+				if (rc != -ENOTTY)
+					return rc;

Maybe leave the comment outside the loop where it used to be so the text lines are longer and it's easier to read.

drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5179

Quoted 7 lines
drivers/pci/pci.c +7 -0
+				break;
+			}
+		}
+		if (i == PCI_RESET_METHODS_NUM)
+			break;
+	}
+	return rc;

I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:

  const struct pci_reset_fn_method pci_reset_fn_methods[] = {
    { },                                                     # 0
    { &pci_dev_specific_reset, .name = "device_specific" },  # 1
    { &pci_dev_acpi_reset, .name = "acpi" },                 # 2
    { &pcie_reset_flr, .name = "flr" },                      # 3
    { &pci_af_flr, .name = "af_flr" },                       # 4
    { &pci_pm_reset, .name = "pm" },                         # 5
    { &pci_reset_bus_function, .name = "bus" },              # 6
  };

dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority order

dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means only pci_dev_specific_reset is supported

dev->reset_methods[] = [3, 5, 0, 0, 0, 0]
means pcie_reset_flr and pci_pm_reset are supported, in that priority order

Then we wouldn't need the nested loop and the return value would be easier to analyze:

  for (i = 0; i < PCI_RESET_METHODS_NUM && (m = dev->reset_methods[i]); i++) {
    rc = pci_reset_fn_methods[m].reset_fn(dev, 0);
    if (rc == 0)
      return 0;
    if (rc != -ENOTTY)
      return rc;
  }
  return -ENOTTY;

pci_init_reset_methods() would be something like:

  n = 0;
  for (i = 1; i < PCI_RESET_METHODS_NUM; i++) {
    rc = pci_reset_fn_methods[i].reset_fn(dev, 1);
    if (!rc)
      dev->reset_methods[n++] = i;
    if (rc != -ENOTTY)
      return;
  }
Quoted 104 lines
drivers/pci/pci.c +20 -21
 }
 EXPORT_SYMBOL_GPL(__pci_reset_function_locked);
 
 /**
- * pci_probe_reset_function - check whether the device can be safely reset
- * @dev: PCI device to reset
+ * pci_init_reset_methods - check whether device can be safely reset
+ * and store supported reset mechanisms.
+ * @dev: PCI device to check for reset mechanisms
  *
  * Some devices allow an individual function to be reset without affecting
  * other functions in the same device.  The PCI device must be responsive
- * to PCI config space in order to use this function.
+ * to reads and writes to its PCI config space in order to use this function.
  *
- * Returns 0 if the device function can be reset or negative if the
- * device doesn't support resetting a single function.
+ * Stores reset mechanisms supported by device in reset_methods byte array
+ * which is a member of struct pci_dev.
  */
-int pci_probe_reset_function(struct pci_dev *dev)
+void pci_init_reset_methods(struct pci_dev *dev)
 {
-	int rc;
+	int i, rc;
+	u8 prio = PCI_RESET_METHODS_NUM;
+	u8 reset_methods[PCI_RESET_METHODS_NUM] = { 0 };
 
-	might_sleep();
+	BUILD_BUG_ON(ARRAY_SIZE(pci_reset_fn_methods) != PCI_RESET_METHODS_NUM);
 
-	rc = pci_dev_specific_reset(dev, 1);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pcie_reset_flr(dev, 1);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_af_flr(dev, 1);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_pm_reset(dev, 1);
-	if (rc != -ENOTTY)
-		return rc;
+	might_sleep();
 
-	return pci_reset_bus_function(dev, 1);
+	for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+		rc = pci_reset_fn_methods[i].reset_fn(dev, 1);
+		if (!rc)
+			reset_methods[i] = prio--;
+		else if (rc != -ENOTTY)
+			break;
+	}
+	memcpy(dev->reset_methods, reset_methods, sizeof(reset_methods));
 }
 
 /**
drivers/pci/pci.h +7 -1
index 37c913bbc..13ec6bd6f 100644
@@ -33,7 +33,7 @@ enum pci_mmap_api {
 int pci_mmap_fits(struct pci_dev *pdev, int resno, struct vm_area_struct *vmai,
 		  enum pci_mmap_api mmap_api);
 
-int pci_probe_reset_function(struct pci_dev *dev);
+void pci_init_reset_methods(struct pci_dev *dev);
 int pci_bridge_secondary_bus_reset(struct pci_dev *dev);
 int pci_bus_error_reset(struct pci_dev *dev);
 
@@ -606,6 +606,12 @@ struct pci_dev_reset_methods {
 	int (*reset)(struct pci_dev *dev, int probe);
 };
 
+struct pci_reset_fn_method {
+	int (*reset_fn)(struct pci_dev *pdev, int probe);
+	char *name;
+};
+
+extern const struct pci_reset_fn_method pci_reset_fn_methods[];
 #ifdef CONFIG_PCI_QUIRKS
 int pci_dev_specific_reset(struct pci_dev *dev, int probe);
 #else
drivers/pci/probe.c +2 -3
index 3a62d09b8..8cf532681 100644
@@ -2404,9 +2404,8 @@ static void pci_init_capabilities(struct pci_dev *dev)
 	pci_rcec_init(dev);		/* Root Complex Event Collector */
 
 	pcie_report_downtraining(dev);
-
-	if (pci_probe_reset_function(dev) == 0)
-		dev->reset_fn = 1;
+	pci_init_reset_methods(dev);
+	dev->reset_fn = pci_reset_supported(dev);
 }
 
 /*
include/linux/pci.h +0 -0
index 20b90c205..0955246f8 100644

include/linux/pci.h line 52

Quoted 4 lines
include/linux/pci.h +1 -0
@@ -49,6 +49,8 @@
 			       PCI_STATUS_SIG_TARGET_ABORT | \
 			       PCI_STATUS_PARITY)
 
+#define PCI_RESET_METHODS_NUM 5

I'm pretty sure this needs to be kept in sync with something, maybe ARRAY_SIZE(pci_reset_fn_methods)? We need some mechanism to enforce this, or at the very least, a comment. Oh, I see you have a BUILD_BUG_ON() in pci_init_reset_methods(). That's good, but a comment here would help, too.

This name should be something like "PCI_RESET_METHODS" or "PCI_NUM_RESET_METHODS". Putting "_NUM" at the end makes it sounds like we're referring to one specific method.

Quoted 15 lines
include/linux/pci.h +4 -0
 /*
  * The PCI interface treats multi-function devices as independent
  * devices.  The slot/function address of each device is encoded
@@ -505,6 +507,10 @@ struct pci_dev {
 	char		*driver_override; /* Driver name to force a match */
 
 	unsigned long	priv_flags;	/* Private flags for the PCI driver */
+	/*
+	 * See pci_reset_fn_methods array in pci.c for ordering.
+	 */
+	u8 reset_methods[PCI_RESET_METHODS_NUM];	/* Reset methods ordered by priority */
 };
 
 static inline struct pci_dev *pci_physfn(struct pci_dev *dev)

include/linux/pci.h u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev, · line 1236

Quoted 4 lines
include/linux/pci.h +1 -0
@@ -1227,6 +1233,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
 void pcie_print_link_status(struct pci_dev *dev);
 int pcie_reset_flr(struct pci_dev *dev, int probe);
 int pcie_flr(struct pci_dev *dev);
+bool pci_reset_supported(struct pci_dev *dev);

This function isn't used outside drivers/pci/, so I'd rather have the prototype in drivers/pci/pci.h.

Quoted 6 lines
include/linux/pci.h +0 -0
 int __pci_reset_function_locked(struct pci_dev *dev);
 int pci_reset_function(struct pci_dev *dev);
 int pci_reset_function_locked(struct pci_dev *dev);
Signature · 1 line
2.31.1
#13 lore raw

Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods

Quoted 68 lines · Bjorn Helgaas

"Add new" in subject and below is slightly redundant.

Quoted 22 lines · Amey Narkhede

Introduce a new array reset_methods in struct pci_dev to keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.

---
 drivers/pci/pci.c   | 108 ++++++++++++++++++++++++++------------------
 drivers/pci/pci.h   |   8 +++-
 drivers/pci/probe.c |   5 +-
 include/linux/pci.h |   7 +++
 4 files changed, 81 insertions(+), 47 deletions(-)
drivers/pci/pci.c +0 -0
index 3bf36924c..39a9ea8bb 100644

drivers/pci/pci.c static void pci_dev_d3_sleep(struct pci_dev *dev) · line 80

Quoted 9 lines
drivers/pci/pci.c +6 -0
@@ -72,6 +72,14 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)
 		msleep(delay);
 }
 
+bool pci_reset_supported(struct pci_dev *dev)
+{
+	u8 null_reset_methods[PCI_RESET_METHODS_NUM] = { 0 };
+
+	return memcmp(null_reset_methods,
+		      dev->reset_methods, PCI_RESET_METHODS_NUM);

memcmp() doesn't actually return a bool. Either just return int and rely on the C "anything non-zero is true, zero is false" or convert the memcmp result to bool, i.e., something like:

if (memcmp(...) == 0)

    return true;

return false;

drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5120

Quoted 12 lines
drivers/pci/pci.c +5 -0
+}
+
 #ifdef CONFIG_PCI_DOMAINS
 int pci_domains_supported = 1;
 #endif
@@ -5107,6 +5115,18 @@ static void pci_dev_restore(struct pci_dev *dev)
 		err_handler->reset_done(dev);
 }
 
+/*
+ * The ordering for functions in pci_reset_fn_methods is required for
+ * reset_methods byte array defined in struct pci_dev.

I'm not quite sure what this comment is telling me. What breaks if I change the order? If I add a new method, how do I know where to put it?

By reading the code, I infer that:

- Each dev has dev->reset_methods[PCI_RESET_METHODS_NUM]

- dev->reset_methods[i] corresponds to pci_reset_fn_methods[i]

- dev->reset_methods[i] == 0 means dev doesn't support that method

Quoted 8 lines

- Otherwise, dev->reset_methods[i] is a value in the range of [1, PCI_RESET_METHODS_NUM], and the higher the number, the higher the reset method priority

- The order in pci_reset_fn_methods[] determines the initial priority via pci_init_reset_methods(), but the priority can be changed via sysfs

Correct. I agree the comment is not clear. Adding new reset method won't break anything unless default order is changed and user has some assumptions from previous versions of kernel.

Quoted 111 lines
Quoted 49 lines
drivers/pci/pci.c +17 -22
+ */
+const struct pci_reset_fn_method pci_reset_fn_methods[] = {
+	{ &pci_dev_specific_reset, .name = "device_specific" },
+	{ &pcie_reset_flr, .name = "flr" },
+	{ &pci_af_flr, .name = "af_flr" },
+	{ &pci_pm_reset, .name = "pm" },
+	{ &pci_reset_bus_function, .name = "bus" },
+};
+
 /**
  * __pci_reset_function_locked - reset a PCI device function while holding
  * the @dev mutex lock.
@@ -5129,65 +5149,67 @@ static void pci_dev_restore(struct pci_dev *dev)
  */
 int __pci_reset_function_locked(struct pci_dev *dev)
 {
-	int rc;
+	int i, rc = -ENOTTY;
+	u8 prio;
 
 	might_sleep();
 
-	/*
-	 * A reset method returns -ENOTTY if it doesn't support this device
-	 * and we should try the next method.
-	 *
-	 * If it returns 0 (success), we're finished.  If it returns any
-	 * other error, we're also finished: this indicates that further
-	 * reset mechanisms might be broken on the device.
-	 */
-	rc = pci_dev_specific_reset(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pcie_reset_flr(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_af_flr(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	rc = pci_pm_reset(dev, 0);
-	if (rc != -ENOTTY)
-		return rc;
-	return pci_reset_bus_function(dev, 0);
+	for (prio = PCI_RESET_METHODS_NUM; prio; prio--) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (dev->reset_methods[i] == prio) {
+				/*
+				 * A reset method returns -ENOTTY if it doesn't
+				 * support this device and we should try the

drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5172

Quoted 10 lines
drivers/pci/pci.c +10 -0
+				 * next method.
+				 *
+				 * If it returns 0 (success), we're finished.
+				 * If it returns any other error, we're also
+				 * finished: this indicates that further reset
+				 * mechanisms might be broken on the device.
+				 */
+				rc = pci_reset_fn_methods[i].reset_fn(dev, 0);
+				if (rc != -ENOTTY)
+					return rc;

Maybe leave the comment outside the loop where it used to be so the text lines are longer and it's easier to read.

drivers/pci/pci.c static void pci_dev_restore(struct pci_dev *dev) · line 5179

Quoted 7 lines
drivers/pci/pci.c +7 -0
+				break;
+			}
+		}
+		if (i == PCI_RESET_METHODS_NUM)
+			break;
+	}
+	return rc;

I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:

  const struct pci_reset_fn_method pci_reset_fn_methods[] = {
    { },                                                     # 0
    { &pci_dev_specific_reset, .name = "device_specific" },  # 1
    { &pci_dev_acpi_reset, .name = "acpi" },                 # 2
    { &pcie_reset_flr, .name = "flr" },                      # 3
    { &pci_af_flr, .name = "af_flr" },                       # 4
    { &pci_pm_reset, .name = "pm" },                         # 5
    { &pci_reset_bus_function, .name = "bus" },              # 6
  };

dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority order

dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means only pci_dev_specific_reset is supported

dev->reset_methods[] = [3, 5, 0, 0, 0, 0]
means pcie_reset_flr and pci_pm_reset are supported, in that priority order

Then we wouldn't need the nested loop and the return value would be easier to analyze:

  for (i = 0; i < PCI_RESET_METHODS_NUM && (m = dev->reset_methods[i]); i++) {
    rc = pci_reset_fn_methods[m].reset_fn(dev, 0);
    if (rc == 0)
      return 0;
    if (rc != -ENOTTY)
      return rc;
  }
  return -ENOTTY;

pci_init_reset_methods() would be something like:

Quoted 9 lines
  n = 0;
  for (i = 1; i < PCI_RESET_METHODS_NUM; i++) {
    rc = pci_reset_fn_methods[i].reset_fn(dev, 1);
    if (!rc)
      dev->reset_methods[n++] = i;
    if (rc != -ENOTTY)
      return;
  }

I had similar idea initially but couldn't put it in words nicely thanks for this. I'll update this.
[...]

Thanks,
Amey

#14 lore raw

Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods

Hi Bjorn,

Quoted 19 lines · Amey Narkhede

I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:

  const struct pci_reset_fn_method pci_reset_fn_methods[] = {
    { },                                                     # 0
    { &pci_dev_specific_reset, .name = "device_specific" },  # 1
    { &pci_dev_acpi_reset, .name = "acpi" },                 # 2
    { &pcie_reset_flr, .name = "flr" },                      # 3
    { &pci_af_flr, .name = "af_flr" },                       # 4
    { &pci_pm_reset, .name = "pm" },                         # 5
    { &pci_reset_bus_function, .name = "bus" },              # 6
  };

dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority order

Quoted 6 lines

dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means only pci_dev_specific_reset is supported

dev->reset_methods[] = [3, 5, 0, 0, 0, 0]
means pcie_reset_flr and pci_pm_reset are supported, in that priority order

What about keeping two bitmap fields 'resets_supported' and 'resets_enabled' in pci_dev object and mange it through sysfs and probe helper function. We can avoid two loops multiple paces and takes only 2Bytes of memory to keep track resets.

resets_supported  ---> initialized during pci_dev setup
resets_enabled ---> Exposed to userspace through sysfs by default set to resets_supported

include/linux/pci.h:
------------------------
/* Different types of PCI resets possible, lower number is higher priority */ #define PCI_RESET_METHOD_DEVSPEC     0
#define PCI_RESET_METHOD_ACPI            1
#define PCI_RESET_METHOD_FLR              2
#define PCI_RESET_METHOD_Af_FLR         3
#define PCI_RESET_METHOD_PM               4
#define PCI_RESET_METHOD_BUS             5
#define PCI_RESET_METHOD_MAX            6

struct pci_dev {
    ...
        u8              resets_supported;
        u8              resets_enabled;
};

static inline bool pci_reset_supported(struct pci_dev *dev) {
        return !!(dev->resets_supported);
}

drivers/pci/pci.c:
--------------------
const struct pci_reset_fn_method pci_reset_fn_methods[PCI_RESET_METHOD_MAX] = { [PCI_RESET_METHOD_DEVSPEC] = { &pci_dev_specific_reset,
                                                                   .name = "device_specific" }, [PCI_RESET_METHOD_ACPI] = { &pci_dev_acpi_reset, .name = "acpi" }, [PCI_RESET_METHOD_FLR] = { &pcie_reset_flr, .name = "flr" }, [PCI_RESET_METHOD_Af_FLR] = { &pci_af_flr, .name = "af_flr" }, [PCI_RESET_METHOD_PM] = { &pci_pm_reset, .name = "pm" }, [PCI_RESET_METHOD_BUS] = { &pci_reset_bus_function, .name = "bus" }
};

void pci_init_reset_methods(struct pci_dev *dev)
{
        int i, rc;

        BUILD_BUG_ON(ARRAY_SIZE(pci_reset_fn_methods) != PCI_RESET_METHOD_MAX); might_sleep();

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                rc = pci_reset_fn_methods[i].reset_fn(dev, PCI_RESET_PROBE); if (!rc)
                        dev->resets_supported |= BIT(i);
                else if (rc != -ENOTTY)
                        break;
        }
        dev->resets_enabled = dev->resets_supported;
}

int __pci_reset_function_locked(struct pci_dev *dev) {
        int i, rc = -ENOTTY;

        might_sleep();

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                if (dev->resets_enabled & BIT(i)) {
                        rc = pci_reset_fn_methods[i].reset_fn(dev, 0); if (rc != -ENOTTY)
                                return rc;
                }
        }

        if (rc == -ENOTTY)
                pci_warn(dev, "No reset happened reason %s\n", !!dev->resets_supported ? "disabled by user" : "not supported");

        return rc;
}

drivers/pci/pci-sysfs.c
----------------------------
static ssize_t reset_method_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) {
        struct pci_dev *pdev = to_pci_dev(dev);
        u8 resets_enabled = 0;
...
        if (sysfs_streq(options, "default")) { pdev->resets_enabled = pdev->resets_supported; goto set_reset_methods;
        }

        while ((name = strsep(&options, ",")) != NULL) { if (sysfs_streq(name, "")) continue;
                name = strim(name);

                for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                        if ((pdev->resets_supported & BIT(i)) && sysfs_streq(name, pci_reset_fn_methods[i].name)) { resets_enabled |= BIT(i);
                                break;
                        }
                }
...
        }

set_reset_methods:
        kfree(options);
        pdev->resets_enabled =  resets_enabled;
        return count;
}

static ssize_t reset_method_show(struct device *dev, struct device_attribute *attr, char *buf)
{
        struct pci_dev *pdev = to_pci_dev(dev); ssize_t len = 0;
        int i;

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                if (pdev->resets_enabled & BIT(i))
                        len += sysfs_emit_at(buf, len, "%s%s", len ? "," : "", pci_reset_fn_methods[i].name); }
        len += sysfs_emit_at(buf, len, len ? "\n" : "");

        return len;
}

#15 lore raw

Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods

Quoted 161 lines · Shanker R Donthineni

Hi Bjorn,

Quoted 19 lines · Amey Narkhede

I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:

  const struct pci_reset_fn_method pci_reset_fn_methods[] = {
    { },                                                     # 0
    { &pci_dev_specific_reset, .name = "device_specific" },  # 1
    { &pci_dev_acpi_reset, .name = "acpi" },                 # 2
    { &pcie_reset_flr, .name = "flr" },                      # 3
    { &pci_af_flr, .name = "af_flr" },                       # 4
    { &pci_pm_reset, .name = "pm" },                         # 5
    { &pci_reset_bus_function, .name = "bus" },              # 6
  };

dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority order

Quoted 6 lines

dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means only pci_dev_specific_reset is supported

dev->reset_methods[] = [3, 5, 0, 0, 0, 0]
means pcie_reset_flr and pci_pm_reset are supported, in that priority order

What about keeping two bitmap fields 'resets_supported' and 'resets_enabled' in pci_dev object and mange it through sysfs and probe helper function. We can avoid two loops multiple paces and takes only 2Bytes of memory to keep track resets.

resets_supported  ---> initialized during pci_dev setup
resets_enabled ---> Exposed to userspace through sysfs by default set to resets_supported

include/linux/pci.h:
------------------------
/* Different types of PCI resets possible, lower number is higher priority */ #define PCI_RESET_METHOD_DEVSPEC     0
#define PCI_RESET_METHOD_ACPI            1
#define PCI_RESET_METHOD_FLR              2
#define PCI_RESET_METHOD_Af_FLR         3
#define PCI_RESET_METHOD_PM               4
#define PCI_RESET_METHOD_BUS             5
#define PCI_RESET_METHOD_MAX            6

struct pci_dev {
    ...
        u8              resets_supported;
        u8              resets_enabled;
};

static inline bool pci_reset_supported(struct pci_dev *dev) {
        return !!(dev->resets_supported);
}

drivers/pci/pci.c:
--------------------
const struct pci_reset_fn_method pci_reset_fn_methods[PCI_RESET_METHOD_MAX] = { [PCI_RESET_METHOD_DEVSPEC] = { &pci_dev_specific_reset,
                                                                   .name = "device_specific" }, [PCI_RESET_METHOD_ACPI] = { &pci_dev_acpi_reset, .name = "acpi" }, [PCI_RESET_METHOD_FLR] = { &pcie_reset_flr, .name = "flr" }, [PCI_RESET_METHOD_Af_FLR] = { &pci_af_flr, .name = "af_flr" }, [PCI_RESET_METHOD_PM] = { &pci_pm_reset, .name = "pm" }, [PCI_RESET_METHOD_BUS] = { &pci_reset_bus_function, .name = "bus" }
};

void pci_init_reset_methods(struct pci_dev *dev)
{
        int i, rc;

        BUILD_BUG_ON(ARRAY_SIZE(pci_reset_fn_methods) != PCI_RESET_METHOD_MAX); might_sleep();

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                rc = pci_reset_fn_methods[i].reset_fn(dev, PCI_RESET_PROBE); if (!rc)
                        dev->resets_supported |= BIT(i);
                else if (rc != -ENOTTY)
                        break;
        }
        dev->resets_enabled = dev->resets_supported;
}

int __pci_reset_function_locked(struct pci_dev *dev) {
        int i, rc = -ENOTTY;

        might_sleep();

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                if (dev->resets_enabled & BIT(i)) {
                        rc = pci_reset_fn_methods[i].reset_fn(dev, 0); if (rc != -ENOTTY)
                                return rc;
                }
        }

        if (rc == -ENOTTY)
                pci_warn(dev, "No reset happened reason %s\n", !!dev->resets_supported ? "disabled by user" : "not supported");

        return rc;
}

drivers/pci/pci-sysfs.c
----------------------------
static ssize_t reset_method_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) {
        struct pci_dev *pdev = to_pci_dev(dev);
        u8 resets_enabled = 0;
...
        if (sysfs_streq(options, "default")) { pdev->resets_enabled = pdev->resets_supported; goto set_reset_methods;
        }

        while ((name = strsep(&options, ",")) != NULL) { if (sysfs_streq(name, "")) continue;
                name = strim(name);

                for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                        if ((pdev->resets_supported & BIT(i)) && sysfs_streq(name, pci_reset_fn_methods[i].name)) { resets_enabled |= BIT(i);
                                break;
                        }
                }
...
        }

set_reset_methods:
        kfree(options);
        pdev->resets_enabled =  resets_enabled;
        return count;
}

static ssize_t reset_method_show(struct device *dev, struct device_attribute *attr, char *buf)
{
        struct pci_dev *pdev = to_pci_dev(dev); ssize_t len = 0;
        int i;

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                if (pdev->resets_enabled & BIT(i))
                        len += sysfs_emit_at(buf, len, "%s%s", len ? "," : "", pci_reset_fn_methods[i].name); }
        len += sysfs_emit_at(buf, len, len ? "\n" : "");

Quoted 3 lines

        return len;
}

Thank you for the idea.
Actually that would be coming full circle because Alex, Raphael and I tried similar approach earlier while prototyping for v2 but this implementation does look better than what I had at that time.

Thanks,
Amey

#16 lore raw

Re: [PATCH v7 2/8] PCI: Add new array for keeping track of ordering of reset methods

Quoted 162 lines · Amey Narkhede
Quoted 161 lines · Shanker R Donthineni
Quoted 19 lines · Amey Narkhede

I wonder if this would be easier if dev->reset_methods[] contained indices into pci_reset_fn_methods[], highest priority first, with the priority being determined when dev->reset_methods[] is updated. For example:

  const struct pci_reset_fn_method pci_reset_fn_methods[] = {
    { },                                                     # 0
    { &pci_dev_specific_reset, .name = "device_specific" },  # 1
    { &pci_dev_acpi_reset, .name = "acpi" },                 # 2
    { &pcie_reset_flr, .name = "flr" },                      # 3
    { &pci_af_flr, .name = "af_flr" },                       # 4
    { &pci_pm_reset, .name = "pm" },                         # 5
    { &pci_reset_bus_function, .name = "bus" },              # 6
  };

dev->reset_methods[] = [1, 2, 3, 4, 5, 6]
means all reset methods are supported, in the default priority order

Quoted 6 lines

dev->reset_methods[] = [1, 0, 0, 0, 0, 0]
means only pci_dev_specific_reset is supported

dev->reset_methods[] = [3, 5, 0, 0, 0, 0]
means pcie_reset_flr and pci_pm_reset are supported, in that priority order

What about keeping two bitmap fields 'resets_supported' and 'resets_enabled' in pci_dev object and mange it through sysfs and probe helper function. We can avoid two loops multiple paces and takes only 2Bytes of memory to keep track resets.

resets_supported  ---> initialized during pci_dev setup
resets_enabled ---> Exposed to userspace through sysfs by default set to resets_supported

include/linux/pci.h:
------------------------
/* Different types of PCI resets possible, lower number is higher priority */ #define PCI_RESET_METHOD_DEVSPEC     0
#define PCI_RESET_METHOD_ACPI            1
#define PCI_RESET_METHOD_FLR              2
#define PCI_RESET_METHOD_Af_FLR         3
#define PCI_RESET_METHOD_PM               4
#define PCI_RESET_METHOD_BUS             5
#define PCI_RESET_METHOD_MAX            6

struct pci_dev {
    ...
        u8              resets_supported;
        u8              resets_enabled;
};

static inline bool pci_reset_supported(struct pci_dev *dev) {
        return !!(dev->resets_supported);
}

drivers/pci/pci.c:
--------------------
const struct pci_reset_fn_method pci_reset_fn_methods[PCI_RESET_METHOD_MAX] = { [PCI_RESET_METHOD_DEVSPEC] = { &pci_dev_specific_reset,
                                                                   .name = "device_specific" }, [PCI_RESET_METHOD_ACPI] = { &pci_dev_acpi_reset, .name = "acpi" }, [PCI_RESET_METHOD_FLR] = { &pcie_reset_flr, .name = "flr" }, [PCI_RESET_METHOD_Af_FLR] = { &pci_af_flr, .name = "af_flr" }, [PCI_RESET_METHOD_PM] = { &pci_pm_reset, .name = "pm" }, [PCI_RESET_METHOD_BUS] = { &pci_reset_bus_function, .name = "bus" }
};

void pci_init_reset_methods(struct pci_dev *dev)
{
        int i, rc;

        BUILD_BUG_ON(ARRAY_SIZE(pci_reset_fn_methods) != PCI_RESET_METHOD_MAX); might_sleep();

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                rc = pci_reset_fn_methods[i].reset_fn(dev, PCI_RESET_PROBE); if (!rc)
                        dev->resets_supported |= BIT(i);
                else if (rc != -ENOTTY)
                        break;
        }
        dev->resets_enabled = dev->resets_supported;
}

int __pci_reset_function_locked(struct pci_dev *dev) {
        int i, rc = -ENOTTY;

        might_sleep();

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                if (dev->resets_enabled & BIT(i)) {
                        rc = pci_reset_fn_methods[i].reset_fn(dev, 0); if (rc != -ENOTTY)
                                return rc;
                }
        }

        if (rc == -ENOTTY)
                pci_warn(dev, "No reset happened reason %s\n", !!dev->resets_supported ? "disabled by user" : "not supported");

        return rc;
}

drivers/pci/pci-sysfs.c
----------------------------
static ssize_t reset_method_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) {
        struct pci_dev *pdev = to_pci_dev(dev);
        u8 resets_enabled = 0;
...
        if (sysfs_streq(options, "default")) { pdev->resets_enabled = pdev->resets_supported; goto set_reset_methods;
        }

        while ((name = strsep(&options, ",")) != NULL) { if (sysfs_streq(name, "")) continue;
                name = strim(name);

                for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                        if ((pdev->resets_supported & BIT(i)) && sysfs_streq(name, pci_reset_fn_methods[i].name)) { resets_enabled |= BIT(i);
                                break;
                        }
                }
...
        }

set_reset_methods:
        kfree(options);
        pdev->resets_enabled =  resets_enabled;
        return count;
}

static ssize_t reset_method_show(struct device *dev, struct device_attribute *attr, char *buf)
{
        struct pci_dev *pdev = to_pci_dev(dev); ssize_t len = 0;
        int i;

        for (i = 0; i < PCI_RESET_METHOD_MAX; i++) {
                if (pdev->resets_enabled & BIT(i))
                        len += sysfs_emit_at(buf, len, "%s%s", len ? "," : "", pci_reset_fn_methods[i].name); }
        len += sysfs_emit_at(buf, len, len ? "\n" : "");

Quoted 7 lines
Quoted 3 lines

        return len;
}

Thank you for the idea.
Actually that would be coming full circle because Alex, Raphael and I tried similar approach earlier while prototyping for v2 but this implementation does look better than what I had at that time.

I thought part of the point of this series was to allow the user to change the *order* of reset types. I don't think we can control the ordering if we only keep a bit (or even two) per reset type.

#17 lore raw

PATCH v7 3/8 PCI: Remove reset_fn field from pci_dev

In reply to Amey Narkhede

reset_fn field is used to indicate whether the device supports any reset mechanism or not. Remove the use of reset_fn in favor of new reset_methods array which can be used to keep track of all supported reset mechanisms of a device and their ordering.

The octeon driver is incorrectly using
reset_fn field to detect if the device supports FLR or not. Use pcie_reset_flr() to probe whether it supports FLR or not.

---
 drivers/net/ethernet/cavium/liquidio/lio_vf_main.c | 2 +-
 drivers/pci/pci-sysfs.c                            | 2 +-
 drivers/pci/pci.c                                  | 6 +++---
 drivers/pci/probe.c                                | 1 -
 drivers/pci/quirks.c                               | 2 +-
 drivers/pci/remove.c                               | 1 -
 include/linux/pci.h                                | 1 -
 7 files changed, 6 insertions(+), 9 deletions(-)
drivers/net/ethernet/cavium/liquidio/lio_vf_main.c +1 -1
index 516f166ce..336d149ee 100644
@@ -526,7 +526,7 @@ static void octeon_destroy_resources(struct octeon_device *oct)
 			oct->irq_name_storage = NULL;
 		}
 		/* Soft reset the octeon device before exiting */
-		if (oct->pci_dev->reset_fn)
+		if (!pcie_reset_flr(oct->pci_dev, 1))
 			octeon_pci_flr(oct);
 		else
 			cn23xx_vf_ask_pf_to_do_flr(oct);
drivers/pci/pci-sysfs.c +1 -1
index beb8d1f4f..316f70c3e 100644
@@ -1367,7 +1367,7 @@ static umode_t pci_dev_reset_attr_is_visible(struct kobject *kobj,
 {
 	struct pci_dev *pdev = to_pci_dev(kobj_to_dev(kobj));
 
-	if (!pdev->reset_fn)
+	if (!pci_reset_supported(pdev))
 		return 0;
 
 	return a->mode;
drivers/pci/pci.c +3 -3
index 39a9ea8bb..2302aa421 100644
@@ -5232,7 +5232,7 @@ int pci_reset_function(struct pci_dev *dev)
 {
 	int rc;
 
-	if (!dev->reset_fn)
+	if (!pci_reset_supported(dev))
 		return -ENOTTY;
 
 	pci_dev_lock(dev);
@@ -5268,7 +5268,7 @@ int pci_reset_function_locked(struct pci_dev *dev)
 {
 	int rc;
 
-	if (!dev->reset_fn)
+	if (!pci_reset_supported(dev))
 		return -ENOTTY;
 
 	pci_dev_save_and_disable(dev);
@@ -5291,7 +5291,7 @@ int pci_try_reset_function(struct pci_dev *dev)
 {
 	int rc;
 
-	if (!dev->reset_fn)
+	if (!pci_reset_supported(dev))
 		return -ENOTTY;
 
 	if (!pci_dev_trylock(dev))
drivers/pci/probe.c +0 -1
index 8cf532681..90fd4f61f 100644
@@ -2405,7 +2405,6 @@ static void pci_init_capabilities(struct pci_dev *dev)
 
 	pcie_report_downtraining(dev);
 	pci_init_reset_methods(dev);
-	dev->reset_fn = pci_reset_supported(dev);
 }
 
 /*
drivers/pci/quirks.c +1 -1
index f977ba79a..e86cf4a3b 100644
@@ -5589,7 +5589,7 @@ static void quirk_reset_lenovo_thinkpad_p50_nvgpu(struct pci_dev *pdev)
 
 	if (pdev->subsystem_vendor != PCI_VENDOR_ID_LENOVO ||
 	    pdev->subsystem_device != 0x222e ||
-	    !pdev->reset_fn)
+	    !pci_reset_supported(pdev))
 		return;
 
 	if (pci_enable_device_mem(pdev))
drivers/pci/remove.c +0 -1
index dd12c2fcc..4c54c7505 100644
@@ -19,7 +19,6 @@ static void pci_stop_dev(struct pci_dev *dev)
 	pci_pme_active(dev, false);
 
 	if (pci_dev_is_added(dev)) {
-		dev->reset_fn = 0;
 
 		device_release_driver(&dev->dev);
 		pci_proc_detach_device(dev);
include/linux/pci.h +0 -1
index 0955246f8..6e9bc4f9c 100644
@@ -429,7 +429,6 @@ struct pci_dev {
 	unsigned int	state_saved:1;
 	unsigned int	is_physfn:1;
 	unsigned int	is_virtfn:1;
-	unsigned int	reset_fn:1;
 	unsigned int	is_hotplug_bridge:1;
 	unsigned int	shpc_managed:1;		/* SHPC owned by shpchp */
 	unsigned int	is_thunderbolt:1;	/* Thunderbolt controller */
Signature · 1 line
2.31.1
Tested-by on v7 3/8
#18 lore raw

Re: [PATCH v7 3/8] PCI: Remove reset_fn field from pci_dev

Quoted 9 lines · Amey Narkhede

reset_fn field is used to indicate whether the device supports any reset mechanism or not. Remove the use of reset_fn in favor of new reset_methods array which can be used to keep track of all supported reset mechanisms of a device and their ordering.

The octeon driver is incorrectly using
reset_fn field to detect if the device supports FLR or not. Use pcie_reset_flr() to probe whether it supports FLR or not.

Quoted 5 lines
#19 lore raw

PATCH v7 4/8 PCI/sysfs: Allow userspace to query and set device reset mechanism

In reply to Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

---
 Documentation/ABI/testing/sysfs-bus-pci |  16 ++++
 drivers/pci/pci-sysfs.c                 | 118 ++++++++++++++++++++++++
 2 files changed, 134 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +16 -0
index ef00fada2..cf6dbbb3c 100644
@@ -121,6 +121,22 @@ Description:
 		child buses, and re-discover devices removed earlier
 		from this part of the device tree.
 
+What:		/sys/bus/pci/devices/.../reset_method
+Date:		March 2021
+Contact:	Amey Narkhede <[email protected]>
+Description:
+		Some devices allow an individual function to be reset
+		without affecting other functions in the same slot.
+		For devices that have this support, a file named reset_method
+		will be present in sysfs. Reading this file will give names
+		of the device supported reset methods and their ordering.
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device. Writing empty string to this file will disable
+		ability to reset the device and writing "default" will return
+		to the original value.
+
 What:		/sys/bus/pci/devices/.../reset
 Date:		July 2009
 Contact:	Michael S. Tsirkin <[email protected]>
drivers/pci/pci-sysfs.c +118 -0

130 lines not on this page · raw message

Signature · 1 line
2.31.1
Reviewed-by on v7 4/8
#20 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 7 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Quoted 52 lines
---
 Documentation/ABI/testing/sysfs-bus-pci |  16 ++++
 drivers/pci/pci-sysfs.c                 | 118 ++++++++++++++++++++++++
 2 files changed, 134 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +16 -0
index ef00fada2..cf6dbbb3c 100644
@@ -121,6 +121,22 @@ Description:
 		child buses, and re-discover devices removed earlier
 		from this part of the device tree.
 
+What:		/sys/bus/pci/devices/.../reset_method
+Date:		March 2021
+Contact:	Amey Narkhede <[email protected]>
+Description:
+		Some devices allow an individual function to be reset
+		without affecting other functions in the same slot.
+		For devices that have this support, a file named reset_method
+		will be present in sysfs. Reading this file will give names
+		of the device supported reset methods and their ordering.
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device. Writing empty string to this file will disable
+		ability to reset the device and writing "default" will return
+		to the original value.
+
 What:		/sys/bus/pci/devices/.../reset
 Date:		July 2009
 Contact:	Michael S. Tsirkin <[email protected]>
drivers/pci/pci-sysfs.c +12 -0
index 316f70c3e..52def79aa 100644
@@ -1334,6 +1334,123 @@ static const struct attribute_group pci_dev_rom_attr_group = {
 	.is_bin_visible = pci_dev_rom_attr_is_visible,
 };
 
+static ssize_t reset_method_show(struct device *dev,
+				 struct device_attribute *attr,
+				 char *buf)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	ssize_t len = 0;
+	int i, prio;
+
+	for (prio = PCI_RESET_METHODS_NUM; prio; prio--) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (prio == pdev->reset_methods[i]) {
+				len += sysfs_emit_at(buf, len, "%s%s",

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1358

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+						     len ? "," : "",
+						     pci_reset_fn_methods[i].name);
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM)
+			break;
+	}
+

Don't you still need to ensure you add the newline even if there are no reset methods set? If the len is zero why don't we need the newline?

Otherwise looks good.

Quoted 19 lines
drivers/pci/pci-sysfs.c +19 -0
+	if (len)
+		len += sysfs_emit_at(buf, len, "\n");
+
+	return len;
+}
+
+static ssize_t reset_method_store(struct device *dev,
+				  struct device_attribute *attr,
+				  const char *buf, size_t count)
+{
+	u8 reset_methods[PCI_RESET_METHODS_NUM];
+	struct pci_dev *pdev = to_pci_dev(dev);
+	u8 prio = PCI_RESET_METHODS_NUM;
+	char *name, *options;
+	int i;
+
+	if (count >= (PAGE_SIZE - 1))
+		return -EINVAL;
+

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1387

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+	options = kstrndup(buf, count, GFP_KERNEL);
+	if (!options)
+		return -ENOMEM;
+
+	/*
+	 * Initialize reset_method such that 0xff indicates
+	 * supported but not currently enabled reset methods
+	 * as we only use priority values which are within
+	 * the range of PCI_RESET_FN_METHODS array size
+	 */

NIT: missing period in above comment.

Quoted 81 lines
drivers/pci/pci-sysfs.c +67 -0
+	for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+		reset_methods[i] = pdev->reset_methods[i] ? 0xff : 0;
+
+	if (sysfs_streq(options, "")) {
+		pci_warn(pdev, "All device reset methods disabled by user");
+		goto set_reset_methods;
+	}
+
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}
+
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}
+
+	if (reset_methods[0] &&
+	    reset_methods[0] != PCI_RESET_METHODS_NUM)
+		pci_warn(pdev, "Device specific reset disabled/de-prioritized by user");
+
+set_reset_methods:
+	kfree(options);
+	memcpy(pdev->reset_methods, reset_methods, sizeof(reset_methods));
+	return count;
+}
+static DEVICE_ATTR_RW(reset_method);
+
+static struct attribute *pci_dev_reset_method_attrs[] = {
+	&dev_attr_reset_method.attr,
+	NULL,
+};
+
+static umode_t pci_dev_reset_method_attr_is_visible(struct kobject *kobj,
+						    struct attribute *a, int n)
+{
+	struct pci_dev *pdev = to_pci_dev(kobj_to_dev(kobj));
+
+	if (!pci_reset_supported(pdev))
+		return 0;
+
+	return a->mode;
+}
+
+static const struct attribute_group pci_dev_reset_method_attr_group = {
+	.attrs = pci_dev_reset_method_attrs,
+	.is_visible = pci_dev_reset_method_attr_is_visible,
+};
+
 static ssize_t reset_store(struct device *dev, struct device_attribute *attr,
 			   const char *buf, size_t count)
 {
@@ -1491,6 +1608,7 @@ const struct attribute_group *pci_dev_groups[] = {
 	&pci_dev_config_attr_group,
 	&pci_dev_rom_attr_group,
 	&pci_dev_reset_attr_group,
+	&pci_dev_reset_method_attr_group,
 	&pci_dev_vpd_attr_group,
 #ifdef CONFIG_DMI
 	&pci_dev_smbios_attr_group,
Signature · 1 line
2.31.1
#21 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Hi Raphael,

Quoted 25 lines · Raphael Norwitz
Quoted 12 lines
drivers/pci/pci-sysfs.c +12 -0
+static ssize_t reset_method_show(struct device *dev,
+                              struct device_attribute *attr,
+                              char *buf)
+{
+     struct pci_dev *pdev = to_pci_dev(dev);
+     ssize_t len = 0;
+     int i, prio;
+
+     for (prio = PCI_RESET_METHODS_NUM; prio; prio--) {
+             for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+                     if (prio == pdev->reset_methods[i]) {
+                             len += sysfs_emit_at(buf, len, "%s%s",

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1358

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+                                                  len ? "," : "",
+                                                  pci_reset_fn_methods[i].name);
+                             break;
+                     }
+             }
+
+             if (i == PCI_RESET_METHODS_NUM)
+                     break;
+     }
+

Don't you still need to ensure you add the newline even if there are no reset methods set? If the len is zero why don't we need the newline?

Quoted 2 lines

Otherwise looks good.

sysfs entry 'reset_method' will not be visible if there are no reset methods.

#22 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Yes - I got this one wrong.

Nevermind, looks good. Just the punctuation NIT.

Quoted 34 lines · Shanker R Donthineni

Hi Raphael,

Quoted 25 lines · Raphael Norwitz
Quoted 12 lines
drivers/pci/pci-sysfs.c +12 -0
+static ssize_t reset_method_show(struct device *dev,
+                              struct device_attribute *attr,
+                              char *buf)
+{
+     struct pci_dev *pdev = to_pci_dev(dev);
+     ssize_t len = 0;
+     int i, prio;
+
+     for (prio = PCI_RESET_METHODS_NUM; prio; prio--) {
+             for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+                     if (prio == pdev->reset_methods[i]) {
+                             len += sysfs_emit_at(buf, len, "%s%s",

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1358

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+                                                  len ? "," : "",
+                                                  pci_reset_fn_methods[i].name);
+                             break;
+                     }
+             }
+
+             if (i == PCI_RESET_METHODS_NUM)
+                     break;
+     }
+

Don't you still need to ensure you add the newline even if there are no reset methods set? If the len is zero why don't we need the newline?

Quoted 2 lines

Otherwise looks good.

sysfs entry 'reset_method' will not be visible if there are no reset methods.

Tested-by on v7 4/8
#23 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

In reply to Amey Narkhede
Quoted 7 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

#24 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

In reply to Amey Narkhede
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Rewrap to fill 75 columns (also apply to other patches if applicable, e.g., 3/8 looks like it could use it).

2/8 looks like it's missing a blank line between paragraphs.

Quoted 21 lines
---
 Documentation/ABI/testing/sysfs-bus-pci |  16 ++++
 drivers/pci/pci-sysfs.c                 | 118 ++++++++++++++++++++++++
 2 files changed, 134 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +5 -0
index ef00fada2..cf6dbbb3c 100644
@@ -121,6 +121,22 @@ Description:
 		child buses, and re-discover devices removed earlier
 		from this part of the device tree.
 
+What:		/sys/bus/pci/devices/.../reset_method
+Date:		March 2021
+Contact:	Amey Narkhede <[email protected]>
+Description:
+		Some devices allow an individual function to be reset

Documentation/ABI/testing/sysfs-bus-pci Description: · line 138

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +10 -0
+		without affecting other functions in the same slot.
+		For devices that have this support, a file named reset_method
+		will be present in sysfs. Reading this file will give names
+		of the device supported reset methods and their ordering.
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device. Writing empty string to this file will disable
+		ability to reset the device and writing "default" will return
+		to the original value.

Rewrap to fill or add a blank line if "For devices ..." is supposed to start a new paragraph.

My guess is you intend reading to show the *currently enabled* reset methods, not the entire "supported" set? So if a user has disabled one of them, it no longer appears when you read the file?

Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-pci +1 -0
+
 What:		/sys/bus/pci/devices/.../reset
 Date:		July 2009
 Contact:	Michael S. Tsirkin <[email protected]>
drivers/pci/pci-sysfs.c +11 -0
index 316f70c3e..52def79aa 100644
@@ -1334,6 +1334,123 @@ static const struct attribute_group pci_dev_rom_attr_group = {
 	.is_bin_visible = pci_dev_rom_attr_is_visible,
 };
 
+static ssize_t reset_method_show(struct device *dev,
+				 struct device_attribute *attr,
+				 char *buf)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	ssize_t len = 0;
+	int i, prio;
+
+	for (prio = PCI_RESET_METHODS_NUM; prio; prio--) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (prio == pdev->reset_methods[i]) {

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1357

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+				len += sysfs_emit_at(buf, len, "%s%s",
+						     len ? "," : "",
+						     pci_reset_fn_methods[i].name);
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM)
+			break;
+	}

I'm guessing that if you adopt the alternate reset_methods[] encoding, this nested loop becomes a single loop and "prio" goes away?

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (len)
+		len += sysfs_emit_at(buf, len, "\n");
+
+	return len;
+}

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1373

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+
+static ssize_t reset_method_store(struct device *dev,
+				  struct device_attribute *attr,
+				  const char *buf, size_t count)
+{
+	u8 reset_methods[PCI_RESET_METHODS_NUM];
+	struct pci_dev *pdev = to_pci_dev(dev);
+	u8 prio = PCI_RESET_METHODS_NUM;
+	char *name, *options;
+	int i;

Reorder decls with to_pci_dev(dev) first, then in order of use.

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (count >= (PAGE_SIZE - 1))
+		return -EINVAL;
+
+	options = kstrndup(buf, count, GFP_KERNEL);
+	if (!options)

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1389

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+		return -ENOMEM;
+
+	/*
+	 * Initialize reset_method such that 0xff indicates
+	 * supported but not currently enabled reset methods
+	 * as we only use priority values which are within
+	 * the range of PCI_RESET_FN_METHODS array size
+	 */
+	for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+		reset_methods[i] = pdev->reset_methods[i] ? 0xff : 0;

I'm hoping the 0xff trick goes away with the alternate encoding?

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1394

Quoted 4 lines
drivers/pci/pci-sysfs.c +4 -0
+	if (sysfs_streq(options, "")) {
+		pci_warn(pdev, "All device reset methods disabled by user");
+		goto set_reset_methods;
+	}

I think you can get this case out of the way early with no kstrndup(), no goto, etc.

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

Quoted 13 lines
drivers/pci/pci-sysfs.c +13 -0
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {
+				reset_methods[i] = prio--;
+				break;
+			}
+		}

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}
+
+	if (reset_methods[0] &&
+	    reset_methods[0] != PCI_RESET_METHODS_NUM)
+		pci_warn(pdev, "Device specific reset disabled/de-prioritized by user");

Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.

Quoted 9 lines
drivers/pci/pci-sysfs.c +9 -0
+set_reset_methods:
+	kfree(options);
+	memcpy(pdev->reset_methods, reset_methods, sizeof(reset_methods));
+	return count;
+}
+static DEVICE_ATTR_RW(reset_method);
+
+static struct attribute *pci_dev_reset_method_attrs[] = {
+	&dev_attr_reset_method.attr,

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1444

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+	NULL,
+};
+
+static umode_t pci_dev_reset_method_attr_is_visible(struct kobject *kobj,
+						    struct attribute *a, int n)
+{
+	struct pci_dev *pdev = to_pci_dev(kobj_to_dev(kobj));
+
+	if (!pci_reset_supported(pdev))
+		return 0;

I think this _is_visible method is executed only once, at device_add()-time. That means if a device doesn't support any resets at that time, "reset_method" will not be visible, and there will be no way to ever enable a reset method at run-time. I assume that's OK; just double-checking.

Quoted 23 lines
drivers/pci/pci-sysfs.c +10 -0
+
+	return a->mode;
+}
+
+static const struct attribute_group pci_dev_reset_method_attr_group = {
+	.attrs = pci_dev_reset_method_attrs,
+	.is_visible = pci_dev_reset_method_attr_is_visible,
+};
+
 static ssize_t reset_store(struct device *dev, struct device_attribute *attr,
 			   const char *buf, size_t count)
 {
@@ -1491,6 +1608,7 @@ const struct attribute_group *pci_dev_groups[] = {
 	&pci_dev_config_attr_group,
 	&pci_dev_rom_attr_group,
 	&pci_dev_reset_attr_group,
+	&pci_dev_reset_method_attr_group,
 	&pci_dev_vpd_attr_group,
 #ifdef CONFIG_DMI
 	&pci_dev_smbios_attr_group,
Signature · 1 line
2.31.1
#25 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 130 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Rewrap to fill 75 columns (also apply to other patches if applicable, e.g., 3/8 looks like it could use it).

2/8 looks like it's missing a blank line between paragraphs.

Quoted 21 lines
---
 Documentation/ABI/testing/sysfs-bus-pci |  16 ++++
 drivers/pci/pci-sysfs.c                 | 118 ++++++++++++++++++++++++
 2 files changed, 134 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +5 -0
index ef00fada2..cf6dbbb3c 100644
@@ -121,6 +121,22 @@ Description:
 		child buses, and re-discover devices removed earlier
 		from this part of the device tree.
 
+What:		/sys/bus/pci/devices/.../reset_method
+Date:		March 2021
+Contact:	Amey Narkhede <[email protected]>
+Description:
+		Some devices allow an individual function to be reset

Documentation/ABI/testing/sysfs-bus-pci Description: · line 138

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +10 -0
+		without affecting other functions in the same slot.
+		For devices that have this support, a file named reset_method
+		will be present in sysfs. Reading this file will give names
+		of the device supported reset methods and their ordering.
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device. Writing empty string to this file will disable
+		ability to reset the device and writing "default" will return
+		to the original value.

Rewrap to fill or add a blank line if "For devices ..." is supposed to start a new paragraph.

My guess is you intend reading to show the *currently enabled* reset methods, not the entire "supported" set? So if a user has disabled one of them, it no longer appears when you read the file?

Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-pci +1 -0
+
 What:		/sys/bus/pci/devices/.../reset
 Date:		July 2009
 Contact:	Michael S. Tsirkin <[email protected]>
drivers/pci/pci-sysfs.c +11 -0
index 316f70c3e..52def79aa 100644
@@ -1334,6 +1334,123 @@ static const struct attribute_group pci_dev_rom_attr_group = {
 	.is_bin_visible = pci_dev_rom_attr_is_visible,
 };
 
+static ssize_t reset_method_show(struct device *dev,
+				 struct device_attribute *attr,
+				 char *buf)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	ssize_t len = 0;
+	int i, prio;
+
+	for (prio = PCI_RESET_METHODS_NUM; prio; prio--) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (prio == pdev->reset_methods[i]) {

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1357

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+				len += sysfs_emit_at(buf, len, "%s%s",
+						     len ? "," : "",
+						     pci_reset_fn_methods[i].name);
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM)
+			break;
+	}

I'm guessing that if you adopt the alternate reset_methods[] encoding, this nested loop becomes a single loop and "prio" goes away?

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (len)
+		len += sysfs_emit_at(buf, len, "\n");
+
+	return len;
+}

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1373

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+
+static ssize_t reset_method_store(struct device *dev,
+				  struct device_attribute *attr,
+				  const char *buf, size_t count)
+{
+	u8 reset_methods[PCI_RESET_METHODS_NUM];
+	struct pci_dev *pdev = to_pci_dev(dev);
+	u8 prio = PCI_RESET_METHODS_NUM;
+	char *name, *options;
+	int i;

Reorder decls with to_pci_dev(dev) first, then in order of use.

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (count >= (PAGE_SIZE - 1))
+		return -EINVAL;
+
+	options = kstrndup(buf, count, GFP_KERNEL);
+	if (!options)

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1389

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+		return -ENOMEM;
+
+	/*
+	 * Initialize reset_method such that 0xff indicates
+	 * supported but not currently enabled reset methods
+	 * as we only use priority values which are within
+	 * the range of PCI_RESET_FN_METHODS array size
+	 */
+	for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+		reset_methods[i] = pdev->reset_methods[i] ? 0xff : 0;

I'm hoping the 0xff trick goes away with the alternate encoding?

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1394

Quoted 4 lines
drivers/pci/pci-sysfs.c +4 -0
+	if (sysfs_streq(options, "")) {
+		pci_warn(pdev, "All device reset methods disabled by user");
+		goto set_reset_methods;
+	}

I think you can get this case out of the way early with no kstrndup(), no goto, etc.

Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,

# cat reset_methods
flr,bus 			# dev->reset_methods = [3, 5, 0, ...]
# echo bus > reset_methods 	# dev->reset_methods = [5, 0, 0, ...]
# cat reset_methods
bus

Now if an user wants to enable flr

# echo flr > reset_methods 	# dev->reset_methods = [3, 0, 0, ...]
OR
# echo bus,flr > reset_methods 	# dev->reset_methods = [5, 3, 0, ...]

either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.

Quoted 19 lines
Quoted 19 lines
drivers/pci/pci-sysfs.c +19 -0
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}
Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424

Quoted 4 lines
drivers/pci/pci-sysfs.c +4 -0
+
+	if (reset_methods[0] &&
+	    reset_methods[0] != PCI_RESET_METHODS_NUM)
+		pci_warn(pdev, "Device specific reset disabled/de-prioritized by user");

Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.

I think generally presence of device specific reset method means other methods are potentially broken. Is it okay to skip this?

Quoted 17 lines
Quoted 17 lines
drivers/pci/pci-sysfs.c +17 -0
+set_reset_methods:
+	kfree(options);
+	memcpy(pdev->reset_methods, reset_methods, sizeof(reset_methods));
+	return count;
+}
+static DEVICE_ATTR_RW(reset_method);
+
+static struct attribute *pci_dev_reset_method_attrs[] = {
+	&dev_attr_reset_method.attr,
+	NULL,
+};
+
+static umode_t pci_dev_reset_method_attr_is_visible(struct kobject *kobj,
+						    struct attribute *a, int n)
+{
+	struct pci_dev *pdev = to_pci_dev(kobj_to_dev(kobj));
+
Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1444

Quoted 2 lines
drivers/pci/pci-sysfs.c +2 -0
+	if (!pci_reset_supported(pdev))
+		return 0;

I think this _is_visible method is executed only once, at device_add()-time. That means if a device doesn't support any resets at that time, "reset_method" will not be visible, and there will be no way to ever enable a reset method at run-time. I assume that's OK; just double-checking.

Correct. Its similar to exisitng reset_fn bitfield which is removed in this patch series.
[...]

Thanks,
Amey

#26 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,

# cat reset_methods
flr,bus 			# dev->reset_methods = [3, 5, 0, ...]
# echo bus > reset_methods 	# dev->reset_methods = [5, 0, 0, ...]
# cat reset_methods
bus
Quoted 8 lines

Now if an user wants to enable flr

# echo flr > reset_methods 	# dev->reset_methods = [3, 0, 0, ...]
OR
# echo bus,flr > reset_methods 	# dev->reset_methods = [5, 3, 0, ...]

either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.

Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.

If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with -ENOTTY? Is that a problem?

Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
drivers/pci/pci-sysfs.c +19 -0
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}
Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424

Quoted 4 lines
drivers/pci/pci-sysfs.c +4 -0
+
+	if (reset_methods[0] &&
+	    reset_methods[0] != PCI_RESET_METHODS_NUM)
+		pci_warn(pdev, "Device specific reset disabled/de-prioritized by user");

Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.

Quoted 2 lines

I think generally presence of device specific reset method means other methods are potentially broken. Is it okay to skip this?

We might want a warning at reset-time if all the methods failed, because that means we may leak state between users. Maybe we also want one here, if *all* reset methods are disabled. I don't really like special treatment of device-specific methods here because it depends on the assumption that "device-specific means all other resets are broken." That's hard to maintain.

Bjorn

#27 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,

# cat reset_methods
flr,bus 			# dev->reset_methods = [3, 5, 0, ...]
# echo bus > reset_methods 	# dev->reset_methods = [5, 0, 0, ...]
# cat reset_methods
bus
Quoted 8 lines

Now if an user wants to enable flr

# echo flr > reset_methods 	# dev->reset_methods = [3, 0, 0, ...]
OR
# echo bus,flr > reset_methods 	# dev->reset_methods = [5, 3, 0, ...]

either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.

Quoted 8 lines

Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.

If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with -ENOTTY? Is that a problem?

I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.

Alternate way of doing this is using 2 bitmaps as outlined here by Shanker https://marc.info/?l=linux-kernel&m=162428773101702&w=2

Quoted 31 lines
Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
drivers/pci/pci-sysfs.c +19 -0
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}
Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424

Quoted 4 lines
drivers/pci/pci-sysfs.c +4 -0
+
+	if (reset_methods[0] &&
+	    reset_methods[0] != PCI_RESET_METHODS_NUM)
+		pci_warn(pdev, "Device specific reset disabled/de-prioritized by user");

Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.

Quoted 2 lines

I think generally presence of device specific reset method means other methods are potentially broken. Is it okay to skip this?

Quoted 8 lines

We might want a warning at reset-time if all the methods failed, because that means we may leak state between users. Maybe we also want one here, if *all* reset methods are disabled. I don't really like special treatment of device-specific methods here because it depends on the assumption that "device-specific means all other resets are broken." That's hard to maintain.

Bjorn

Makes sense. I'll update this.

Thanks,
Amey

#28 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,

# cat reset_methods
flr,bus 			# dev->reset_methods = [3, 5, 0, ...]
# echo bus > reset_methods 	# dev->reset_methods = [5, 0, 0, ...]
# cat reset_methods
bus
Quoted 8 lines

Now if an user wants to enable flr

# echo flr > reset_methods 	# dev->reset_methods = [3, 0, 0, ...]
OR
# echo bus,flr > reset_methods 	# dev->reset_methods = [5, 3, 0, ...]

either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.

Quoted 8 lines

Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.

If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with -ENOTTY? Is that a problem?

Quoted 8 lines

I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.

I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "reset_method" sets the "reset methods and their ordering."

It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "reset_method" to enable reset X, can we just probe reset X to see if it's supported?

Bjorn

#29 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 58 lines · Bjorn Helgaas
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1400

Quoted 5 lines
drivers/pci/pci-sysfs.c +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,

# cat reset_methods
flr,bus 			# dev->reset_methods = [3, 5, 0, ...]
# echo bus > reset_methods 	# dev->reset_methods = [5, 0, 0, ...]
# cat reset_methods
bus
Quoted 8 lines

Now if an user wants to enable flr

# echo flr > reset_methods 	# dev->reset_methods = [3, 0, 0, ...]
OR
# echo bus,flr > reset_methods 	# dev->reset_methods = [5, 3, 0, ...]

either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.

Quoted 8 lines

Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.

If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with -ENOTTY? Is that a problem?

Quoted 8 lines

I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.

I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "reset_method" sets the "reset methods and their ordering."

Quoted 5 lines

It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "reset_method" to enable reset X, can we just probe reset X to see if it's supported?

Bjorn

Although final result is same whether user writes a supported reset method or their ordering that is,

# echo bus > reset_methods
and
# echo bus,flr > reset_methods

are the same but in the first version, users don't have to explicitly set the ordering if they just want to perform bus reset. Current implementation allows the flexibility for switching between first and second option.

Does this address your doubt?

Thanks,
Amey

#30 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 65 lines · Amey Narkhede
Quoted 53 lines · Bjorn Helgaas
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,

# cat reset_methods
flr,bus 			# dev->reset_methods = [3, 5, 0, ...]
# echo bus > reset_methods 	# dev->reset_methods = [5, 0, 0, ...]
# cat reset_methods
bus
Quoted 8 lines

Now if an user wants to enable flr

# echo flr > reset_methods 	# dev->reset_methods = [3, 0, 0, ...]
OR
# echo bus,flr > reset_methods 	# dev->reset_methods = [5, 3, 0, ...]

either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.

Quoted 8 lines

Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.

If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with -ENOTTY? Is that a problem?

Quoted 8 lines

I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.

Quoted 8 lines

I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "reset_method" sets the "reset methods and their ordering."

It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "reset_method" to enable reset X, can we just probe reset X to see if it's supported?

Although final result is same whether user writes a supported reset method or their ordering that is,

Quoted 8 lines
# echo bus > reset_methods
and
# echo bus,flr > reset_methods

are the same but in the first version, users don't have to explicitly set the ordering if they just want to perform bus reset. Current implementation allows the flexibility for switching between first and second option.

Sorry, I can't quite make sense of the above.

Your doc implies the following are different:

  # echo bus,flr > reset_methods
  # echo flr,bus > reset_methods

Are they? If you don't need to provide control over the order of trying resets, this can all be simplified quite a bit.

Bjorn

#31 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 79 lines · Bjorn Helgaas
Quoted 65 lines · Amey Narkhede
Quoted 53 lines · Bjorn Helgaas
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,

# cat reset_methods
flr,bus 			# dev->reset_methods = [3, 5, 0, ...]
# echo bus > reset_methods 	# dev->reset_methods = [5, 0, 0, ...]
# cat reset_methods
bus
Quoted 8 lines

Now if an user wants to enable flr

# echo flr > reset_methods 	# dev->reset_methods = [3, 0, 0, ...]
OR
# echo bus,flr > reset_methods 	# dev->reset_methods = [5, 3, 0, ...]

either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.

Quoted 8 lines

Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.

If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with -ENOTTY? Is that a problem?

Quoted 8 lines

I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.

Quoted 8 lines

I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "reset_method" sets the "reset methods and their ordering."

It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "reset_method" to enable reset X, can we just probe reset X to see if it's supported?

Although final result is same whether user writes a supported reset method or their ordering that is,

Quoted 8 lines
# echo bus > reset_methods
and
# echo bus,flr > reset_methods

are the same but in the first version, users don't have to explicitly set the ordering if they just want to perform bus reset. Current implementation allows the flexibility for switching between first and second option.

Sorry, I can't quite make sense of the above.

Your doc implies the following are different:

Quoted 7 lines
  # echo bus,flr > reset_methods
  # echo flr,bus > reset_methods

Are they? If you don't need to provide control over the order of trying resets, this can all be simplified quite a bit.

Bjorn

The v1 of the patch series allowed only single reset method to be written instead of ordering of all supported reset methods. With your example, current implementation allows both writing single value and list of supported reset methods.

# echo bus > reset_methods
and
# echo bus,flr > reset_methods

OR

# echo flr > reset_methods
and
echo flr,bus > reset_methods

Its more of a preference than a functional point. Ultimately the __pci_reset_function_locked() function will only perform 'bus' reset in this example because we make sure 'reset_methods' file only contains valid device supported reset methods all the time so __pci_reset_function_locked() won't go into -ENOTTY case but with new encoding theres no way to make sure reset_methodssysfs attribute will contain valid supported reset methods all the time because of the reset methods can be masked implicitly if user uses first option of writing only single value.

My point is current implementation allows more flexibility for the user.

Thanks,
Amey

#32 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 96 lines · Amey Narkhede
Quoted 79 lines · Bjorn Helgaas
Quoted 65 lines · Amey Narkhede
Quoted 53 lines · Bjorn Helgaas
Quoted 43 lines · Amey Narkhede
Quoted 34 lines · Bjorn Helgaas
Quoted 24 lines · Amey Narkhede
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+	if (sysfs_streq(options, "default")) {
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++)
+			reset_methods[i] = reset_methods[i] ? prio-- : 0;
+		goto set_reset_methods;
+	}

If you use pci_init_reset_methods() here, you can also get this case out of the way early.

The problem with alternate encoding is we won't be able to know if one of the reset methods was disabled previously. For example,

# cat reset_methods
flr,bus 			# dev->reset_methods = [3, 5, 0, ...]
# echo bus > reset_methods 	# dev->reset_methods = [5, 0, 0, ...]
# cat reset_methods
bus
Quoted 8 lines

Now if an user wants to enable flr

# echo flr > reset_methods 	# dev->reset_methods = [3, 0, 0, ...]
OR
# echo bus,flr > reset_methods 	# dev->reset_methods = [5, 3, 0, ...]

either they need to write "default" first then flr or we will need to reprobe reset methods each time when user writes to reset_method attribute.

Quoted 8 lines

Not sure I completely understand the problem here. I think relying on previous state that is invisible to the user is a little problematic because it's hard for the user to predict what will happen.

If the user enables a method that was previously "disabled" because the probe failed, won't the reset method itself just fail with -ENOTTY? Is that a problem?

Quoted 8 lines

I think I didn't explain this correctly. With current implementation its not necessary to explicitly set *order of availabe* reset methods. User can directly write a single supported reset method only and then perform the reset. Side effect of that is other methods are disabled if user writes single or less than available number of supported reset method. Current implementation is able to handle this case but with new encoding we'll need to reprobe reset methods everytime because we have no way of distingushing supported and currently enabled reset method.

Quoted 8 lines

I'm confused. I thought the point of the nested loops to find the highest priority enabled reset method was to allow the user to control the order. The sysfs doc says writing "reset_method" sets the "reset methods and their ordering."

It seems complicated to track "supported" and "enabled" separately, and I don't know what the benefit is. If we write "reset_method" to enable reset X, can we just probe reset X to see if it's supported?

Although final result is same whether user writes a supported reset method or their ordering that is,

Quoted 8 lines
# echo bus > reset_methods
and
# echo bus,flr > reset_methods

are the same but in the first version, users don't have to explicitly set the ordering if they just want to perform bus reset. Current implementation allows the flexibility for switching between first and second option.

Sorry, I can't quite make sense of the above.

Your doc implies the following are different:

Quoted 7 lines
  # echo bus,flr > reset_methods
  # echo flr,bus > reset_methods

Are they? If you don't need to provide control over the order of trying resets, this can all be simplified quite a bit.

Bjorn

The v1 of the patch series allowed only single reset method to be written instead of ordering of all supported reset methods. With your example, current implementation allows both writing single value and list of supported reset methods.

# echo bus > reset_methods
and
# echo bus,flr > reset_methods
Quoted 6 lines

OR

# echo flr > reset_methods
and
echo flr,bus > reset_methods
# echo flr,bus and echo bus,flr are different.
Quoted 5 lines

Its more of a preference than a functional point. Ultimately the __pci_reset_function_locked() function will only perform 'bus' reset in this example because we make sure 'reset_methods' file only contains valid device supported reset methods all the time so __pci_reset_function_locked() won't go into -ENOTTY case but with new

Oops I'm wrong here. __pci_reset_function_locked() can return -ENOTTY and follow through if a reset fails.

Rest of the point should hold.

Quoted 9 lines

encoding theres no way to make sure reset_methodssysfs attribute will contain valid supported reset methods all the time because of the reset methods can be masked implicitly if user uses first option of writing only single value.

My point is current implementation allows more flexibility for the user.

Thanks,
Amey

#33 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

In reply to Bjorn Helgaas
Quoted 34 lines · On Mon, 21 Jun 2021 08:01:35 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 30 lines · Amey Narkhede
Quoted 20 lines · Bjorn Helgaas
Quoted 19 lines · Amey Narkhede
drivers/pci/pci-sysfs.c +19 -0
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}
Quoted 9 lines

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1424

Quoted 4 lines
drivers/pci/pci-sysfs.c +4 -0
+
+	if (reset_methods[0] &&
+	    reset_methods[0] != PCI_RESET_METHODS_NUM)
+		pci_warn(pdev, "Device specific reset disabled/de-prioritized by user");  

Is there a specific reason for this warning? Is it just telling the user that he might have shot himself in the foot? Not sure that's necessary.

Quoted 2 lines

I think generally presence of device specific reset method means other methods are potentially broken. Is it okay to skip this?

Quoted 6 lines

We might want a warning at reset-time if all the methods failed, because that means we may leak state between users. Maybe we also want one here, if *all* reset methods are disabled. I don't really like special treatment of device-specific methods here because it depends on the assumption that "device-specific means all other resets are broken." That's hard to maintain.

I'd say the device specific reset is special. The device itself can support a number of resets and they're theoretically all equivalent, it's a policy decision which to use. But the device specific reset is a software provided reset. Someone has specifically gone to the trouble to create a reset mechanism that is in some way better than the other methods. Not using that one by default sure feels like something worthy of leaving a breadcrumb in dmesg for debugging. Thanks,

Alex

#34 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

In reply to Amey Narkhede
Quoted 18 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

drivers/pci/pci-sysfs.c +13 -0
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device.
 
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}

Asking again since we didn't get this clarified before. The above tells me that "reset_methods" allows the user to control the *order* in which we try reset methods.

Consider the following two uses:

(1) # echo bus,flr > reset_methods

(2) # echo flr,bus > reset_methods

Do these have the same effect or not?

If "reset_methods" allows control over the order, I expect them to be
different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.

#35 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 34 lines · Bjorn Helgaas
Quoted 18 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

drivers/pci/pci-sysfs.c +13 -0
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device.
 
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}

Asking again since we didn't get this clarified before. The above tells me that "reset_methods" allows the user to control the *order* in which we try reset methods.

Quoted 8 lines

Consider the following two uses:

(1) # echo bus,flr > reset_methods

(2) # echo flr,bus > reset_methods

Do these have the same effect or not?

They have different effect.

Quoted 3 lines

If "reset_methods" allows control over the order, I expect them to be
different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.

Exactly you are right.

Now the point I was presenting was with new encoding we have to write list of *all of the supported reset methods* in order for example, in above example flr,bus or bus,flr. We can't just write 'flr' or 'bus' then switch back to writing flr,bus/bus,flr(these have different effect as mentioned earlier).

Basically with new encoding an user can't write subset of reset methods they have to write list of *all* supported methods everytime.

Thanks,
Amey

#36 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 37 lines · Amey Narkhede
Quoted 36 lines · Bjorn Helgaas
Quoted 18 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

drivers/pci/pci-sysfs.c +13 -0
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device.
 
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}

Asking again since we didn't get this clarified before. The above tells me that "reset_methods" allows the user to control the *order* in which we try reset methods.

Consider the following two uses:

Quoted 7 lines
Quoted 6 lines

(1) # echo bus,flr > reset_methods

(2) # echo flr,bus > reset_methods

Do these have the same effect or not?

They have different effect.

I asked about this because Shanker's idea [1] of using two bitmaps only keeps track of which resets are *enabled*. It does not keep track of the *ordering*. Since you want to control the ordering, I think we need more state than just the supported/enabled bitmaps.

Quoted 5 lines
Quoted 4 lines

If "reset_methods" allows control over the order, I expect them to be different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.

Quoted 7 lines

Exactly you are right.

Now the point I was presenting was with new encoding we have to write list of *all of the supported reset methods* in order for example, in above example flr,bus or bus,flr. We can't just write 'flr' or 'bus' then switch back to writing flr,bus/bus,flr (these have different effect as mentioned earlier).

It sounds like you're saying this sequence can't work:

  # echo flr > reset_methods
  # echo bus,flr > reset_methods

But I'm afraid you'll have to walk me through the reasons why this can't be made to work.

Quoted 3 lines

Basically with new encoding an user can't write subset of reset methods they have to write list of *all* supported methods everytime.

Why does the user have to write all supported methods? Is that to preserve the fact that "cat reset_methods" always shows all the supported methods so the user knows what's available?

I'm wondering why we can't do something like this (pidgin code):

  if (option == "default") {
    pci_init_reset_methods(dev);
    return;
  }

  n = 0;
  foreach method in option {
    i = lookup_reset_method(method);
    if (pci_reset_methods[i].reset_fn(dev, PROBE) == 0)
      dev->reset_methods[n++] = i;           # method i supported
  }
  dev->reset_methods[n++] = 0;               # end of supported methods

If we did something like the above, the user could always find the list of all methods supported by a device by doing this:

  # echo default > reset_methods
  # cat reset_methods

Yes, this does call the "probe" methods several times. I don't think that's necessarily a problem.

Bjorn

[1] https://lore.kernel.org/r/[email protected]

#37 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 19 lines · Bjorn Helgaas

Why does the user have to write all supported methods? Is that to preserve the fact that "cat reset_methods" always shows all the supported methods so the user knows what's available?

I'm wondering why we can't do something like this (pidgin code):

  if (option == "default") {
    pci_init_reset_methods(dev);
    return;
  }

  n = 0;
  foreach method in option {
    i = lookup_reset_method(method);
    if (pci_reset_methods[i].reset_fn(dev, PROBE) == 0)
      dev->reset_methods[n++] = i;           # method i supported
  }
  dev->reset_methods[n++] = 0;               # end of supported methods
Quoted 8 lines

If we did something like the above, the user could always find the list of all methods supported by a device by doing this:

  # echo default > reset_methods
  # cat reset_methods

Yes, this does call the "probe" methods several times. I don't think that's necessarily a problem.

Agree, I don't think admin/user will change reset methods frequently and no side effects or performance impacts on probing multiple times.

We should support enabling partially ordered reset methods and restore default methods either by re-probing resets or remember supported resets in pci_dev.

#38 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

In reply to Bjorn Helgaas
Quoted 64 lines · Bjorn Helgaas
Quoted 37 lines · Amey Narkhede
Quoted 36 lines · Bjorn Helgaas
Quoted 18 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

drivers/pci/pci-sysfs.c +13 -0
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device.
 
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}

Asking again since we didn't get this clarified before. The above tells me that "reset_methods" allows the user to control the *order* in which we try reset methods.

Consider the following two uses:

Quoted 7 lines
Quoted 6 lines

(1) # echo bus,flr > reset_methods

(2) # echo flr,bus > reset_methods

Do these have the same effect or not?

They have different effect.

I asked about this because Shanker's idea [1] of using two bitmaps only keeps track of which resets are *enabled*. It does not keep track of the *ordering*. Since you want to control the ordering, I think we need more state than just the supported/enabled bitmaps.

Quoted 5 lines
Quoted 4 lines

If "reset_methods" allows control over the order, I expect them to be different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.

Quoted 7 lines

Exactly you are right.

Now the point I was presenting was with new encoding we have to write list of *all of the supported reset methods* in order for example, in above example flr,bus or bus,flr. We can't just write 'flr' or 'bus' then switch back to writing flr,bus/bus,flr (these have different effect as mentioned earlier).

Quoted 3 lines

It sounds like you're saying this sequence can't work:

  # echo flr > reset_methods

# dev->reset_methods = [3, 0, 0, ..]

Quoted 1 line
  # echo bus,flr > reset_methods
# to get dev->reset_methods = [6, 3, 0, ...]
we'll need to probe reset methods here.
Quoted 3 lines

But I'm afraid you'll have to walk me through the reasons why this can't be made to work.

I wrote incomplete description. It can work but we'll need to probe everytime which involves reading different capabilities(PCI_CAP_ID_AF, PCI_PM_CTRL etc) from device. With current encoding we just have to probe at the begining.

Quoted 11 lines
Quoted 4 lines

Basically with new encoding an user can't write subset of reset methods they have to write list of *all* supported methods everytime.

Why does the user have to write all supported methods? Is that to preserve the fact that "cat reset_methods" always shows all the supported methods so the user knows what's available?

I'm wondering why we can't do something like this (pidgin code):

Quoted 9 lines
  if (option == "default") {
    pci_init_reset_methods(dev);
    return;
  }

  n = 0;
  foreach method in option {
    i = lookup_reset_method(method);
    if (pci_reset_methods[i].reset_fn(dev, PROBE) == 0)

Repeatedly calling probe might have some impact as it involves reading device registers as explained earlier.

Quoted 10 lines
      dev->reset_methods[n++] = i;           # method i supported

}
dev->reset_methods[n++] = 0; # end of supported methods

If we did something like the above, the user could always find the list of all methods supported by a device by doing this:

  # echo default > reset_methods
  # cat reset_methods

This is one solution for current problem with new encoding.

Quoted 2 lines

Yes, this does call the "probe" methods several times. I don't think that's necessarily a problem.

I thought this would be a problem because of your earlier suggestion of caching flr capability to avoid probing multiple times. In this case we'll need to read different device registers multiple times. With current encoding we don't have to do that multiple times.

Thanks,
Amey

Quoted 4 lines

Bjorn

[1] https://lore.kernel.org/r/[email protected]

#39 lore raw

Re: [PATCH v7 4/8] PCI/sysfs: Allow userspace to query and set device reset mechanism

Quoted 119 lines · Amey Narkhede
Quoted 64 lines · Bjorn Helgaas
Quoted 37 lines · Amey Narkhede
Quoted 36 lines · Bjorn Helgaas
Quoted 18 lines · Amey Narkhede

Add reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.

drivers/pci/pci-sysfs.c +13 -0
+		Writing the name or comma separated list of names of any of
+		the device supported reset methods to this file will set the
+		reset methods and their ordering to be used when resetting
+		the device.
 
+	while ((name = strsep(&options, ",")) != NULL) {
+		if (sysfs_streq(name, ""))
+			continue;
+
+		name = strim(name);
+
+		for (i = 0; i < PCI_RESET_METHODS_NUM; i++) {
+			if (reset_methods[i] &&
+			    sysfs_streq(name, pci_reset_fn_methods[i].name)) {

drivers/pci/pci-sysfs.c static const struct attribute_group pci_dev_rom_attr_group = { · line 1420

Quoted 10 lines
drivers/pci/pci-sysfs.c +10 -0
+				reset_methods[i] = prio--;
+				break;
+			}
+		}
+
+		if (i == PCI_RESET_METHODS_NUM) {
+			kfree(options);
+			return -EINVAL;
+		}
+	}

Asking again since we didn't get this clarified before. The above tells me that "reset_methods" allows the user to control the *order* in which we try reset methods.

Consider the following two uses:

Quoted 7 lines
Quoted 6 lines

(1) # echo bus,flr > reset_methods

(2) # echo flr,bus > reset_methods

Do these have the same effect or not?

They have different effect.

I asked about this because Shanker's idea [1] of using two bitmaps only keeps track of which resets are *enabled*. It does not keep track of the *ordering*. Since you want to control the ordering, I think we need more state than just the supported/enabled bitmaps.

Quoted 5 lines
Quoted 4 lines

If "reset_methods" allows control over the order, I expect them to be different: (1) would try a bus reset and, if the bus reset failed, an FLR, while (2) would try an FLR and, if the FLR failed, a bus reset.

Quoted 7 lines

Exactly you are right.

Now the point I was presenting was with new encoding we have to write list of *all of the supported reset methods* in order for example, in above example flr,bus or bus,flr. We can't just write 'flr' or 'bus' then switch back to writing flr,bus/bus,flr (these have different effect as mentioned earlier).

Quoted 3 lines

It sounds like you're saying this sequence can't work:

  # echo flr > reset_methods

# dev->reset_methods = [3, 0, 0, ..]

Quoted 1 line
  # echo bus,flr > reset_methods
# to get dev->reset_methods = [6, 3, 0, ...]
we'll need to probe reset methods here.
Quoted 2 lines

But I'm afraid you'll have to walk me through the reasons why this can't be made to work.

I wrote incomplete description. It can work but we'll need to probe everytime which involves reading different capabilities(PCI_CAP_ID_AF, PCI_PM_CTRL etc) from device. With current encoding we just have to probe at the begining.

Quoted 10 lines
Quoted 3 lines

Basically with new encoding an user can't write subset of reset methods they have to write list of *all* supported methods everytime.

Why does the user have to write all supported methods? Is that to preserve the fact that "cat reset_methods" always shows all the supported methods so the user knows what's available?

I'm wondering why we can't do something like this (pidgin code):

Quoted 9 lines
  if (option == "default") {
    pci_init_reset_methods(dev);
    return;
  }

  n = 0;
  foreach method in option {
    i = lookup_reset_method(method);
    if (pci_reset_methods[i].reset_fn(dev, PROBE) == 0)

Repeatedly calling probe might have some impact as it involves reading device registers as explained earlier.

Quoted 9 lines
      dev->reset_methods[n++] = i;           # method i supported

}
dev->reset_methods[n++] = 0; # end of supported methods

If we did something like the above, the user could always find the list of all methods supported by a device by doing this:

  # echo default > reset_methods
  # cat reset_methods

This is one solution for current problem with new encoding.

Quoted 7 lines
Quoted 2 lines

Yes, this does call the "probe" methods several times. I don't think that's necessarily a problem.

I thought this would be a problem because of your earlier suggestion of caching flr capability to avoid probing multiple times. In this case we'll need to read different device registers multiple times. With current encoding we don't have to do that multiple times.

I don't think it's a problem to run "probe" methods when we're setting the enabled reset methods (either at enumeration-time or when we write to "reset_methods"). These are low-frequency events and I don't think there's any performance issue.

I don't think we should have to run "probe" methods every time we call pci_reset_function().

I suggested a dev->has_flr bit for two reasons:

1) It avoids reading PCI_EXP_DEVCAP every time a driver calls pcie_reset_flr(), and

2) It gives a nice opportunity for quirks to disable FLR for devices where it's broken.

Quoted 1 line
Quoted 1 line

[1] https://lore.kernel.org/r/[email protected]

#40 lore raw

PATCH v7 5/8 PCI: Setup ACPI_COMPANION early

In reply to Amey Narkhede

From: Shanker Donthineni <[email protected]>

Currently, the ACPI_COMPANION is not available until device_add(). The software features which have dependency on ACPI fwnode properties and needs to be handled before device_add() will not work. One use case, software has to check the existence of _RST method to support ACPI based reset mechanism.

This patch adds a new function pci_set_acpi_fwnode() for setting the ACPI_COMPANION, same code which is available in acpi_pci_bridge_d3().

Call pci_set_acpi_fwnode() from pci_scan_device() to fix the issue.

---
 drivers/pci/pci-acpi.c | 12 ++++++++----
 drivers/pci/pci.h      |  2 ++
 drivers/pci/probe.c    |  2 ++
 3 files changed, 12 insertions(+), 4 deletions(-)
drivers/pci/pci-acpi.c +8 -4
index 36bc23e21..eaddbf701 100644
@@ -934,6 +934,13 @@ static pci_power_t acpi_pci_choose_state(struct pci_dev *pdev)
 
 static struct acpi_device *acpi_pci_find_companion(struct device *dev);
 
+void pci_set_acpi_fwnode(struct pci_dev *dev)
+{
+	if (!ACPI_COMPANION(&dev->dev) && !pci_dev_is_added(dev))
+		ACPI_COMPANION_SET(&dev->dev,
+				   acpi_pci_find_companion(&dev->dev));
+}
+
 static bool acpi_pci_bridge_d3(struct pci_dev *dev)
 {
 	const struct fwnode_handle *fwnode;
@@ -945,11 +952,8 @@ static bool acpi_pci_bridge_d3(struct pci_dev *dev)
 		return false;
 
 	/* Assume D3 support if the bridge is power-manageable by ACPI. */
+	pci_set_acpi_fwnode(dev);
 	adev = ACPI_COMPANION(&dev->dev);
-	if (!adev && !pci_dev_is_added(dev)) {
-		adev = acpi_pci_find_companion(&dev->dev);
-		ACPI_COMPANION_SET(&dev->dev, adev);
-	}
 
 	if (adev && acpi_device_power_manageable(adev))
 		return true;
drivers/pci/pci.h +2 -0
index 13ec6bd6f..d22da6d3c 100644
@@ -703,7 +703,9 @@ static inline int pci_aer_raw_clear_status(struct pci_dev *dev) { return -EINVAL
 #ifdef CONFIG_ACPI
 int pci_acpi_program_hp_params(struct pci_dev *dev);
 extern const struct attribute_group pci_dev_acpi_attr_group;
+void pci_set_acpi_fwnode(struct pci_dev *dev);
 #else
+static inline void pci_set_acpi_fwnode(struct pci_dev *dev) {}
 static inline int pci_acpi_program_hp_params(struct pci_dev *dev)
 {
 	return -ENODEV;
drivers/pci/probe.c +2 -0
index 90fd4f61f..dfefa5ed0 100644
@@ -2359,6 +2359,8 @@ static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn)
 		return NULL;
 	}
 
+	pci_set_acpi_fwnode(dev);
+
 	return dev;
 }
 
Signature · 1 line
2.31.1
#41 lore raw

PATCH v7 6/8 PCI: Add support for ACPI _RST reset method

In reply to Amey Narkhede

From: Shanker Donthineni <[email protected]>

The _RST is a standard method specified in the ACPI specification. It provides a function level reset when it is described in the acpi_device context associated with PCI-device. Implement a new reset function pci_dev_acpi_reset() for probing RST method and execute if it is defined in the firmware.

The default priority of the ACPI reset is set to below device-specific and above hardware resets.

---
 drivers/pci/pci-acpi.c | 23 +++++++++++++++++++++++
 drivers/pci/pci.c      |  1 +
 drivers/pci/pci.h      |  6 ++++++
 include/linux/pci.h    |  2 +-
 4 files changed, 31 insertions(+), 1 deletion(-)
drivers/pci/pci-acpi.c +23 -0
index eaddbf701..40dd24cd3 100644
@@ -941,6 +941,29 @@ void pci_set_acpi_fwnode(struct pci_dev *dev)
 				   acpi_pci_find_companion(&dev->dev));
 }
 
+/**
+ * pci_dev_acpi_reset - do a function level reset using _RST method
+ * @dev: device to reset
+ * @probe: check if _RST method is included in the acpi_device context.
+ */
+int pci_dev_acpi_reset(struct pci_dev *dev, int probe)
+{
+	acpi_handle handle = ACPI_HANDLE(&dev->dev);
+
+	if (!handle || !acpi_has_method(handle, "_RST"))
+		return -ENOTTY;
+
+	if (probe)
+		return 0;
+
+	if (ACPI_FAILURE(acpi_evaluate_object(handle, "_RST", NULL, NULL))) {
+		pci_warn(dev, "ACPI _RST failed\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 static bool acpi_pci_bridge_d3(struct pci_dev *dev)
 {
 	const struct fwnode_handle *fwnode;
drivers/pci/pci.c +1 -0
index 2302aa421..2e7efd7e7 100644
@@ -5121,6 +5121,7 @@ static void pci_dev_restore(struct pci_dev *dev)
  */
 const struct pci_reset_fn_method pci_reset_fn_methods[] = {
 	{ &pci_dev_specific_reset, .name = "device_specific" },
+	{ &pci_dev_acpi_reset, .name = "acpi" },
 	{ &pcie_reset_flr, .name = "flr" },
 	{ &pci_af_flr, .name = "af_flr" },
 	{ &pci_pm_reset, .name = "pm" },
drivers/pci/pci.h +6 -0
index d22da6d3c..e9cfb7cd6 100644
@@ -704,7 +704,13 @@ static inline int pci_aer_raw_clear_status(struct pci_dev *dev) { return -EINVAL
 int pci_acpi_program_hp_params(struct pci_dev *dev);
 extern const struct attribute_group pci_dev_acpi_attr_group;
 void pci_set_acpi_fwnode(struct pci_dev *dev);
+int pci_dev_acpi_reset(struct pci_dev *dev, int probe);
 #else
+static inline int pci_dev_acpi_reset(struct pci_dev *dev, int probe)
+{
+	return -ENOTTY;
+}
+
 static inline void pci_set_acpi_fwnode(struct pci_dev *dev) {}
 static inline int pci_acpi_program_hp_params(struct pci_dev *dev)
 {
include/linux/pci.h +1 -1
index 6e9bc4f9c..a7f063da2 100644
@@ -49,7 +49,7 @@
 			       PCI_STATUS_SIG_TARGET_ABORT | \
 			       PCI_STATUS_PARITY)
 
-#define PCI_RESET_METHODS_NUM 5
+#define PCI_RESET_METHODS_NUM 6
 
 /*
  * The PCI interface treats multi-function devices as independent
Signature · 1 line
2.31.1
#42 lore raw

PATCH v7 7/8 PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs

In reply to Amey Narkhede

From: Shanker Donthineni <[email protected]>

On select platforms, some Nvidia GPU devices do not work with SBR. Triggering SBR would leave the device inoperable for the current system boot. It requires a system hard-reboot to get the GPU device back to normal operating condition post-SBR. For the affected devices, enable NO_BUS_RESET quirk to fix the issue.

This issue will be fixed in the next generation of hardware.

---
 drivers/pci/quirks.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/pci/quirks.c +12 -0
index e86cf4a3b..45a8c3caa 100644
@@ -3546,6 +3546,18 @@ static void quirk_no_bus_reset(struct pci_dev *dev)
 	dev->dev_flags |= PCI_DEV_FLAGS_NO_BUS_RESET;
 }
 
+/*
+ * Some Nvidia GPU devices do not work with bus reset, SBR needs to be
+ * prevented for those affected devices.
+ */
+static void quirk_nvidia_no_bus_reset(struct pci_dev *dev)
+{
+	if ((dev->device & 0xffc0) == 0x2340)
+		quirk_no_bus_reset(dev);
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
+			 quirk_nvidia_no_bus_reset);
+
 /*
  * Some Atheros AR9xxx and QCA988x chips do not behave after a bus reset.
  * The device will throw a Link Down error on AER-capable systems and
Signature · 1 line
2.31.1
#43 lore raw

Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs

Quoted 3 lines · Amey Narkhede

From: Shanker Donthineni <[email protected]>

On select platforms, some Nvidia GPU devices do not work with SBR.

Interesting that you say "on select platforms." Apparently SBR does work for some of these GPUs, but not on all platforms? If you have any clarification here, I can still update the commit log.

Quoted 9 lines

Triggering SBR would leave the device inoperable for the current system boot. It requires a system hard-reboot to get the GPU device back to normal operating condition post-SBR. For the affected devices, enable NO_BUS_RESET quirk to fix the issue.

This issue will be fixed in the next generation of hardware.

This patch doesn't seem to have any dependencies or particular connection to the rest of the reset series, so I applied this patch by itself to for-linus for v5.13 and marked it for stable.

If that's not right, let me know.

Quoted 30 lines
---
 drivers/pci/quirks.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/pci/quirks.c +12 -0
index e86cf4a3b..45a8c3caa 100644
@@ -3546,6 +3546,18 @@ static void quirk_no_bus_reset(struct pci_dev *dev)
 	dev->dev_flags |= PCI_DEV_FLAGS_NO_BUS_RESET;
 }
 
+/*
+ * Some Nvidia GPU devices do not work with bus reset, SBR needs to be
+ * prevented for those affected devices.
+ */
+static void quirk_nvidia_no_bus_reset(struct pci_dev *dev)
+{
+	if ((dev->device & 0xffc0) == 0x2340)
+		quirk_no_bus_reset(dev);
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
+			 quirk_nvidia_no_bus_reset);
+
 /*
  * Some Atheros AR9xxx and QCA988x chips do not behave after a bus reset.
  * The device will throw a Link Down error on AER-capable systems and
Signature · 1 line
2.31.1
#44 lore raw

Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs

Hi Bjorn,

Quoted 7 lines · Bjorn Helgaas
Quoted 7 lines

Triggering SBR would leave the device inoperable for the current system boot. It requires a system hard-reboot to get the GPU device back to normal operating condition post-SBR. For the affected devices, enable NO_BUS_RESET quirk to fix the issue.

This issue will be fixed in the next generation of hardware.

Quoted 8 lines
Quoted 2 lines

This patch doesn't seem to have any dependencies or particular connection to the rest of the reset series, so I applied this patch by itself to for-linus for v5.13 and marked it for stable.

If that's not right, let me know.

Yes, you're right this patch no dependency on reset method series.

#45 lore raw

Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs

In reply to Bjorn Helgaas

Hi Bjorn,

Quoted 7 lines · Bjorn Helgaas
Quoted 3 lines

From: Shanker Donthineni <[email protected]>

On select platforms, some Nvidia GPU devices do not work with SBR.

Interesting that you say "on select platforms." Apparently SBR does work for some of these GPUs, but not on all platforms? If you have any clarification here, I can still update the commit log.

Yes, SBR works for some GPUs but GPUs which are listed in this quirk will not work and these GPUs are available only on selected server platforms. I believe commit text reflects the issue but please update if needed.

-

#46 lore raw

Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs

Quoted 11 lines · Shanker R Donthineni
Quoted 7 lines · Bjorn Helgaas
Quoted 3 lines

From: Shanker Donthineni <[email protected]>

On select platforms, some Nvidia GPU devices do not work with SBR.

Interesting that you say "on select platforms." Apparently SBR does work for some of these GPUs, but not on all platforms? If you have any clarification here, I can still update the commit log.

Yes, SBR works for some GPUs but GPUs which are listed in this quirk will not work and these GPUs are available only on selected server platforms. I believe commit text reflects the issue but please update if needed.

It sounds like there is no actual dependency on the platform. So even though these GPUs are only available on certain platforms, if one were to move one of them to a different, non-supported platform, SBR would still not work.

So I think I'll remove the reference to "select platforms" since it doesn't add any useful information and might suggest that SBR should work on some platforms, if you could only find the right ones.

Bjorn

#47 lore raw

Re: [PATCH v7 7/8] PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs

Hi Bjorn,

Quoted 8 lines · Bjorn Helgaas

It sounds like there is no actual dependency on the platform. So even though these GPUs are only available on certain platforms, if one were to move one of them to a different, non-supported platform, SBR would still not work.

So I think I'll remove the reference to "select platforms" since it doesn't add any useful information and might suggest that SBR should work on some platforms, if you could only find the right ones.

Appreciate your time on code review, providing better text, and picking patch for v5.14. Please let us know if any code improvements or suggestions for the remaining reset patch series to be considered for v5.14

#48 lore raw

PATCH v7 8/8 PCI: Change the type of probe argument in reset functions

In reply to Amey Narkhede

Introduce a new enum pci_reset_mode_t to make the context of probe argument in reset functions clear and the code easier to read.
Change the type of probe argument in functions which implement reset methods from int to pci_reset_mode_t to make the intent clear.

Add a new line in return statement of pci_reset_bus_function().

---
 drivers/crypto/cavium/nitrox/nitrox_main.c    |  2 +-
 .../ethernet/cavium/liquidio/lio_vf_main.c    |  2 +-
 drivers/pci/hotplug/pciehp.h                  |  2 +-
 drivers/pci/hotplug/pciehp_hpc.c              |  4 +-
 drivers/pci/pci-acpi.c                        | 10 ++-
 drivers/pci/pci.c                             | 85 ++++++++++++-------
 drivers/pci/pci.h                             | 12 +--
 drivers/pci/pcie/aer.c                        |  2 +-
 drivers/pci/quirks.c                          | 37 ++++----
 include/linux/pci.h                           |  8 +-
 include/linux/pci_hotplug.h                   |  2 +-
 11 files changed, 101 insertions(+), 65 deletions(-)
drivers/crypto/cavium/nitrox/nitrox_main.c +1 -1
index 15d6c8452..f97fa8e99 100644
@@ -306,7 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)
 		return -ENOMEM;
 	}
 
-	pcie_reset_flr(pdev, 0);
+	pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
 
 	pci_restore_state(pdev);
 
drivers/net/ethernet/cavium/liquidio/lio_vf_main.c +1 -1
index 336d149ee..6e666be69 100644
@@ -526,7 +526,7 @@ static void octeon_destroy_resources(struct octeon_device *oct)
 			oct->irq_name_storage = NULL;
 		}
 		/* Soft reset the octeon device before exiting */
-		if (!pcie_reset_flr(oct->pci_dev, 1))
+		if (!pcie_reset_flr(oct->pci_dev, PCI_RESET_PROBE))
 			octeon_pci_flr(oct);
 		else
 			cn23xx_vf_ask_pf_to_do_flr(oct);
drivers/pci/hotplug/pciehp.h +1 -1
index 4fd200d8b..87da03adc 100644
@@ -181,7 +181,7 @@ void pciehp_release_ctrl(struct controller *ctrl);
 
 int pciehp_sysfs_enable_slot(struct hotplug_slot *hotplug_slot);
 int pciehp_sysfs_disable_slot(struct hotplug_slot *hotplug_slot);
-int pciehp_reset_slot(struct hotplug_slot *hotplug_slot, int probe);
+int pciehp_reset_slot(struct hotplug_slot *hotplug_slot, pci_reset_mode_t mode);
 int pciehp_get_attention_status(struct hotplug_slot *hotplug_slot, u8 *status);
 int pciehp_set_raw_indicator_status(struct hotplug_slot *h_slot, u8 status);
 int pciehp_get_raw_indicator_status(struct hotplug_slot *h_slot, u8 *status);
drivers/pci/hotplug/pciehp_hpc.c +2 -2
index fb3840e22..24b3c8787 100644
@@ -834,14 +834,14 @@ void pcie_disable_interrupt(struct controller *ctrl)
  * momentarily, if we see that they could interfere. Also, clear any spurious
  * events after.
  */
-int pciehp_reset_slot(struct hotplug_slot *hotplug_slot, int probe)
+int pciehp_reset_slot(struct hotplug_slot *hotplug_slot, pci_reset_mode_t mode)
 {
 	struct controller *ctrl = to_ctrl(hotplug_slot);
 	struct pci_dev *pdev = ctrl_dev(ctrl);
 	u16 stat_mask = 0, ctrl_mask = 0;
 	int rc;
 
-	if (probe)
+	if (mode == PCI_RESET_PROBE)
 		return 0;
 
 	down_write(&ctrl->reset_lock);
drivers/pci/pci-acpi.c +7 -3
index 40dd24cd3..9de334457 100644
@@ -944,16 +944,20 @@ void pci_set_acpi_fwnode(struct pci_dev *dev)
 /**
  * pci_dev_acpi_reset - do a function level reset using _RST method
  * @dev: device to reset
- * @probe: check if _RST method is included in the acpi_device context.
+ * @probe: If PCI_RESET_PROBE, check whether _RST method is included
+ *         in the acpi_device context.
  */
-int pci_dev_acpi_reset(struct pci_dev *dev, int probe)
+int pci_dev_acpi_reset(struct pci_dev *dev, pci_reset_mode_t mode)
 {
 	acpi_handle handle = ACPI_HANDLE(&dev->dev);
 
+	if (mode >= PCI_RESET_MODE_MAX)
+		return -EINVAL;
+
 	if (!handle || !acpi_has_method(handle, "_RST"))
 		return -ENOTTY;
 
-	if (probe)
+	if (mode == PCI_RESET_PROBE)
 		return 0;
 
 	if (ACPI_FAILURE(acpi_evaluate_object(handle, "_RST", NULL, NULL))) {
drivers/pci/pci.c +53 -32

268 lines not on this page · raw message

drivers/pci/pci.h +6 -6
index e9cfb7cd6..9787700f8 100644
@@ -603,19 +603,19 @@ static inline int pci_enable_ptm(struct pci_dev *dev, u8 *granularity)
 struct pci_dev_reset_methods {
 	u16 vendor;
 	u16 device;
-	int (*reset)(struct pci_dev *dev, int probe);
+	int (*reset)(struct pci_dev *dev, pci_reset_mode_t mode);
 };
 
 struct pci_reset_fn_method {
-	int (*reset_fn)(struct pci_dev *pdev, int probe);
+	int (*reset_fn)(struct pci_dev *pdev, pci_reset_mode_t mode);
 	char *name;
 };
 
 extern const struct pci_reset_fn_method pci_reset_fn_methods[];
 #ifdef CONFIG_PCI_QUIRKS
-int pci_dev_specific_reset(struct pci_dev *dev, int probe);
+int pci_dev_specific_reset(struct pci_dev *dev, pci_reset_mode_t mode);
 #else
-static inline int pci_dev_specific_reset(struct pci_dev *dev, int probe)
+static inline int pci_dev_specific_reset(struct pci_dev *dev, pci_reset_mode_t mode)
 {
 	return -ENOTTY;
 }
@@ -704,9 +704,9 @@ static inline int pci_aer_raw_clear_status(struct pci_dev *dev) { return -EINVAL
 int pci_acpi_program_hp_params(struct pci_dev *dev);
 extern const struct attribute_group pci_dev_acpi_attr_group;
 void pci_set_acpi_fwnode(struct pci_dev *dev);
-int pci_dev_acpi_reset(struct pci_dev *dev, int probe);
+int pci_dev_acpi_reset(struct pci_dev *dev, pci_reset_mode_t mode);
 #else
-static inline int pci_dev_acpi_reset(struct pci_dev *dev, int probe)
+static inline int pci_dev_acpi_reset(struct pci_dev *dev, pci_reset_mode_t mode)
 {
 	return -ENOTTY;
 }
drivers/pci/pcie/aer.c +1 -1
index 98077595a..cfa7a1775 100644
@@ -1405,7 +1405,7 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)
 	}
 
 	if (type == PCI_EXP_TYPE_RC_EC || type == PCI_EXP_TYPE_RC_END) {
-		rc = pcie_reset_flr(dev, 0);
+		rc = pcie_reset_flr(dev, PCI_RESET_DO_RESET);
 		if (!rc)
 			pci_info(dev, "has been reset\n");
 		else
drivers/pci/quirks.c +21 -16
index 45a8c3caa..60fd101ac 100644
@@ -3681,7 +3681,7 @@ DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL,
  * reset a single function if other methods (e.g. FLR, PM D0->D3) are
  * not available.
  */
-static int reset_intel_82599_sfp_virtfn(struct pci_dev *dev, int probe)
+static int reset_intel_82599_sfp_virtfn(struct pci_dev *dev, pci_reset_mode_t mode)
 {
 	/*
 	 * http://www.intel.com/content/dam/doc/datasheet/82599-10-gbe-controller-datasheet.pdf
@@ -3691,7 +3691,7 @@ static int reset_intel_82599_sfp_virtfn(struct pci_dev *dev, int probe)
 	 * Thus we must call pcie_flr() directly without first checking if it is
 	 * supported.
 	 */
-	if (!probe)
+	if (mode == PCI_RESET_DO_RESET)
 		pcie_flr(dev);
 	return 0;
 }
@@ -3703,13 +3703,13 @@ static int reset_intel_82599_sfp_virtfn(struct pci_dev *dev, int probe)
 #define NSDE_PWR_STATE		0xd0100
 #define IGD_OPERATION_TIMEOUT	10000     /* set timeout 10 seconds */
 
-static int reset_ivb_igd(struct pci_dev *dev, int probe)
+static int reset_ivb_igd(struct pci_dev *dev, pci_reset_mode_t mode)
 {
 	void __iomem *mmio_base;
 	unsigned long timeout;
 	u32 val;
 
-	if (probe)
+	if (mode == PCI_RESET_PROBE)
 		return 0;
 
 	mmio_base = pci_iomap(dev, 0, 0);
@@ -3746,7 +3746,7 @@ static int reset_ivb_igd(struct pci_dev *dev, int probe)
 }
 
 /* Device-specific reset method for Chelsio T4-based adapters */
-static int reset_chelsio_generic_dev(struct pci_dev *dev, int probe)
+static int reset_chelsio_generic_dev(struct pci_dev *dev, pci_reset_mode_t mode)
 {
 	u16 old_command;
 	u16 msix_flags;
@@ -3762,7 +3762,7 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, int probe)
 	 * If this is the "probe" phase, return 0 indicating that we can
 	 * reset this device.
 	 */
-	if (probe)
+	if (mode == PCI_RESET_PROBE)
 		return 0;
 
 	/*
@@ -3824,17 +3824,17 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, int probe)
  *    Chapter 3: NVMe control registers
  *    Chapter 7.3: Reset behavior
  */
-static int nvme_disable_and_flr(struct pci_dev *dev, int probe)
+static int nvme_disable_and_flr(struct pci_dev *dev, pci_reset_mode_t mode)
 {
 	void __iomem *bar;
 	u16 cmd;
 	u32 cfg;
 
 	if (dev->class != PCI_CLASS_STORAGE_EXPRESS ||
-	    pcie_reset_flr(dev, 1) || !pci_resource_start(dev, 0))
+	    pcie_reset_flr(dev, PCI_RESET_PROBE) || !pci_resource_start(dev, 0))
 		return -ENOTTY;
 
-	if (probe)
+	if (mode == PCI_RESET_PROBE)
 		return 0;
 
 	bar = pci_iomap(dev, 0, NVME_REG_CC + sizeof(cfg));
@@ -3898,11 +3898,13 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)
  * device too soon after FLR.  A 250ms delay after FLR has heuristically
  * proven to produce reliably working results for device assignment cases.
  */
-static int delay_250ms_after_flr(struct pci_dev *dev, int probe)
+static int delay_250ms_after_flr(struct pci_dev *dev, pci_reset_mode_t mode)
 {
-	int ret = pcie_reset_flr(dev, probe);
+	int ret;
+
+	ret = pcie_reset_flr(dev, mode);
 
-	if (probe)
+	if (ret || mode == PCI_RESET_PROBE)
 		return ret;
 
 	msleep(250);
@@ -3918,13 +3920,13 @@ static int delay_250ms_after_flr(struct pci_dev *dev, int probe)
 #define HINIC_OPERATION_TIMEOUT     15000	/* 15 seconds */
 
 /* Device-specific reset method for Huawei Intelligent NIC virtual functions */
-static int reset_hinic_vf_dev(struct pci_dev *pdev, int probe)
+static int reset_hinic_vf_dev(struct pci_dev *pdev, pci_reset_mode_t mode)
 {
 	unsigned long timeout;
 	void __iomem *bar;
 	u32 val;
 
-	if (probe)
+	if (mode == PCI_RESET_PROBE)
 		return 0;
 
 	bar = pci_iomap(pdev, 0, 0);
@@ -3995,16 +3997,19 @@ static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
  * because when a host assigns a device to a guest VM, the host may need
  * to reset the device but probably doesn't have a driver for it.
  */
-int pci_dev_specific_reset(struct pci_dev *dev, int probe)
+int pci_dev_specific_reset(struct pci_dev *dev, pci_reset_mode_t mode)
 {
 	const struct pci_dev_reset_methods *i;
 
+	if (mode >= PCI_RESET_MODE_MAX)
+		return -EINVAL;
+
 	for (i = pci_dev_reset_methods; i->reset; i++) {
 		if ((i->vendor == dev->vendor ||
 		     i->vendor == (u16)PCI_ANY_ID) &&
 		    (i->device == dev->device ||
 		     i->device == (u16)PCI_ANY_ID))
-			return i->reset(dev, probe);
+			return i->reset(dev, mode);
 	}
 
 	return -ENOTTY;
include/linux/pci.h +7 -1
index a7f063da2..c46df52e6 100644
@@ -51,6 +51,12 @@
 
 #define PCI_RESET_METHODS_NUM 6
 
+typedef enum pci_reset_mode {
+	PCI_RESET_DO_RESET,
+	PCI_RESET_PROBE,
+	PCI_RESET_MODE_MAX,
+} pci_reset_mode_t;
+
 /*
  * The PCI interface treats multi-function devices as independent
  * devices.  The slot/function address of each device is encoded
@@ -1230,7 +1236,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
 			     enum pci_bus_speed *speed,
 			     enum pcie_link_width *width);
 void pcie_print_link_status(struct pci_dev *dev);
-int pcie_reset_flr(struct pci_dev *dev, int probe);
+int pcie_reset_flr(struct pci_dev *dev, pci_reset_mode_t mode);
 int pcie_flr(struct pci_dev *dev);
 bool pci_reset_supported(struct pci_dev *dev);
 int __pci_reset_function_locked(struct pci_dev *dev);
include/linux/pci_hotplug.h +1 -1
index b482e42d7..9e8da46e7 100644
@@ -44,7 +44,7 @@ struct hotplug_slot_ops {
 	int (*get_attention_status)	(struct hotplug_slot *slot, u8 *value);
 	int (*get_latch_status)		(struct hotplug_slot *slot, u8 *value);
 	int (*get_adapter_status)	(struct hotplug_slot *slot, u8 *value);
-	int (*reset_slot)		(struct hotplug_slot *slot, int probe);
+	int (*reset_slot)		(struct hotplug_slot *slot, pci_reset_mode_t mode);
 };
 
 /**

--
2.31.1

Reviewed-by on v7 8/8
#49 lore raw

Re: [PATCH v7 8/8] PCI: Change the type of probe argument in reset functions

Quoted 10 lines · Amey Narkhede

Introduce a new enum pci_reset_mode_t to make the context of probe argument in reset functions clear and the code easier to read.
Change the type of probe argument in functions which implement reset methods from int to pci_reset_mode_t to make the intent clear.

Add a new line in return statement of pci_reset_bus_function().

Quoted 13 lines
---
 drivers/crypto/cavium/nitrox/nitrox_main.c    |  2 +-
 .../ethernet/cavium/liquidio/lio_vf_main.c    |  2 +-
 drivers/pci/hotplug/pciehp.h                  |  2 +-
 drivers/pci/hotplug/pciehp_hpc.c              |  4 +-
 drivers/pci/pci-acpi.c                        | 10 ++-
 drivers/pci/pci.c                             | 85 ++++++++++++-------
 drivers/pci/pci.h                             | 12 +--
 drivers/pci/pcie/aer.c                        |  2 +-
 drivers/pci/quirks.c                          | 37 ++++----
 include/linux/pci.h                           |  8 +-
 include/linux/pci_hotplug.h                   |  2 +-
 11 files changed, 101 insertions(+), 65 deletions(-)
#50 lore raw
In reply to Amey Narkhede

On 08.06.21 07:48, Amey Narkhede wrote:

Hi,

Quoted 5 lines

PCI and PCIe devices may support a number of possible reset mechanisms for example Function Level Reset (FLR) provided via Advanced Feature or PCIe capabilities, Power Management reset, bus reset, or device specific reset. Currently the PCI subsystem creates a policy prioritizing these reset methods which provides neither visibility nor control to userspace.

Since I've got a current use case for that - could you perhaps tell more about the whole pci device reset mechanisms ?

In my case I've got a board that wires reset lines to the soc's gpios. Not sure how exactly to qualify this, but I guess it would count as a bus wide reset.

Now the big question for me is how to implement that in a board specific platform driver (which already does setup of gpios and other attached devices), so we can reset the card in slot X in a generic way.

Any help highly appreciated.

--mtx

Signature · 8 lines
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
#51 lore raw
Quoted 17 lines · metux IT consult

On 08.06.21 07:48, Amey Narkhede wrote:

Hi,

Quoted 5 lines

PCI and PCIe devices may support a number of possible reset mechanisms for example Function Level Reset (FLR) provided via Advanced Feature or PCIe capabilities, Power Management reset, bus reset, or device specific reset. Currently the PCI subsystem creates a policy prioritizing these reset methods which provides neither visibility nor control to userspace.

Since I've got a current use case for that - could you perhaps tell more about the whole pci device reset mechanisms ?

In my case I've got a board that wires reset lines to the soc's gpios. Not sure how exactly to qualify this, but I guess it would count as a bus wide reset.

Quoted 9 lines

Now the big question for me is how to implement that in a board specific platform driver (which already does setup of gpios and other attached devices), so we can reset the card in slot X in a generic way.

Any help highly appreciated.

--mtx

In case of bus reset(pci_reset_secondary_bus()), it uses bridge control register to assert reset on bus so I think it should out of the box but not 100% sure about it.

Thanks,
Amey