Expose and manage PCI device reset
Subsystems (5)
v2 cover Expose and manage PCI device reset
[RESEND with Shanker's patches as those depend on this series]
PCI and PCIe devices may support a number of possible reset mechanisms for example Function Level Reset (FLR) provided via Advanced Feature or PCIe capabilities, Power Management reset, bus reset, or device specific reset. Currently the PCI subsystem creates a policy prioritizing these reset methods which provides neither visibility nor control to userspace.
Expose the reset methods available per device to userspace, via sysfs and allow an administrative user or device owner to have ability to manage per device reset method priorities or exclusions. This feature aims to allow greater control of a device for use cases as device assignment, where specific device or platform issues may interact poorly with a given reset method, and for which device specific quirks have not been developed.
Changes in v2:
- Use byte array instead of bitmap to keep track of ordering of reset methods
- Fix incorrect use of reset_fn field in octeon driver
- Allow writing comma separated list of names of supported reset methods to reset_method sysfs attribute
- Writing empty string instead of "none" to reset_method attribute disables ability of reset the device
Sending Raphael's patch again as this series depends on it.
Amey Narkhede (4):
PCI: Add pcie_reset_flr to follow calling convention of other reset
methods
PCI: Add new array for keeping track of ordering of reset methods
PCI: Remove reset_fn field from pci_dev
PCI/sysfs: Allow userspace to query and set device reset mechanism
Raphael Norwitz (1):
PCI: merge slot and bus reset implementations
Shanker Donthineni (2):
PCI: Add support for a function level reset based on _RST method
PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
Documentation/ABI/testing/sysfs-bus-pci | 16 ++ drivers/crypto/cavium/nitrox/nitrox_main.c | 4 +- .../ethernet/cavium/liquidio/lio_vf_main.c | 2 +- drivers/pci/pci-sysfs.c | 93 +++++++- drivers/pci/pci.c | 206 +++++++++++------- drivers/pci/pci.h | 10 +- drivers/pci/pcie/aer.c | 12 +- drivers/pci/probe.c | 4 +- drivers/pci/quirks.c | 23 +- include/linux/pci.h | 11 +- 10 files changed, 278 insertions(+), 103 deletions(-)
--
2.31.1
PATCH RESEND v2 1/7 PCI: merge slot and bus reset implementations
From: Raphael Norwitz <[email protected]>
Slot resets are bus resets with additional logic to prevent a device from being removed during the reset. Currently slot and bus resets have separate implementations in pci.c, complicating higher level logic. As discussed on the mailing list, they should be combined into a generic function which performs an SBR. This change adds a function, pci_reset_bus_function(), which first attempts a slot reset and then attempts a bus reset if -ENOTTY is returned, such that there is now a single device agnostic function to perform an SBR.
This new function is also needed to add SBR reset quirks and therefore is exposed in pci.h.
--- drivers/pci/pci.c | 19 +++++++++++-------- include/linux/pci.h | 1 + 2 files changed, 12 insertions(+), 8 deletions(-)
drivers/pci/pci.c +11 -8
@@ -4982,6 +4982,15 @@ static int pci_dev_reset_slot_function(struct pci_dev *dev, int probe)return pci_reset_hotplug_slot (dev->slot->hotplug, probe); } +int pci_reset_bus_function (struct pci_dev *dev,int probe) +{ +int rc =pci_dev_reset_slot_function (dev, probe); + +if (rc != -ENOTTY ) +return rc; +return pci_parent_bus_reset (dev, probe); +} +static void pci_dev_lock (struct pci_dev *dev) {pci_cfg_access_lock (dev);
@@ -5102,10 +5111,7 @@ int __pci_reset_function_locked(struct pci_dev *dev) rc =pci_pm_reset (dev,0 );if (rc != -ENOTTY )return rc; - rc =pci_dev_reset_slot_function (dev,0 ); -if (rc != -ENOTTY ) -return rc; -return pci_parent_bus_reset (dev,0 ); +return pci_reset_bus_function (dev,0 ); }EXPORT_SYMBOL_GPL (__pci_reset_function_locked);
@@ -5135,13 +5141,10 @@ int pci_probe_reset_function(struct pci_dev *dev)if (rc != -ENOTTY )return rc; rc =pci_pm_reset (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_dev_reset_slot_function (dev,1 );if (rc != -ENOTTY )return rc; -return pci_parent_bus_reset (dev,1 ); +return pci_reset_bus_function (dev,1 ); }/**
include/linux/pci.h +1 -0
@@ -1228,6 +1228,7 @@ int pci_probe_reset_bus(struct pci_bus *bus);int pci_reset_bus (struct pci_dev *dev);void pci_reset_secondary_bus (struct pci_dev *dev);void pcibios_reset_secondary_bus (struct pci_dev *dev); +int pci_reset_bus_function (struct pci_dev *dev,int probe);void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int __must_check pci_reassign_resource (struct pci_dev *dev,int i,resource_size_t add_size,resource_size_t align);
Signature · 1 line
2.31.1
Re: [PATCH RESEND v2 1/7] PCI: merge slot and bus reset implementations
Hi Amey,
Thank you for working on this! Few comments and suggestions below.
[...]
Quoted 1 line
Linking to lkml.org is fine, however it became a canon now to link to lore, so this would be:
https://lore.kernel.org/lkml/[email protected]/
I personally find it a bit easier to read on lore compared to lkml.org when it goes to a large and long running threads.
[...]
drivers/pci/pci.c static int pci_dev_reset_slot_function(struct pci_dev *dev, int probe) · line 4992
Quoted 8 lines
drivers/pci/pci.c +8
-0+int pci_reset_bus_function (struct pci_dev *dev,int probe) +{ +int rc =pci_dev_reset_slot_function (dev, probe); + +if (rc != -ENOTTY ) +return rc; +return pci_parent_bus_reset (dev, probe); +}
Depends on the style, but I would suggest using a boolean type for the probe argument here and in the other functions that enable or disable something. I makes the intent clear, and this is also a popular pattern you can see throughout the PCI tree.
Also, I would suggest adding a newline to separate final return, so that it's easier to read the code, and to keep things consistent.
[...]
drivers/pci/pci.c int __pci_reset_function_locked(struct pci_dev *dev) · line 5114
Quoted 5 lines
drivers/pci/pci.c +1
-4- rc =pci_dev_reset_slot_function (dev,0 ); -if (rc != -ENOTTY ) -return rc; -return pci_parent_bus_reset (dev,0 ); +return pci_reset_bus_function (dev,0 );
See above about using boolean type here.
[...]
drivers/pci/pci.c int pci_probe_reset_function(struct pci_dev *dev) · line 5147
Quoted 6 lines
drivers/pci/pci.c +1
-2- rc =pci_dev_reset_slot_function (dev,1 );if (rc != -ENOTTY )return rc; -return pci_parent_bus_reset (dev,1 ); +return pci_reset_bus_function (dev,1 );
Same as above.
Krzysztof
PATCH RESEND v2 2/7 PCI: Add pcie_reset_flr to follow calling convention of other reset methods
In reply to Amey NarkhedeCurrently there is separate function pcie_has_flr to probe if pcie flr is supported by the device which does not match the calling convention followed by reset methods which use second function argument to decide whether to probe or not. Add new function pcie_reset_flr that follows the calling convention of reset methods.
--- drivers/crypto/cavium/nitrox/nitrox_main.c | 4 +- drivers/pci/pci.c | 62 ++++++++++++---------- drivers/pci/pcie/aer.c | 12 ++--- drivers/pci/quirks.c | 9 ++-- include/linux/pci.h | 2 +- 5 files changed, 43 insertions(+), 46 deletions(-)
drivers/crypto/cavium/nitrox/nitrox_main.c +1 -3
@@ -306,9 +306,7 @@ static int nitrox_device_flr(struct pci_dev *pdev)return -ENOMEM ; } -/* check flr support */ -if (pcie_has_flr (pdev)) -pcie_flr (pdev); +pcie_reset_flr (pdev,0 );pci_restore_state (pdev);
drivers/pci/pci.c +33 -29
90 lines not on this page · raw message
drivers/pci/pcie/aer.c +5 -7
@@ -1405,13 +1405,11 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev) }if (type ==PCI_EXP_TYPE_RC_EC || type ==PCI_EXP_TYPE_RC_END ) { -if (pcie_has_flr (dev)) { - rc =pcie_flr (dev); -pci_info (dev,"has been reset (%d)\n" , rc); - }else { -pci_info (dev,"not reset (no FLR support)\n" ); - rc = -ENOTTY ; - } + rc =pcie_reset_flr (dev,0 ); +if (!rc) +pci_info (dev,"has been reset\n" ); +else +pci_info (dev,"not reset (no FLR support: %d)\n" , rc); }else { rc =pci_bus_error_reset (dev); pci_info(dev, "%s Port link has been reset (%d)\n",
drivers/pci/quirks.c +3 -6
@@ -3831,7 +3831,7 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)u32 cfg;if (dev->class !=PCI_CLASS_STORAGE_EXPRESS || - !pcie_has_flr (dev) || !pci_resource_start (dev,0 )) +pcie_reset_flr (dev,1 ) || !pci_resource_start (dev,0 ))return -ENOTTY ;if (probe)
@@ -3900,13 +3900,10 @@ static int nvme_disable_and_flr(struct pci_dev *dev, int probe)*/ static int delay_250ms_after_flr (struct pci_dev *dev,int probe) { -if (!pcie_has_flr (dev)) -return -ENOTTY ; +int ret =pcie_reset_flr (dev, probe);if (probe) -return 0 ; - -pcie_flr (dev); +return ret;msleep (250 );
include/linux/pci.h +1 -1
@@ -1217,7 +1217,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,enum pci_bus_speed *speed, enum pcie_link_width *width);void pcie_print_link_status (struct pci_dev *dev); -bool pcie_has_flr (struct pci_dev *dev); +int pcie_reset_flr (struct pci_dev *dev,int probe);int pcie_flr (struct pci_dev *dev);int __pci_reset_function_locked (struct pci_dev *dev);int pci_reset_function (struct pci_dev *dev);
--
2.31.1
Re: [PATCH RESEND v2 2/7] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Hi Amey,
[...]
Quoted 6 lines
drivers/pci/pci.c +6
-0+int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) +return -ENOTTY ;
drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4626
Quoted 10 lines
drivers/pci/pci.c +10
-0+ +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev); +}
Similarly to my suggestion in the first patch in the series, perhaps using a boolean here would be an option.
Having said that, the following existing functions aren't doing it, so for the sake of keeping things consistent it might not be the best option, as per:
static int pci_af_flr(struct pci_dev *dev, int probe) int nvme_disable_and_flr(struct pci_dev *dev, int probe)
Krzysztof
Re: [PATCH RESEND v2 2/7] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Quoted 23 lines · Krzysztof Wilczyński
Hi Amey,
[...]
Quoted 6 lines
drivers/pci/pci.c +6
-0+int pcie_reset_flr (struct pci_dev *dev,int probe) +{ +u32 cap; + +if (dev->dev_flags &PCI_DEV_FLAGS_NO_FLR_RESET ) +return -ENOTTY ;drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4626
Quoted 10 lines
drivers/pci/pci.c +10
-0+ +pcie_capability_read_dword (dev,PCI_EXP_DEVCAP , &cap); +if (!(cap &PCI_EXP_DEVCAP_FLR )) +return -ENOTTY ; + +if (probe) +return 0 ; + +return pcie_flr (dev); +}Similarly to my suggestion in the first patch in the series, perhaps using a boolean here would be an option.
Quoted 8 lines
Having said that, the following existing functions aren't doing it, so for the sake of keeping things consistent it might not be the best option, as per:
static int
pci_af_flr(struct pci_dev *dev, int probe)intnvme_disable_and_flr(struct pci_dev *dev, int probe)Krzysztof
All the functions which implement different types of resets including quirks have ...reset(struct pci_dev *dev, int probe) signature. Should I modify all of them?
Thanks,
Amey
Re: [PATCH RESEND v2 2/7] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Hi Amey,
Sorry for late reply!
[...]
Quoted 7 lines
Quoted 7 lines
Similarly to my suggestion in the first patch in the series, perhaps using a boolean here would be an option.
Having said that, the following existing functions aren't doing it, so for the sake of keeping things consistent it might not be the best option, as per:
Quoted 8 lines
Quoted 4 lines
static int
pci_af_flr(struct pci_dev *dev, int probe)intnvme_disable_and_flr(struct pci_dev *dev, int probe)Krzysztof
All the functions which implement different types of resets including quirks have ...reset(struct
pci_dev*dev, int probe) signature. Should I modify all of them?
Might not be worth it to change anything then, especially if the other functions there already use an integer argument to enable or disable the problem or something else. At least no in this series.
Krzysztof
Re: [PATCH RESEND v2 2/7] PCI: Add pcie_reset_flr to follow calling convention of other reset methods
Quoted 21 lines · Krzysztof Wilczyński
Hi Amey,
Sorry for late reply!
[...]
Quoted 7 lines
Quoted 7 lines
Similarly to my suggestion in the first patch in the series, perhaps using a boolean here would be an option.
Having said that, the following existing functions aren't doing it, so for the sake of keeping things consistent it might not be the best option, as per:
Quoted 8 lines
Quoted 4 lines
static int
pci_af_flr(struct pci_dev *dev, int probe)intnvme_disable_and_flr(struct pci_dev *dev, int probe)Krzysztof
All the functions which implement different types of resets including quirks have ...reset(struct
pci_dev*dev, int probe) signature. Should I modify all of them?
Quoted 5 lines
Might not be worth it to change anything then, especially if the other functions there already use an integer argument to enable or disable the problem or something else. At least no in this series.
Krzysztof
Actually I made a new separate patch at the end to implement this change. I'll send v3 soon.
Thanks,
Amey
PATCH RESEND v2 3/7 PCI: Add new array for keeping track of ordering of reset methods
In reply to Amey NarkhedeIntroduce a new array reset_methods in struct pci_dev to keep track of reset mechanisms supported by the device and their ordering. Also refactor probing and reset functions to take advantage of calling convention of reset functions.
--- drivers/pci/pci.c | 107 ++++++++++++++++++++++++++------------------ drivers/pci/pci.h | 10 ++++- drivers/pci/probe.c | 5 +-- include/linux/pci.h | 7 +++ 4 files changed, 82 insertions(+), 47 deletions(-)
drivers/pci/pci.c +64 -43
@@ -72,6 +72,14 @@ static void pci_dev_d3_sleep(struct pci_dev *dev)msleep (delay); } +bool pci_reset_supported (struct pci_dev *dev) +{ +u8 null_reset_methods[PCI_RESET_FN_METHODS ] = {0 }; + +return memcmp (null_reset_methods, + dev->reset_methods,PCI_RESET_FN_METHODS ); +} +#ifdef CONFIG_PCI_DOMAINS int pci_domains_supported =1 ;#endif
@@ -5068,6 +5076,19 @@ static void pci_dev_restore(struct pci_dev *dev) err_handler->reset_done (dev); } +/* +* The ordering for functions in pci_reset_fn_methods +* is required for reset_methods byte array defined +* in struct pci_dev +*/ +const struct pci_reset_fn_method pci_reset_fn_methods[] = { + { .reset_fn = &pci_dev_specific_reset, .name ="device_specific" }, + { .reset_fn = &pcie_reset_flr, .name ="flr" }, + { .reset_fn = &pci_af_flr, .name ="af_flr" }, + { .reset_fn = &pci_pm_reset, .name ="pm" }, + { .reset_fn = &pci_reset_bus_function, .name ="bus" }, +}; +/** * __pci_reset_function_locked - reset a PCI device function while holding * the @dev mutex lock.
@@ -5090,65 +5111,65 @@ static void pci_dev_restore(struct pci_dev *dev)*/ int __pci_reset_function_locked (struct pci_dev *dev) { -int rc; +int i, rc = -ENOTTY ; +u8 prio;might_sleep (); -/* -* A reset method returns -ENOTTY if it doesn't support this device -* and we should try the next method. -* -* If it returns 0 (success), we're finished. If it returns any -* other error, we're also finished: this indicates that further -* reset mechanisms might be broken on the device. -*/ - rc =pci_dev_specific_reset (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pcie_reset_flr (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_af_flr (dev,0 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_pm_reset (dev,0 ); -if (rc != -ENOTTY ) -return rc; -return pci_reset_bus_function (dev,0 ); +for (prio =PCI_RESET_FN_METHODS ; prio; prio--) { +for (i =0 ; i <PCI_RESET_FN_METHODS ; i++) { +if (dev->reset_methods[i] == prio) { +/* +* A reset method returns -ENOTTY if it doesn't support this device +* and we should try the next method. +* +* If it returns 0 (success), we're finished. If it returns any +* other error, we're also finished: this indicates that further +* reset mechanisms might be broken on the device. +*/ + rc = pci_reset_fn_methods[i].reset_fn (dev,0 ); +if (rc != -ENOTTY ) +return rc; +break ; + } + } +if (i ==PCI_RESET_FN_METHODS ) +break ; + } +return rc; }EXPORT_SYMBOL_GPL (__pci_reset_function_locked);/** -* pci_probe_reset_function - check whether the device can be safely reset -* @dev: PCI device to reset +* pci_init_reset_methods - check whether device can be safely reset +* and store supported reset mechanisms. +* @dev: PCI device to check for reset mechanisms * * Some devices allow an individual function to be reset without affecting * other functions in the same device. The PCI device must be responsive -* to PCI config space in order to use this function. +* to reads and writes to its PCI config space in order to use this function. * -* Returns 0 if the device function can be reset or negative if the -* device doesn't support resetting a single function. +* Stores reset mechanisms supported by device in reset_methods byte array +* which is a member of struct pci_dev */ -int pci_probe_reset_function (struct pci_dev *dev) +void pci_init_reset_methods (struct pci_dev *dev) { -int rc; +int i, rc; +u8 prio =PCI_RESET_FN_METHODS ; +u8 reset_methods[PCI_RESET_FN_METHODS ] = {0 }; -might_sleep (); +BUILD_BUG_ON (ARRAY_SIZE (pci_reset_fn_methods) !=PCI_RESET_FN_METHODS ); - rc =pci_dev_specific_reset (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pcie_reset_flr (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_af_flr (dev,1 ); -if (rc != -ENOTTY ) -return rc; - rc =pci_pm_reset (dev,1 ); -if (rc != -ENOTTY ) -return rc; +might_sleep (); -return pci_reset_bus_function (dev,1 ); +for (i =0 ; i <PCI_RESET_FN_METHODS ; i++) { + rc = pci_reset_fn_methods[i].reset_fn (dev,1 ); +if (!rc) + reset_methods[i] = prio--; +else if (rc != -ENOTTY ) +break ; + } +memcpy (dev->reset_methods, reset_methods,sizeof (reset_methods)); }/**
drivers/pci/pci.h +9 -1
@@ -39,7 +39,7 @@ enum pci_mmap_api {int pci_mmap_fits (struct pci_dev *pdev,int resno,struct vm_area_struct *vmai,enum pci_mmap_api mmap_api); -int pci_probe_reset_function (struct pci_dev *dev); +void pci_init_reset_methods (struct pci_dev *dev);int pci_bridge_secondary_bus_reset (struct pci_dev *dev);int pci_bus_error_reset (struct pci_dev *dev);
@@ -612,6 +612,14 @@ struct pci_dev_reset_methods {int (*reset)(struct pci_dev *dev,int probe); }; +typedef int (*pci_reset_fn_t )(struct pci_dev *,int ); + +struct pci_reset_fn_method { +pci_reset_fn_t reset_fn; +char *name; +}; + +extern const struct pci_reset_fn_method pci_reset_fn_methods[];#ifdef CONFIG_PCI_QUIRKS int pci_dev_specific_reset (struct pci_dev *dev,int probe);#else
drivers/pci/probe.c +2 -3
@@ -2403,9 +2403,8 @@ static void pci_init_capabilities(struct pci_dev *dev)pci_rcec_init (dev);/* Root Complex Event Collector */ pcie_report_downtraining (dev); - -if (pci_probe_reset_function (dev) ==0 ) - dev->reset_fn =1 ; +pci_init_reset_methods (dev); + dev->reset_fn =pci_reset_supported (dev); }/*
include/linux/pci.h +7 -0
@@ -49,6 +49,8 @@ PCI_STATUS_SIG_TARGET_ABORT | \ PCI_STATUS_PARITY) +#define PCI_RESET_FN_METHODS 5 +/* * The PCI interface treats multi-function devices as independent * devices. The slot/function address of each device is encoded
@@ -506,6 +508,10 @@ struct pci_dev {char *driver_override;/* Driver name to force a match */ unsigned long priv_flags;/* Private flags for the PCI driver */ +/* +* See pci_reset_fn_methods array in pci.c for ordering +*/ +u8 reset_methods[PCI_RESET_FN_METHODS ];/* Array for storing ordering of reset methods */ };static inline struct pci_dev *pci_physfn (struct pci_dev *dev)
@@ -1219,6 +1225,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,void pcie_print_link_status (struct pci_dev *dev);int pcie_reset_flr (struct pci_dev *dev,int probe);int pcie_flr (struct pci_dev *dev); +bool pci_reset_supported (struct pci_dev *dev);int __pci_reset_function_locked (struct pci_dev *dev);int pci_reset_function (struct pci_dev *dev);int pci_reset_function_locked (struct pci_dev *dev);
Signature · 1 line
2.31.1
Re: [PATCH RESEND v2 3/7] PCI: Add new array for keeping track of ordering of reset methods
Hi Amey,
[...]
Quoted 5 lines
+/*
+ * The ordering for functions inpci_reset_fn_methods
+ * is required forreset_methodsbyte array defined
+ * in structpci_dev
+ */
A small nitpick: missing period at the end of the sentence in the comment above, and in other comments too. Might add for completeness and consistency.
[...]
drivers/pci/pci.h struct pci_dev_reset_methods { · line 620
Quoted 6 lines
drivers/pci/pci.h +6
-0+typedef int (*pci_reset_fn_t )(struct pci_dev *,int ); + +struct pci_reset_fn_method { +pci_reset_fn_t reset_fn; +char *name; +};
Question about the custom type definition above: would it be really needed? It there is only potentially a limited use for it, then perhaps it would not be useful to have one?
Linus also has some preference on usage of custom types, as per:
https://yarchive.net/comp/linux/typedefs.html
But, in the end, this really boils down to a matter of style and/or preference.
[...]
Quoted 1 line
+#define
PCI_RESET_FN_METHODS5
Not sure if worth changing name of this constant, but what about the following:
#define PCI_RESET_FN_METHODS_NUM 5
Or even perhaps:
#define PCI_RESET_METHODS_NUM 5
So it's a little bit more self-explanatory. This would be in the similar notion, as per:
https://elixir.bootlin.com/linux/v5.13-rc2/source/include/linux/pci.h#L115
[...]
include/linux/pci.h struct pci_dev { · line 514
Quoted 1 line
include/linux/pci.h +1
-0+u8 reset_methods[PCI_RESET_FN_METHODS ];/* Array for storing ordering of reset methods */
This comment reads somewhat awkward - we know that an array would be used, most likely, for storing things, thus what about the following:
/* Reset methods ordered by priority */
Just a suggestion, though.
Krzysztof
PATCH RESEND v2 4/7 PCI: Remove reset_fn field from pci_dev
In reply to Amey Narkhedereset_fn field is used to indicate whether the device supports any reset mechanism or not. Deprecate use of reset_fn in favor of new reset_methods array which can be used to keep track of all supported reset mechanisms of a device and their ordering.
The octeon driver is incorrectly using reset_fn field to detect if the device supports FLR or not. Use pcie_reset_flr to probe whether it supports FLR or not.
--- drivers/net/ethernet/cavium/liquidio/lio_vf_main.c | 2 +- drivers/pci/pci-sysfs.c | 6 ++---- drivers/pci/pci.c | 6 +++--- drivers/pci/probe.c | 1 - drivers/pci/quirks.c | 2 +- include/linux/pci.h | 1 - 6 files changed, 7 insertions(+), 11 deletions(-)
drivers/net/ethernet/cavium/liquidio/lio_vf_main.c +1 -1
@@ -526,7 +526,7 @@ static void octeon_destroy_resources(struct octeon_device *oct) oct->irq_name_storage =NULL ; }/* Soft reset the octeon device before exiting */ -if (oct->pci_dev->reset_fn) +if (!pcie_reset_flr (oct->pci_dev,1 ))octeon_pci_flr (oct);else cn23xx_vf_ask_pf_to_do_flr (oct);
drivers/pci/pci-sysfs.c +2 -4
@@ -1334,7 +1334,7 @@ static int pci_create_capabilities_sysfs(struct pci_dev *dev)pcie_vpd_create_sysfs_dev_files (dev); -if (dev->reset_fn) { +if (pci_reset_supported (dev)) { retval =device_create_file (&dev->dev, &dev_attr_reset);if (retval) goto error;
@@ -1417,10 +1417,8 @@ int __must_check pci_create_sysfs_dev_files(struct pci_dev *pdev)static void pci_remove_capabilities_sysfs (struct pci_dev *dev) {pcie_vpd_remove_sysfs_dev_files (dev); -if (dev->reset_fn) { +if (pci_reset_supported (dev))device_remove_file (&dev->dev, &dev_attr_reset); - dev->reset_fn =0 ; - } }/**
drivers/pci/pci.c +3 -3
@@ -5192,7 +5192,7 @@ int pci_reset_function(struct pci_dev *dev) {int rc; -if (!dev->reset_fn) +if (!pci_reset_supported (dev))return -ENOTTY ;pci_dev_lock (dev);
@@ -5228,7 +5228,7 @@ int pci_reset_function_locked(struct pci_dev *dev) {int rc; -if (!dev->reset_fn) +if (!pci_reset_supported (dev))return -ENOTTY ;pci_dev_save_and_disable (dev);
@@ -5251,7 +5251,7 @@ int pci_try_reset_function(struct pci_dev *dev) {int rc; -if (!dev->reset_fn) +if (!pci_reset_supported (dev))return -ENOTTY ;if (!pci_dev_trylock (dev))
drivers/pci/probe.c +0 -1
@@ -2404,7 +2404,6 @@ static void pci_init_capabilities(struct pci_dev *dev)pcie_report_downtraining (dev);pci_init_reset_methods (dev); - dev->reset_fn =pci_reset_supported (dev); }/*
drivers/pci/quirks.c +1 -1
@@ -5535,7 +5535,7 @@ static void quirk_reset_lenovo_thinkpad_p50_nvgpu(struct pci_dev *pdev)if (pdev->subsystem_vendor !=PCI_VENDOR_ID_LENOVO || pdev->subsystem_device !=0x222e || - !pdev->reset_fn) + !pci_reset_supported (pdev))return ;if (pci_enable_device_mem (pdev))
include/linux/pci.h +0 -1
@@ -429,7 +429,6 @@ struct pci_dev { unsigned int state_saved:1; unsigned int is_physfn:1; unsigned int is_virtfn:1; - unsigned int reset_fn:1; unsigned int is_hotplug_bridge:1; unsigned int shpc_managed:1; /* SHPC owned by shpchp */ unsigned int is_thunderbolt:1; /* Thunderbolt controller */
Signature · 1 line
2.31.1
PATCH RESEND v2 5/7 PCI/sysfs: Allow userspace to query and set device reset mechanism
In reply to Amey NarkhedeAdd reset_method sysfs attribute to enable user to query and set user preferred device reset methods and their ordering.
--- Documentation/ABI/testing/sysfs-bus-pci | 16 +++++ drivers/pci/pci-sysfs.c | 91 ++++++++++++++++++++++++- 2 files changed, 104 insertions(+), 3 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +16 -0
@@ -121,6 +121,22 @@ Description: child buses, and re-discover devices removed earlier from this part of the device tree. +What: /sys/bus/pci/devices/.../reset_method +Date: March 2021 +Contact: Amey Narkhede <[email protected]> +Description: + Some devices allow an individual function to be reset + without affecting other functions in the same slot. + For devices that have this support, a file named reset_method + will be present in sysfs. Reading this file will give names + of the device supported reset methods and their ordering. + Writing the name or comma separated list of names of any of + the device supported reset methods to this file will set the + reset methods and their ordering to be used when resetting + the device. Writing empty string to this file will disable + ability to reset the device and writing "default" will return + to the original value. + What: /sys/bus/pci/devices/.../reset Date: July 2009 Contact: Michael S. Tsirkin <[email protected]>
drivers/pci/pci-sysfs.c +88 -3
113 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [PATCH RESEND v2 5/7] PCI/sysfs: Allow userspace to query and set device reset mechanism
Hi Amey,
[...]
drivers/pci/pci-sysfs.c static const struct bin_attribute pcie_config_attr = { · line 1354
Quoted 4 lines
drivers/pci/pci-sysfs.c +4
-0+if (sysfs_streq (buf,"" )) { +pci_warn (pdev,"All device reset methods disabled by user" ); + goto set_reset_methods; + }
The sysfs_streq() is nice, indeed.
drivers/pci/pci-sysfs.c static const struct bin_attribute pcie_config_attr = { · line 1362
Quoted 1 line
drivers/pci/pci-sysfs.c +1
-0+while ((name =strsep ((char **)&buf,"," )) !=NULL ) {
I believe we could make this parsing a little bit more resilient, especially since we are handling user input and this cannot ever be really fully trusted, so for example:
while ((name = strsep((char **)&buf, ","))) {
if !(strlen(name)) <--- sysfs_streq() could be used here too.
continue;
name = strim(name); <--- remove leading and trailing whitespaces, if any.
(...)[...]
drivers/pci/pci-sysfs.c static const struct bin_attribute pcie_config_attr = { · line 1376
Quoted 3 lines
drivers/pci/pci-sysfs.c +3
-0+if (reset_methods[0 ] && + reset_methods[0 ] !=PCI_RESET_FN_METHODS ) +pci_warn (pdev,"Device specific reset disabled/de-prioritized by user" );
What would be difference between disabling and de-prioritizing, is there be a way for us to distinguish between the two? I was wondering if we could, notify the user when the device specific reset is disable or when it has been de-prioritized?
Krzysztof
PATCH RESEND v2 6/7 PCI: Add support for a function level reset based on _RST method
In reply to Amey NarkhedeFrom: Shanker Donthineni <sdonthineni () nvidia ! com>
The _RST is a standard method specified in the ACPI specification. It provides a function level reset when it is described in the acpi_device context associated with PCI-device.
Implement a new reset function pci_dev_acpi_reset() for probing RST method and execute if it is defined in the firmware. The ACPI binding information is available only after calling device_add(), so move pci_init_reset_methods() to end of the pci_device_add().
The default priority of the acpi reset is set to below device-specific and above hardware resets.
--- drivers/pci/pci.c | 30 ++++++++++++++++++++++++++++++ drivers/pci/probe.c | 2 +- include/linux/pci.h | 2 +- 3 files changed, 32 insertions(+), 2 deletions(-)
drivers/pci/pci.c +30 -0
@@ -5076,6 +5076,35 @@ static void pci_dev_restore(struct pci_dev *dev) err_handler->reset_done (dev); } +/** +* pci_dev_acpi_reset - do a function level reset using _RST method +* @dev: device to reset +* @probe: check if _RST method is included in the acpi_device context. +*/ +static int pci_dev_acpi_reset (struct pci_dev *dev,int probe) +{ +#ifdef CONFIG_ACPI +acpi_handle handle =ACPI_HANDLE (&dev->dev); + +/* Return -ENOTTY if _RST method is not included in the dev context */ +if (!handle || !acpi_has_method (handle,"_RST" )) +return -ENOTTY ; + +/* Return 0 for probe phase indicating that we can reset this device */ +if (probe) +return 0 ; + +/* Invoke _RST() method to perform a function level reset */ +if (ACPI_FAILURE (acpi_evaluate_object (handle,"_RST" ,NULL ,NULL ))) { +pci_warn (dev,"Failed to reset the device\n" ); +return -EINVAL ; + } +return 0 ; +#else +return -ENOTTY ; +#endif +} +/* * The ordering for functions in pci_reset_fn_methods * is required for reset_methods byte array defined
@@ -5083,6 +5112,7 @@ static void pci_dev_restore(struct pci_dev *dev)*/ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { .reset_fn = &pci_dev_specific_reset, .name ="device_specific" }, + { .reset_fn = &pci_dev_acpi_reset, .name ="acpi" }, { .reset_fn = &pcie_reset_flr, .name ="flr" }, { .reset_fn = &pci_af_flr, .name ="af_flr" }, { .reset_fn = &pci_pm_reset, .name ="pm" },
drivers/pci/probe.c +1 -1
@@ -2403,7 +2403,6 @@ static void pci_init_capabilities(struct pci_dev *dev)pci_rcec_init (dev);/* Root Complex Event Collector */ pcie_report_downtraining (dev); -pci_init_reset_methods (dev); }/*
@@ -2494,6 +2493,7 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus) dev->match_driver = false; ret =device_add (&dev->dev);WARN_ON (ret <0 ); +pci_init_reset_methods (dev); }struct pci_dev *pci_scan_single_device (struct pci_bus *bus,int devfn)
include/linux/pci.h +1 -1
@@ -49,7 +49,7 @@ PCI_STATUS_SIG_TARGET_ABORT | \ PCI_STATUS_PARITY) -#define PCI_RESET_FN_METHODS 5 +#define PCI_RESET_FN_METHODS 6/* * The PCI interface treats multi-function devices as independent
Signature · 1 line
2.31.1
PATCH RESEND v2 7/7 PCI: Enable NO_BUS_RESET quirk for Nvidia GPUs
In reply to Amey NarkhedeFrom: Shanker Donthineni <sdonthineni () nvidia ! com>
On select platforms, some Nvidia GPU devices do not work with SBR. Triggering SBR would leave the device inoperable for the current system boot. It requires a system hard-reboot to get the GPU device back to normal operating condition post-SBR. For the affected devices, enable NO_BUS_RESET quirk to fix the issue.
This issue will be fixed in the next generation of hardware.
--- drivers/pci/quirks.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
drivers/pci/quirks.c +12 -0
@@ -3558,6 +3558,18 @@ static void quirk_no_bus_reset(struct pci_dev *dev) dev->dev_flags |=PCI_DEV_FLAGS_NO_BUS_RESET ; } +/* +* Some Nvidia GPU devices do not work with bus reset, SBR needs to be +* prevented for those affected devices. +*/ +static void quirk_nvidia_no_bus_reset (struct pci_dev *dev) +{ +if ((dev->device &0xffc0 ) ==0x2340 ) +quirk_no_bus_reset (dev); +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , + quirk_nvidia_no_bus_reset); +/* * Some Atheros AR9xxx and QCA988x chips do not behave after a bus reset. * The device will throw a Link Down error on AER-capable systems and
Signature · 1 line
2.31.1