PCI: Reduce FLR delay to 10ms for MSFT devices
v3 cover PCI: Reduce FLR delay to 10ms for MSFT devices
From: Graham Whyte <[email protected]>
Add a new flr_delay member of the pci_dev struct to allow customization of the delay after FLR for devices that do not support immediate readiness or readiness time reporting. The main scenario this addresses is VF removal and rescan during runtime repairs and driver updates, which, if fixed to 100ms, introduces significant delays across multiple VFs. These delays are unnecessary for devices that complete the FLR well within this timeframe.
Patch 1 adds the flr_delay member to the pci_dev struct Patch 2 adds the msft device specific quirk to utilize the flr_delay
---v2->v3:
- Removed Microsoft specific pcie reset reset, replaced with customizable flr_delay parameter
- Changed msleep in pcie_flr to usleep_range to support flr delays of under 20msv1->v2:
- Removed unnecessary EXPORT_SYMBOL_GPL for function pci_dev_wait
- Link to thread:https://lore.kernel.org/linux-pci/?q=f%3Agrwhyte&x=t#m7453647902a1b22840f5e39434a631fd7b2515ce'
Link to V1: https://lore.kernel.org/linux-pci/20250522085253.GN7435@unreal/T/#m7453647902a1b22840f5e39434a631fd7b2515ce
Graham Whyte (2):
PCI: Add flr_delay parameter to pci_dev struct
PCI: Reduce FLR delay to 10ms for MSFT devices
drivers/pci/pci.c | 8 ++++++-- drivers/pci/pci.h | 2 ++ drivers/pci/quirks.c | 20 ++++++++++++++++++++ include/linux/pci.h | 1 + 4 files changed, 29 insertions(+), 2 deletions(-)
Signature · 1 line
2.25.1
PATCH v3 1/2 PCI: Add flr_delay parameter to pci_dev struct
From: Graham Whyte <[email protected]>
Add a new flr_delay member of the pci_dev struct to allow customization of the delay after FLR for devices that do not support immediate readiness.
--- drivers/pci/pci.c | 8 ++++++-- drivers/pci/pci.h | 2 ++ include/linux/pci.h | 1 + 3 files changed, 9 insertions(+), 2 deletions(-)
drivers/pci/pci.c +6 -2
@@ -3233,6 +3233,8 @@ void pci_pm_init(struct pci_dev *dev) dev->bridge_d3 =pci_bridge_d3_possible (dev); dev->d3cold_allowed = true; + dev->flr_delay =PCI_FLR_DELAY ; + dev->d1_support = false; dev->d2_support = false;if (!pci_no_d1d2 (dev)) {
@@ -4529,9 +4531,11 @@ int pcie_flr(struct pci_dev *dev)/* * Per PCIe r4.0, sec 6.6.2, a device must complete an FLR within * 100ms, but may silently discard requests while the FLR is in -* progress. Wait 100ms before trying to access the device. +* progress. Wait 100ms before trying to access the device, unless +* otherwise modified if the device supports a faster reset. +* Use usleep_range to support delays under 20ms. */ -msleep (100 ); +usleep_range (dev->flr_delay, dev->flr_delay+1 );return pci_dev_wait (dev,"FLR" ,PCIE_RESET_READY_POLL_MS ); }
drivers/pci/pci.h +2 -0
@@ -135,6 +135,8 @@ struct pci_cap_saved_state *pci_find_saved_ext_cap(struct pci_dev *dev,#define PCI_PM_D3HOT_WAIT 10/* msec */ #define PCI_PM_D3COLD_WAIT 100/* msec */ +#define PCI_FLR_DELAY 100000/* usec */ +void pci_update_current_state (struct pci_dev *dev,pci_power_t state);void pci_refresh_power_state (struct pci_dev *dev);int pci_power_up (struct pci_dev *dev);
include/linux/pci.h +1 -0
@@ -402,6 +402,7 @@ struct pci_dev {bit manually */ unsigned int d3hot_delay;/* D3hot->D0 transition time in ms */ unsigned int d3cold_delay;/* D3cold->D0 transition time in ms */ +unsigned int flr_delay;/* pci post flr sleep time in us */ u16 l1ss;/* L1SS Capability pointer */ #ifdef CONFIG_PCIEASPM
Signature · 1 line
2.25.1
PATCH v3 2/2 PCI: Reduce FLR delay to 10ms for MSFT devices
In reply to [email protected]From: Graham Whyte <[email protected]>
Add a new quirk to reduce the delay after a FLR to 10ms
for MSFT devices. These devices complete the FLR well within the default 100ms timeframe and this path can be optimized for VF removal during runtime repairs and driver updates. These devices do not support immediate readiness or readiness time reporting
--- drivers/pci/quirks.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
drivers/pci/quirks.c +20 -0
@@ -6335,3 +6335,23 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_GLI ,0x9750 , pci_mask_replay_timer_timeout);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_GLI ,0x9755 , pci_mask_replay_timer_timeout);#endif + +#define MICROSOFT_2051_SVC 0xb210 +#define MICROSOFT_2051_MANA_MGMT 0x00b8 +#define MICROSOFT_2051_MANA_MGMT_GFT 0xb290 + +/* +* For devices that don't require the full 100ms sleep +* after FLR and do not support immediate readiness or readiness +* time reporting +*/ +static void pci_fixup_pci_flr_10msec (struct pci_dev *pdev) +{ + pdev->flr_delay =10000 ; +} +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_MICROSOFT ,MICROSOFT_2051_SVC , + pci_fixup_pci_flr_10msec); +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_MICROSOFT ,MICROSOFT_2051_MANA_MGMT , + pci_fixup_pci_flr_10msec); +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_MICROSOFT ,MICROSOFT_2051_MANA_MGMT_GFT , + pci_fixup_pci_flr_10msec);
Signature · 1 line
2.25.1
Quoted 9 lines · On Wed, Jun 11, 2025 at 12:05:50AM +0000, [email protected] wrote:
From: Graham Whyte <[email protected]>
Add a new
flr_delaymember of thepci_devstruct to allow customization of the delay after FLR for devices that do not support immediate readiness or readiness time reporting. The main scenario this addresses is VF removal and rescan during runtime repairs and driver updates, which, if fixed to 100ms, introduces significant delays across multiple VFs. These delays are unnecessary for devices that complete the FLR well within this timeframe.
Please work with the PCIe SIG to have a standard capability for this instead of piling up hacks like this quirk.
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · On Wed, Jun 11, 2025 at 12:05:50AM +0000, [email protected] wrote:
From: Graham Whyte <[email protected]>
Add a new
flr_delaymember of thepci_devstruct to allow customization of the delay after FLR for devices that do not support immediate readiness or readiness time reporting. The main scenario this addresses is VF removal and rescan during runtime repairs and driver updates, which, if fixed to 100ms, introduces significant delays across multiple VFs. These delays are unnecessary for devices that complete the FLR well within this timeframe.
Quoted 2 lines
Please work with the PCIe SIG to have a standard capability for this instead of piling up hacks like this quirk.
There is already some support in PCIe for this.
For Conventional Reset, see PCIe 6.0, section 6.6.1, there is the "Readiness Time Reporting Extended Capability":
"For a Device that implements the Readiness Time Reporting Extended Capability, and has reported a Reset Time shorter than 100ms, software is permitted to send a Configuration Request to the Device after waiting the reported Reset Time from Conventional Reset."
For FLR, see PCIe 6.0, section 6.6.2, there is the "Function Readiness Status":
"After an FLR has been initiated by writing a 1b to the Initiate Function Level Reset bit, the Function must complete the FLR within 100 ms. [...] If Function Readiness Status (FRS - see § Section 6.22.2 ) is implemented, then system software is permitted to issue Configuration Requests to the Function immediately following receipt of an FRS Message indicating Configuration-Ready, however, this does not necessarily indicate that outstanding Requests initiated by the Function have completed."
Kind regards,
Niklas
Quoted 35 lines · Niklas Cassel
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · On Wed, Jun 11, 2025 at 12:05:50AM +0000, [email protected] wrote:
From: Graham Whyte <[email protected]>
Add a new
flr_delaymember of thepci_devstruct to allow customization of the delay after FLR for devices that do not support immediate readiness or readiness time reporting. The main scenario this addresses is VF removal and rescan during runtime repairs and driver updates, which, if fixed to 100ms, introduces significant delays across multiple VFs. These delays are unnecessary for devices that complete the FLR well within this timeframe.Quoted 2 lines
Please work with the PCIe SIG to have a standard capability for this instead of piling up hacks like this quirk.
There is already some support in PCIe for this.
For Conventional Reset, see PCIe 6.0, section 6.6.1, there is the "Readiness Time Reporting Extended Capability":
"For a Device that implements the Readiness Time Reporting Extended Capability, and has reported a Reset Time shorter than 100ms, software is permitted to send a Configuration Request to the Device after waiting the reported Reset Time from Conventional Reset."For FLR, see PCIe 6.0, section 6.6.2, there is the "Function Readiness Status":
"After an FLR has been initiated by writing a 1b to the Initiate Function Level Reset bit, the Function must complete the FLR within 100 ms. [...] If Function Readiness Status (FRS - see § Section 6.22.2 ) is implemented, then system software is permitted to issue Configuration Requests to the Function immediately following receipt of an FRS Message indicating Configuration-Ready, however, this does not necessarily indicate that outstanding Requests initiated by the Function have completed."
Quoted 2 lines
Kind regards,
Niklas
We can ask our HW engineers to implement function readiness but we need to be able to support exiting products, hence why posting it as a quirk.
Quoted 2 lines · Graham Whyte
We can ask our HW engineers to implement function readiness but we need to be able to support exiting products, hence why posting it as a quirk.
Your report sounds like it works perfectly fine, it's just that you want to reduce the delay. For that you'll need to stick to the standard methods instead of adding quirks, which are for buggy hardware that does not otherwise work.
Quoted 8 lines · Christoph Hellwig
Quoted 2 lines · Graham Whyte
We can ask our HW engineers to implement function readiness but we need to be able to support exiting products, hence why posting it as a quirk.
Your report sounds like it works perfectly fine, it's just that you want to reduce the delay. For that you'll need to stick to the standard methods instead of adding quirks, which are for buggy hardware that does not otherwise work.
Bjorn, what would you recommend as next steps here?
Re: [PATCH v3 1/2] PCI: Add flr_delay parameter to pci_dev struct
In reply to [email protected]Quoted 26 lines · On Wed, 11 Jun 2025, [email protected] wrote:
From: Graham Whyte <[email protected]>
Add a new
flr_delaymember of thepci_devstruct to allow customization of the delay after FLR for devices that do not support immediate readiness.Signed-off-by Graham Whyte <[email protected]>--- drivers/pci/pci.c | 8 ++++++-- drivers/pci/pci.h | 2 ++ include/linux/pci.h | 1 + 3 files changed, 9 insertions(+), 2 deletions(-)drivers/pci/pci.c +2
-0@@ -3233,6 +3233,8 @@ void pci_pm_init(struct pci_dev *dev) dev->bridge_d3 =pci_bridge_d3_possible (dev); dev->d3cold_allowed = true; + dev->flr_delay =PCI_FLR_DELAY ; + dev->d1_support = false; dev->d2_support = false;if (!pci_no_d1d2 (dev)) {
drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4538
Quoted 10 lines
drivers/pci/pci.c +4
-2@@ -4529,9 +4531,11 @@ int pcie_flr(struct pci_dev *dev)/* * Per PCIe r4.0, sec 6.6.2, a device must complete an FLR within * 100ms, but may silently discard requests while the FLR is in -* progress. Wait 100ms before trying to access the device. +* progress. Wait 100ms before trying to access the device, unless +* otherwise modified if the device supports a faster reset. +* Use usleep_range to support delays under 20ms. */ -msleep (100 ); +usleep_range (dev->flr_delay, dev->flr_delay+1 );
Missing spaces around +.
Are you sure + 1us is really useful as the range? Usually much bigger numbers are used.
There's also fsleep() which would autoselect the sleep mechanism.
drivers/pci/pci.h struct pci_cap_saved_state *pci_find_saved_ext_cap(struct pci_dev *dev, · line 138
Quoted 11 lines
drivers/pci/pci.c +0
-0return pci_dev_wait (dev,"FLR" ,PCIE_RESET_READY_POLL_MS ); }drivers/pci/pci.h +1
-0@@ -135,6 +135,8 @@ struct pci_cap_saved_state *pci_find_saved_ext_cap(struct pci_dev *dev,#define PCI_PM_D3HOT_WAIT 10/* msec */ #define PCI_PM_D3COLD_WAIT 100/* msec */ +#define PCI_FLR_DELAY 100000/* usec */
Please put the unit into the define name (_US).
include/linux/pci.h struct pci_dev { · line 405
Quoted 13 lines
drivers/pci/pci.h +1
-0+void pci_update_current_state (struct pci_dev *dev,pci_power_t state);void pci_refresh_power_state (struct pci_dev *dev);int pci_power_up (struct pci_dev *dev);include/linux/pci.h +1
-0@@ -402,6 +402,7 @@ struct pci_dev {bit manually */ unsigned int d3hot_delay;/* D3hot->D0 transition time in ms */ unsigned int d3cold_delay;/* D3cold->D0 transition time in ms */ +unsigned int flr_delay;/* pci post flr sleep time in us */
Please follow how the spec writes things in capitalization of letters.
Quoted 4 lines
include/linux/pci.h +0
-0u16 l1ss;/* L1SS Capability pointer */ #ifdef CONFIG_PCIEASPM
Signature · 1 line
i.
Quoted 5 lines · On Wed, 11 Jun 2025, [email protected] wrote:
From: Graham Whyte <[email protected]>
Add a new quirk to reduce the delay after a FLR to 10ms
for MSFT devices. These devices complete the FLR well within the default 100ms timeframe and this path can be optimized for VF removal during
What is "this path" in this context? Try to avoid vague references like that.
Quoted 2 lines
runtime repairs and driver updates. These devices do not support immediate readiness or readiness time reporting
When talking about something that relates to PCIe spec, please also refer to PCIe spec and use the terminology matching to the spec (+ capitalization).
Missing .
Please also reflow the paragraph as the first line is clearly not full.
This probably belongs more to the previous patch changelog than this one, as the justification: I suggest you start by stating the problem. So state the spec defined wait (+ spec reference), and why that is problem. Then explain the solution in another paragraph.
Quoted 16 lines
Signed-off-by Graham Whyte <[email protected]>--- drivers/pci/quirks.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)drivers/pci/quirks.c +3
-0@@ -6335,3 +6335,23 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_GLI ,0x9750 , pci_mask_replay_timer_timeout);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_GLI ,0x9755 , pci_mask_replay_timer_timeout);#endif + +#define MICROSOFT_2051_SVC 0xb210 +#define MICROSOFT_2051_MANA_MGMT 0x00b8
drivers/pci/quirks.c static void pci_mask_replay_timer_timeout(struct pci_dev *pdev) · line 6350
Quoted 10 lines
drivers/pci/quirks.c +10
-0+#define MICROSOFT_2051_MANA_MGMT_GFT 0xb290 + +/* +* For devices that don't require the full 100ms sleep +* after FLR and do not support immediate readiness or readiness +* time reporting +*/ +static void pci_fixup_pci_flr_10msec (struct pci_dev *pdev) +{ + pdev->flr_delay =10000 ;
10 * USEC_PER_MSEC
Quoted 8 lines
drivers/pci/quirks.c +7
-0+} +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_MICROSOFT ,MICROSOFT_2051_SVC , + pci_fixup_pci_flr_10msec); +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_MICROSOFT ,MICROSOFT_2051_MANA_MGMT , + pci_fixup_pci_flr_10msec); +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_MICROSOFT ,MICROSOFT_2051_MANA_MGMT_GFT , + pci_fixup_pci_flr_10msec);
Signature · 1 line
i.
Quoted 10 lines · On Wed, Jun 11, 2025 at 12:05:50AM +0000, [email protected] wrote:
From: Graham Whyte <[email protected]>
Add a new
flr_delaymember of thepci_devstruct to allow customization of the delay after FLR for devices that do not support immediate readiness or readiness time reporting. The main scenario this addresses is VF removal and rescan during runtime repairs and driver updates, which, if fixed to 100ms, introduces significant delays across multiple VFs. These delays are unnecessary for devices that complete the FLR well within this timeframe.
I don't think it is acceptable to *reduce* the standard delay just because your device completes it more quickly. Proper way to reduce the timing would be to support FRS as you said, but we cannot have arbitrary delays for random devices.
- Mani
Quoted 27 lines
Patch 1 adds the
flr_delaymember to thepci_devstruct Patch 2 adds the msft device specific quirk to utilize theflr_delay---
v2->v3:
- Removed Microsoft specific pcie reset reset, replaced with customizableflr_delayparameter
- Changed msleep inpcie_flrtousleep_rangeto support flr delays of under 20msv1->v2:
- Removed unnecessaryEXPORT_SYMBOL_GPLfor functionpci_dev_wait
- Link to thread:https://lore.kernel.org/linux-pci/?q=f%3Agrwhyte&x=t#m7453647902a1b22840f5e39434a631fd7b2515ce'Link to V1: https://lore.kernel.org/linux-pci/20250522085253.GN7435@unreal/T/#m7453647902a1b22840f5e39434a631fd7b2515ce
Graham Whyte (2):
PCI: Addflr_delayparameter topci_devstruct
PCI: Reduce FLR delay to 10ms for MSFT devicesdrivers/pci/pci.c | 8 ++++++-- drivers/pci/pci.h | 2 ++ drivers/pci/quirks.c | 20 ++++++++++++++++++++ include/linux/pci.h | 1 + 4 files changed, 29 insertions(+), 2 deletions(-)Signature · 1 line
2.25.1
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 10 lines · Manivannan Sadhasivam
Quoted 8 lines · On Wed, Jun 11, 2025 at 12:05:50AM +0000, [email protected] wrote:
Add a new
flr_delaymember of thepci_devstruct to allow customization of the delay after FLR for devices that do not support immediate readiness or readiness time reporting. The main scenario this addresses is VF removal and rescan during runtime repairs and driver updates, which, if fixed to 100ms, introduces significant delays across multiple VFs. These delays are unnecessary for devices that complete the FLR well within this timeframe.
Quoted 4 lines
I don't think it is acceptable to *reduce* the standard delay just because your device completes it more quickly. Proper way to reduce the timing would be to support FRS as you said, but we cannot have arbitrary delays for random devices.
To be fair, we already have that for certain devices:
The quirk delay_250ms_after_flr() is referenced by three different Vendor ID / Device ID combos and *lengthens* the delay after FLR.
It's probably difficult to justify rejecting custom delays for certain MANA devices, even though we allowed them for three other devices.
The proposed patch introduces a generic solution which avoids further cluttering up pci_dev_reset_methods[] with extra entries, so I think it's an approach worth considering.
There are a bunch of nits in the proposed patches, such as "pci" not being capitalized, but the general approach seems fine to me.
Thanks,
Lukas
Quoted 16 lines · Lukas Wunner
Quoted 10 lines · Manivannan Sadhasivam
Quoted 8 lines · On Wed, Jun 11, 2025 at 12:05:50AM +0000, [email protected] wrote:
Add a new
flr_delaymember of thepci_devstruct to allow customization of the delay after FLR for devices that do not support immediate readiness or readiness time reporting. The main scenario this addresses is VF removal and rescan during runtime repairs and driver updates, which, if fixed to 100ms, introduces significant delays across multiple VFs. These delays are unnecessary for devices that complete the FLR well within this timeframe.Quoted 4 lines
I don't think it is acceptable to *reduce* the standard delay just because your device completes it more quickly. Proper way to reduce the timing would be to support FRS as you said, but we cannot have arbitrary delays for random devices.
Quoted 5 lines
To be fair, we already have that for certain devices:
The quirk
delay_250ms_after_flr()is referenced by three different Vendor ID / Device ID combos and *lengthens* the delay after FLR.
This quirk is fine as it works around an issue in the device. But this patch is not fixing/working around an issue in the device, but rather optimizing the delay for performance, which is not what quirks are used for AFAIK.
Quoted 4 lines
It's probably difficult to justify rejecting custom delays for certain MANA devices, even though we allowed them for three other devices.
If the MANA devices require extended delay, then a quirk indeed makes sense. But it is the other way around.
Quoted 7 lines
The proposed patch introduces a generic solution which avoids further cluttering up pci_dev_reset_methods[] with extra entries, so I think it's an approach worth considering.
There are a bunch of nits in the proposed patches, such as "pci" not being capitalized, but the general approach seems fine to me.
I honestly don't know if there is any other way to handle this. So I think it is upto Bjorn to take a call.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 11 lines · Graham Whyte
Quoted 8 lines · Christoph Hellwig
Quoted 2 lines · Graham Whyte
We can ask our HW engineers to implement function readiness but we need to be able to support exiting products, hence why posting it as a quirk.
Your report sounds like it works perfectly fine, it's just that you want to reduce the delay. For that you'll need to stick to the standard methods instead of adding quirks, which are for buggy hardware that does not otherwise work.
Bjorn, what would you recommend as next steps here?
This is a tough call and I don't pretend to have an obvious answer. I understand the desire to improve performance. On the other hand, PCI has been successful over the long term because devices adhere to standardized ways of doing things, which makes generic software possible. Quirks degrade that story, of course, especially when there is an existing standardized solution that isn't being used. I'm not at all happy about vendors that decide against the standard solution and then ask OS folks to do extra work to compensate.
Bjorn
Quoted 22 lines · Bjorn Helgaas
Quoted 11 lines · Graham Whyte
Quoted 8 lines · Christoph Hellwig
Quoted 2 lines · Graham Whyte
We can ask our HW engineers to implement function readiness but we need to be able to support exiting products, hence why posting it as a quirk.
Your report sounds like it works perfectly fine, it's just that you want to reduce the delay. For that you'll need to stick to the standard methods instead of adding quirks, which are for buggy hardware that does not otherwise work.
Bjorn, what would you recommend as next steps here?
This is a tough call and I don't pretend to have an obvious answer. I understand the desire to improve performance. On the other hand, PCI has been successful over the long term because devices adhere to standardized ways of doing things, which makes generic software possible. Quirks degrade that story, of course, especially when there is an existing standardized solution that isn't being used. I'm not at all happy about vendors that decide against the standard solution and then ask OS folks to do extra work to compensate.
Quoted 1 line
Bjorn
Hi Bjorn,
Should someone want to implement readiness time reporting down the road, they'll need to do the same work as patch 1 in this series (making the flr delay a configurable parameter). This change lays the groundwork for that work, while also supporting devices that can't use readiness time reporting.
Alternatively could we use a sysfs file to make this parameter configurable via the user space application? Similar to switching between d3hot and d3cold by writing to /sys/bus/pci/slots/$DEVICE/power.
Thanks,
Graham
Quoted 23 lines · Graham Whyte
Quoted 13 lines · Bjorn Helgaas
Quoted 11 lines · Graham Whyte
Quoted 8 lines · Christoph Hellwig
Quoted 2 lines · Graham Whyte
We can ask our HW engineers to implement function readiness but we need to be able to support exiting products, hence why posting it as a quirk.
Your report sounds like it works perfectly fine, it's just that you want to reduce the delay. For that you'll need to stick to the standard methods instead of adding quirks, which are for buggy hardware that does not otherwise work.
Bjorn, what would you recommend as next steps here?
Quoted 8 lines
This is a tough call and I don't pretend to have an obvious answer. I understand the desire to improve performance. On the other hand, PCI has been successful over the long term because devices adhere to standardized ways of doing things, which makes generic software possible. Quirks degrade that story, of course, especially when there is an existing standardized solution that isn't being used. I'm not at all happy about vendors that decide against the standard solution and then ask OS folks to do extra work to compensate.
Quoted 3 lines
Should someone want to implement readiness time reporting down the road, they'll need to do the same work as patch 1 in this series (making the flr delay a configurable parameter).
Sure. That's a trivial change. The problem is the quirk itself.
The Readiness Time Reporting Extended Capability is read-only with no control bits in it so it requires no actual logic in the device. Maybe you can just implement that capability with a firmware change on the device and add the corresponding Linux support for it.
Quoted 28 lines · Bjorn Helgaas
Quoted 23 lines · Graham Whyte
Quoted 13 lines · Bjorn Helgaas
Quoted 11 lines · Graham Whyte
Quoted 8 lines · Christoph Hellwig
Quoted 2 lines · Graham Whyte
We can ask our HW engineers to implement function readiness but we need to be able to support exiting products, hence why posting it as a quirk.
Your report sounds like it works perfectly fine, it's just that you want to reduce the delay. For that you'll need to stick to the standard methods instead of adding quirks, which are for buggy hardware that does not otherwise work.
Bjorn, what would you recommend as next steps here?
Quoted 8 lines
This is a tough call and I don't pretend to have an obvious answer. I understand the desire to improve performance. On the other hand, PCI has been successful over the long term because devices adhere to standardized ways of doing things, which makes generic software possible. Quirks degrade that story, of course, especially when there is an existing standardized solution that isn't being used. I'm not at all happy about vendors that decide against the standard solution and then ask OS folks to do extra work to compensate.
Quoted 3 lines
Should someone want to implement readiness time reporting down the road, they'll need to do the same work as patch 1 in this series (making the flr delay a configurable parameter).
Quoted 6 lines
Sure. That's a trivial change. The problem is the quirk itself.
The Readiness Time Reporting Extended Capability is read-only with no control bits in it so it requires no actual logic in the device. Maybe you can just implement that capability with a firmware change on the device and add the corresponding Linux support for it.
Hi Bjorn,
We checked with our HW folks, it's not possible for us to update the pci register components with this particular card, they are read only. What are your thoughts on the sysfs approach mentioned in the previous email?
Thanks,
Graham
Quoted 38 lines · Graham Whyte
Quoted 28 lines · Bjorn Helgaas
Quoted 23 lines · Graham Whyte
Quoted 13 lines · Bjorn Helgaas
Quoted 11 lines · Graham Whyte
Quoted 8 lines · Christoph Hellwig
Quoted 2 lines · Graham Whyte
We can ask our HW engineers to implement function readiness but we need to be able to support exiting products, hence why posting it as a quirk.
Your report sounds like it works perfectly fine, it's just that you want to reduce the delay. For that you'll need to stick to the standard methods instead of adding quirks, which are for buggy hardware that does not otherwise work.
Bjorn, what would you recommend as next steps here?
Quoted 8 lines
This is a tough call and I don't pretend to have an obvious answer. I understand the desire to improve performance. On the other hand, PCI has been successful over the long term because devices adhere to standardized ways of doing things, which makes generic software possible. Quirks degrade that story, of course, especially when there is an existing standardized solution that isn't being used. I'm not at all happy about vendors that decide against the standard solution and then ask OS folks to do extra work to compensate.
Quoted 3 lines
Should someone want to implement readiness time reporting down the road, they'll need to do the same work as patch 1 in this series (making the flr delay a configurable parameter).
Quoted 6 lines
Sure. That's a trivial change. The problem is the quirk itself.
The Readiness Time Reporting Extended Capability is read-only with no control bits in it so it requires no actual logic in the device. Maybe you can just implement that capability with a firmware change on the device and add the corresponding Linux support for it.
Quoted 8 lines
Hi Bjorn,
We checked with our HW folks, it's not possible for us to update the pci register components with this particular card, they are read only. What are your thoughts on the sysfs approach mentioned in the previous email?
Thanks,
Graham
Hi Bjorn, just wanted to follow up on this here.
Re: [PATCH v3 1/2] PCI: Add flr_delay parameter to pci_dev struct
In reply to [email protected]Quoted 17 lines · On Wed, Jun 11, 2025 at 12:05:51AM +0000, [email protected] wrote:
From: Graham Whyte <[email protected]>
Add a new
flr_delaymember of thepci_devstruct to allow customization of the delay after FLR for devices that do not support immediate readiness.Signed-off-by Graham Whyte <[email protected]>--- drivers/pci/pci.c | 8 ++++++-- drivers/pci/pci.h | 2 ++ include/linux/pci.h | 1 + 3 files changed, 9 insertions(+), 2 deletions(-)drivers/pci/pci.c +0
-0
drivers/pci/pci.c void pci_pm_init(struct pci_dev *dev) · line 3236
Quoted 4 lines
drivers/pci/pci.c +1
-0@@ -3233,6 +3233,8 @@ void pci_pm_init(struct pci_dev *dev) dev->bridge_d3 =pci_bridge_d3_possible (dev); dev->d3cold_allowed = true; + dev->flr_delay =PCI_FLR_DELAY ;
There are some delays mentioned in pci_pm_init(), but I don't think this one has anything to do with the PM Capability, so it should go elsewhere. Maybe somewhere related to this recent change:
https://git.kernel.org/linus/5c0d0ee36f16 ("PCI: Support Immediate Readiness on devices without PM cap abilities").
This would be more attractive if we actually added support for the Readiness Time Reporting Capability (PCIe r7.0, sec 7.9.16). Then devices that implement that would get actual benefit from this.
I'm not committing to merging a quirk just for the Microsoft devices, but I definitely wouldn't merge it unless devices that did the work of supporting the standard mechanism also got the benefit. The hardest part might be *finding* a device that supports Readiness Time Reporting so we could test it.
Quoted 4 lines
drivers/pci/pci.c +0
-0dev->d1_support = false; dev->d2_support = false;if (!pci_no_d1d2 (dev)) {
drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 4538
Quoted 10 lines
drivers/pci/pci.c +4
-2@@ -4529,9 +4531,11 @@ int pcie_flr(struct pci_dev *dev)/* * Per PCIe r4.0, sec 6.6.2, a device must complete an FLR within * 100ms, but may silently discard requests while the FLR is in -* progress. Wait 100ms before trying to access the device. +* progress. Wait 100ms before trying to access the device, unless +* otherwise modified if the device supports a faster reset. +* Use usleep_range to support delays under 20ms. */ -msleep (100 ); +usleep_range (dev->flr_delay, dev->flr_delay+1 );
As Ilpo suggested, I think fsleep() is the right thing for this. Readiness Time Reporting supports values down to 1ns, but I suspect we can live with microsecond resolution for now.
drivers/pci/pci.h struct pci_cap_saved_state *pci_find_saved_ext_cap(struct pci_dev *dev, · line 138
Quoted 11 lines
drivers/pci/pci.c +0
-0return pci_dev_wait (dev,"FLR" ,PCIE_RESET_READY_POLL_MS ); }drivers/pci/pci.h +1
-0@@ -135,6 +135,8 @@ struct pci_cap_saved_state *pci_find_saved_ext_cap(struct pci_dev *dev,#define PCI_PM_D3HOT_WAIT 10/* msec */ #define PCI_PM_D3COLD_WAIT 100/* msec */ +#define PCI_FLR_DELAY 100000/* usec */
Isn't PCIE_RESET_CONFIG_WAIT_MS the right value for the default delay? (I think that name was probably settled on after you posted this.)
Maybe it's not; I see that Readiness Time Reporting includes separate values for Reset Time, DL_Up Time, FLR Time, and D3Hot to D0 Time. I'm not sure Linux comprehends those differences yet.
include/linux/pci.h struct pci_dev { · line 405
Quoted 12 lines
drivers/pci/pci.h +0
-0void pci_update_current_state (struct pci_dev *dev,pci_power_t state);void pci_refresh_power_state (struct pci_dev *dev);int pci_power_up (struct pci_dev *dev);include/linux/pci.h +1
-0@@ -402,6 +402,7 @@ struct pci_dev {bit manually */ unsigned int d3hot_delay;/* D3hot->D0 transition time in ms */ unsigned int d3cold_delay;/* D3cold->D0 transition time in ms */ +unsigned int flr_delay;/* pci post flr sleep time in us */
I think this should be named to correspond with the Readiness Time Reporting Capability.
Quoted 5 lines
include/linux/pci.h +0
-0u16 l1ss;/* L1SS Capability pointer */ #ifdef CONFIG_PCIEASPM Signature · 1 line
2.25.1