Skip to contentSkip to conversation

PCI/PM: Switch D3Hot delay to also use usleep_range

PATCH Superseded Will McVicker 3 participants 6 messages Aug 17, 2022 – Sep 7, 2022 PCI

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 PCI/PM: Switch D3Hot delay to also use usleep_range

From: Sajid Dalvi <[email protected]>

Since the PCI spec requires a 10ms D3Hot delay (defined by PCI_PM_D3HOT_WAIT) and a few of the PCI quirks update the D3Hot delay up to 120ms, let's add support for both usleep_range and msleep based on the delay time to improve the delay accuracy.

This patch is based off of a commit from Sajid Dalvi <[email protected]> in the Pixel 6 kernel tree [1]. Testing on a Pixel 6, found that the 10ms delay for the Exynos PCIe device was on average delaying for 19ms when the spec requires 10ms. Switching from msleep to uslseep_range therefore decreases the resume time on a Pixel 6 on average by 9ms.

[1] https://android.googlesource.com/kernel/gs/+/18a8cad68d8e6d50f339a716a18295e6d987cee3

---
 drivers/pci/pci.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

v3:
* Use DIV_ROUND_CLOSEST instead of bit manipulation.
* Minor refactor to use max() where relavant.

v2:
* Update to use 20-25% upper bound
* Update to use usleep_range() for <=20ms, else use msleep()

drivers/pci/pci.c +11 -5
index 95bc329e74c0..cfa8386314f2 100644
@@ -66,13 +66,19 @@ struct pci_pme_device {
 
 static void pci_dev_d3_sleep(struct pci_dev *dev)
 {
-	unsigned int delay = dev->d3hot_delay;
+	unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
 
-	if (delay < pci_pm_d3hot_delay)
-		delay = pci_pm_d3hot_delay;
+	if (delay_ms) {
+		if (delay_ms <= 20) {
+			/* Use a 20% upper bound with 1ms minimum */
+			unsigned int upper = max(DIV_ROUND_CLOSEST(delay_ms, 5), 1U);
 
-	if (delay)
-		msleep(delay);
+			usleep_range(delay_ms * USEC_PER_MSEC,
+				     (delay_ms + upper) * USEC_PER_MSEC);
+		} else {
+			msleep(delay_ms);
+		}
+	}
 }
 
 bool pci_reset_supported(struct pci_dev *dev)

base-commit: 274a2eebf80c60246f9edd6ef8e9a095ad121264

Signature · 1 line
2.37.1.595.g718a3a8f04-goog
Reviewed-by on v3 1/1
#1 lore raw
Quoted 13 lines · Will McVicker

From: Sajid Dalvi <[email protected]>

Since the PCI spec requires a 10ms D3Hot delay (defined by PCI_PM_D3HOT_WAIT) and a few of the PCI quirks update the D3Hot delay up to 120ms, let's add support for both usleep_range and msleep based on the delay time to improve the delay accuracy.

This patch is based off of a commit from Sajid Dalvi <[email protected]> in the Pixel 6 kernel tree [1]. Testing on a Pixel 6, found that the 10ms delay for the Exynos PCIe device was on average delaying for 19ms when the spec requires 10ms. Switching from msleep to uslseep_range therefore decreases the resume time on a Pixel 6 on average by 9ms.

Quoted 4 lines

[1] https://android.googlesource.com/kernel/gs/+/18a8cad68d8e6d50f339a716a18295e6d987cee3

#2 lore raw
In reply to Will McVicker
Quoted 59 lines · Will McVicker

From: Sajid Dalvi <[email protected]>

Since the PCI spec requires a 10ms D3Hot delay (defined by PCI_PM_D3HOT_WAIT) and a few of the PCI quirks update the D3Hot delay up to 120ms, let's add support for both usleep_range and msleep based on the delay time to improve the delay accuracy.

This patch is based off of a commit from Sajid Dalvi <[email protected]> in the Pixel 6 kernel tree [1]. Testing on a Pixel 6, found that the 10ms delay for the Exynos PCIe device was on average delaying for 19ms when the spec requires 10ms. Switching from msleep to uslseep_range therefore decreases the resume time on a Pixel 6 on average by 9ms.

[1] https://android.googlesource.com/kernel/gs/+/18a8cad68d8e6d50f339a716a18295e6d987cee3

---
 drivers/pci/pci.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

v3:
* Use DIV_ROUND_CLOSEST instead of bit manipulation.
* Minor refactor to use max() where relavant.

v2:
* Update to use 20-25% upper bound
* Update to use usleep_range() for <=20ms, else use msleep()

drivers/pci/pci.c +11 -5
index 95bc329e74c0..cfa8386314f2 100644
@@ -66,13 +66,19 @@ struct pci_pme_device {
 
 static void pci_dev_d3_sleep(struct pci_dev *dev)
 {
-	unsigned int delay = dev->d3hot_delay;
+	unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
 
-	if (delay < pci_pm_d3hot_delay)
-		delay = pci_pm_d3hot_delay;
+	if (delay_ms) {
+		if (delay_ms <= 20) {
+			/* Use a 20% upper bound with 1ms minimum */
+			unsigned int upper = max(DIV_ROUND_CLOSEST(delay_ms, 5), 1U);
 
-	if (delay)
-		msleep(delay);
+			usleep_range(delay_ms * USEC_PER_MSEC,
+				     (delay_ms + upper) * USEC_PER_MSEC);
+		} else {
+			msleep(delay_ms);
+		}
+	}
 }
 
 bool pci_reset_supported(struct pci_dev *dev)
Quoted 4 lines

base-commit: 274a2eebf80c60246f9edd6ef8e9a095ad121264

Signature · 1 line
2.37.1.595.g718a3a8f04-goog

Hi Bjorn,

I'm just checking up on this patch in case it got lost in your inbox. Please take a look and let me know if you have any concerns.

And thanks again for the reviews Matthias!

Regards,
Will

#3 lore raw
In reply to Will McVicker
Quoted 7 lines · Will McVicker

From: Sajid Dalvi <[email protected]>

Since the PCI spec requires a 10ms D3Hot delay (defined by PCI_PM_D3HOT_WAIT) and a few of the PCI quirks update the D3Hot delay up to 120ms, let's add support for both usleep_range and msleep based on the delay time to improve the delay accuracy.

Quoted 5 lines

This patch is based off of a commit from Sajid Dalvi <[email protected]> in the Pixel 6 kernel tree [1]. Testing on a Pixel 6, found that the 10ms delay for the Exynos PCIe device was on average delaying for 19ms when the spec requires 10ms. Switching from msleep to uslseep_range therefore decreases the resume time on a Pixel 6 on average by 9ms.

Add the "PCIe r6.0, sec 5.9" spec reference for the 10ms delay for transitions to or from D3hot.

s/D3Hot/D3hot/ to match other usage (at least in Linux; the spec does use "D3Hot")

s/uslseep_range/usleep_range/

Add "()" after function names.

In the subject, "Switch ... to *also* use usleep_range": what does the "also" mean? It sounds like it's referring to some other place where we also use usleep_range()?

Quoted 30 lines

[1] https://android.googlesource.com/kernel/gs/+/18a8cad68d8e6d50f339a716a18295e6d987cee3

---
 drivers/pci/pci.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

v3:
* Use DIV_ROUND_CLOSEST instead of bit manipulation.
* Minor refactor to use max() where relavant.

v2:
* Update to use 20-25% upper bound
* Update to use usleep_range() for <=20ms, else use msleep()

drivers/pci/pci.c +2 -3
index 95bc329e74c0..cfa8386314f2 100644
@@ -66,13 +66,19 @@ struct pci_pme_device {
 
 static void pci_dev_d3_sleep(struct pci_dev *dev)
 {
-	unsigned int delay = dev->d3hot_delay;
+	unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
 
-	if (delay < pci_pm_d3hot_delay)
-		delay = pci_pm_d3hot_delay;
+	if (delay_ms) {

drivers/pci/pci.c struct pci_pme_device { · line 79

Quoted 10 lines
drivers/pci/pci.c +7 -2
+		if (delay_ms <= 20) {
+			/* Use a 20% upper bound with 1ms minimum */
+			unsigned int upper = max(DIV_ROUND_CLOSEST(delay_ms, 5), 1U);
 
-	if (delay)
-		msleep(delay);
+			usleep_range(delay_ms * USEC_PER_MSEC,
+				     (delay_ms + upper) * USEC_PER_MSEC);
+		} else {
+			msleep(delay_ms);

I hate the fact that we have to check for those ancient Intel chips at all, but having to read through the usleep_range() vs msleep() thing is just painful.

fsleep() would be much simpler, but I am sympathetic that the fsleep() range of 10-20ms probably wouldn't get the benefit you want.

I wish Documentation/timers/timers-howto.rst included a reason why we should use msleep() instead of usleep_range() for longer sleeps. Is there a reason not to do this:

   static void pci_dev_d3_sleep(struct pci_dev *dev)
   {
        unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
        unsigned int upper;
        if (delay_ms) {
                /* 20% upper bound, 1ms minimum */
                upper = max(DIV_ROUND_CLOSEST(delay_ms, 5), 1U)
                usleep_range(delay_ms * USEC_PER_MSEC,
                             (delay_ms + upper) * USEC_PER_MSEC);
        }

}

Since the Intel quirk is for 120ms, a 20% upper bound would make the range 120-144ms. Would that be a problem? Those chips are ancient; the list is untouched since it was added in 2006. The point of usleep_range() is to allow the scheduler to coalesce the wakeup with other events, so it seems unlikely we'd ever wait the whole 144ms. I vote for optimizing the readability over sleep/resume time for already-broken chips.

Bjorn

#4 lore raw
Quoted 17 lines · Bjorn Helgaas
Quoted 7 lines · Will McVicker

From: Sajid Dalvi <[email protected]>

Since the PCI spec requires a 10ms D3Hot delay (defined by PCI_PM_D3HOT_WAIT) and a few of the PCI quirks update the D3Hot delay up to 120ms, let's add support for both usleep_range and msleep based on the delay time to improve the delay accuracy.

Quoted 5 lines

This patch is based off of a commit from Sajid Dalvi <[email protected]> in the Pixel 6 kernel tree [1]. Testing on a Pixel 6, found that the 10ms delay for the Exynos PCIe device was on average delaying for 19ms when the spec requires 10ms. Switching from msleep to uslseep_range therefore decreases the resume time on a Pixel 6 on average by 9ms.

Add the "PCIe r6.0, sec 5.9" spec reference for the 10ms delay for transitions to or from D3hot.

Quoted 6 lines

s/D3Hot/D3hot/ to match other usage (at least in Linux; the spec does use "D3Hot")

s/uslseep_range/usleep_range/

Add "()" after function names.

Thanks for the suggestions! I'll update these in the next patchset.

Quoted 4 lines

In the subject, "Switch ... to *also* use usleep_range": what does the "also" mean? It sounds like it's referring to some other place where we also use usleep_range()?

I was intending to mean usleep_range() will be used when the the delay <=20 ms, else msleep() will be used. If we drop msleep() altogether as suggested in your comments below, then I can change this to be: "Switch D3hot delay to use usleep_range."

Quoted 46 lines
Quoted 31 lines

[1] https://android.googlesource.com/kernel/gs/+/18a8cad68d8e6d50f339a716a18295e6d987cee3

---
 drivers/pci/pci.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

v3:
* Use DIV_ROUND_CLOSEST instead of bit manipulation.
* Minor refactor to use max() where relavant.

v2:
* Update to use 20-25% upper bound
* Update to use usleep_range() for <=20ms, else use msleep()

drivers/pci/pci.c +2 -3
index 95bc329e74c0..cfa8386314f2 100644
@@ -66,13 +66,19 @@ struct pci_pme_device {
 
 static void pci_dev_d3_sleep(struct pci_dev *dev)
 {
-	unsigned int delay = dev->d3hot_delay;
+	unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
 
-	if (delay < pci_pm_d3hot_delay)
-		delay = pci_pm_d3hot_delay;
+	if (delay_ms) {

drivers/pci/pci.c struct pci_pme_device { · line 79

Quoted 10 lines
drivers/pci/pci.c +7 -2
+		if (delay_ms <= 20) {
+			/* Use a 20% upper bound with 1ms minimum */
+			unsigned int upper = max(DIV_ROUND_CLOSEST(delay_ms, 5), 1U);
 
-	if (delay)
-		msleep(delay);
+			usleep_range(delay_ms * USEC_PER_MSEC,
+				     (delay_ms + upper) * USEC_PER_MSEC);
+		} else {
+			msleep(delay_ms);

I hate the fact that we have to check for those ancient Intel chips at all, but having to read through the usleep_range() vs msleep() thing is just painful.

Quoted 6 lines

fsleep() would be much simpler, but I am sympathetic that the fsleep() range of 10-20ms probably wouldn't get the benefit you want.

I wish Documentation/timers/timers-howto.rst included a reason why we should use msleep() instead of usleep_range() for longer sleeps. Is there a reason not to do this:

I'm not familiar with the reasons behind these two beside what the documentation states. I don't know what happens when you use usleep_range() with a delay >20ms.

Quoted 14 lines
   static void pci_dev_d3_sleep(struct pci_dev *dev)
   {
        unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
        unsigned int upper;
        if (delay_ms) {
                /* 20% upper bound, 1ms minimum */
                upper = max(DIV_ROUND_CLOSEST(delay_ms, 5), 1U)
                usleep_range(delay_ms * USEC_PER_MSEC,
                             (delay_ms + upper) * USEC_PER_MSEC);
        }

}

Quoted 7 lines

Since the Intel quirk is for 120ms, a 20% upper bound would make the range 120-144ms. Would that be a problem? Those chips are ancient; the list is untouched since it was added in 2006. The point of usleep_range() is to allow the scheduler to coalesce the wakeup with other events, so it seems unlikely we'd ever wait the whole 144ms. I vote for optimizing the readability over sleep/resume time for already-broken chips.

I'm totally fine with this, but I don't really know what the impact would be to those old Intel chips.

Quoted 2 lines

Bjorn

Thanks,
Will

#5 lore raw
Quoted 23 lines · William McVicker
Quoted 13 lines · Bjorn Helgaas
   static void pci_dev_d3_sleep(struct pci_dev *dev)
   {
        unsigned int delay_ms = max(dev->d3hot_delay, pci_pm_d3hot_delay);
        unsigned int upper;
        if (delay_ms) {
                /* 20% upper bound, 1ms minimum */
                upper = max(DIV_ROUND_CLOSEST(delay_ms, 5), 1U)
                usleep_range(delay_ms * USEC_PER_MSEC,
                             (delay_ms + upper) * USEC_PER_MSEC);
        }

}

Quoted 7 lines

Since the Intel quirk is for 120ms, a 20% upper bound would make the range 120-144ms. Would that be a problem? Those chips are ancient; the list is untouched since it was added in 2006. The point of usleep_range() is to allow the scheduler to coalesce the wakeup with other events, so it seems unlikely we'd ever wait the whole 144ms. I vote for optimizing the readability over sleep/resume time for already-broken chips.

Quoted 2 lines

I'm totally fine with this, but I don't really know what the impact would be to those old Intel chips.

Worst-case, a few more ms to wakeup. Since we're starting with a huge 120ms *per device* delay, I think that's acceptable. Let's do this.

Bjorn