Skip to contentSkip to conversation

Use maximum latency when determining L1 ASPM

PATCH In review Ian Kumlien 3 participants 42 messages Oct 7, 2020 – May 15, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Use maximum latency when determining L1 ASPM

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

   +----------------+
   |                |
   |  Root complex  |
   |                |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 1
           |
   +----------------+
   |     |8 μs|     |
   |     +----+     |
   |    Switch A    |
   |     +----+     |
   |     |8 μs|     |
   +----------------+
           |
           |  Link 2
           |
   +----------------+
   |    |32 μs|     |
   |    +-----+     |
   |    Switch B    |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 3
           |
   +----------------+
   |     |8μs|      |
   |     +---+      |
   |   Endpoint C   |
   |                |
   |                |
   +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
lspci -PP -s 04:00.0
00:01.2/01:00.0/02:04.0/04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

lspci -vvv -s 04:00.0

		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

...
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
...

Which means that it can't be followed by any switch that is in L1 state.

This patch fixes it by disabling L1 on 02:04.0, 01:00.0 and 00:01.2.

                                                    LnkCtl    LnkCtl
           ------DevCap-------  ----LnkCap-------  -Before-  -After--
  00:01.2                                L1 <32us       L1+       L1-
  01:00.0                                L1 <32us       L1+       L1-
  02:04.0                                L1 <32us       L1+       L1-
  04:00.0  L0s <512 L1 <64us             L1 <64us       L1+       L1-
---
 drivers/pci/pcie/aspm.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +15 -8
index 253c30cc1967..893b37669087 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-	u32 latency, l1_switch_latency = 0;
+	u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
 	struct aspm_latency *acceptable;
 	struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
 		    (link->latency_dw.l0s > acceptable->l0s))
 			link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
 		/*
 		 * Check L1 latency.
-		 * Every switch on the path to root complex need 1
-		 * more microsecond for L1. Spec doesn't mention L0s.
+		 *
+		 * PCIe r5.0, sec 5.4.1.2.2 states:
+		 * A Switch is required to initiate an L1 exit transition on its
+		 * Upstream Port Link after no more than 1 μs from the beginning of an
+		 * L1 exit transition on any of its Downstream Port Links.
 		 *
 		 * The exit latencies for L1 substates are not advertised
 		 * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		 * L1 exit latencies advertised by a device include L1
 		 * substate latencies (and hence do not do any check).
 		 */
-		latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-		if ((link->aspm_capable & ASPM_STATE_L1) &&
-		    (latency + l1_switch_latency > acceptable->l1))
-			link->aspm_capable &= ~ASPM_STATE_L1;
-		l1_switch_latency += 1000;
+		if (link->aspm_capable & ASPM_STATE_L1) {
+			latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+			l1_max_latency = max_t(u32, latency, l1_max_latency);
+			if (l1_max_latency + l1_switch_latency > acceptable->l1)
+				link->aspm_capable &= ~ASPM_STATE_L1;
+
+			l1_switch_latency += 1000;
+		}
 
 		link = link->parent;
 	}
Signature · 1 line
2.28.0
Tested-by on v1 1/1
#1 lore raw
Quoted 79 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

  +----------------+
  |                |
  |  Root complex  |
  |                |
  |    +-----+     |
  |    |32 μs|     |
  +----------------+
          |
          |  Link 1
          |
  +----------------+
  |     |8 μs|     |
  |     +----+     |
  |    Switch A    |
  |     +----+     |
  |     |8 μs|     |
  +----------------+
          |
          |  Link 2
          |
  +----------------+
  |    |32 μs|     |
  |    +-----+     |
  |    Switch B    |
  |    +-----+     |
  |    |32 μs|     |
  +----------------+
          |
          |  Link 3
          |
  +----------------+
  |     |8μs|      |
  |     +---+      |
  |   Endpoint C   |
  |                |
  |                |
  +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
lspci -PP -s 04:00.0
00:01.2/01:00.0/02:04.0/04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

lspci -vvv -s 04:00.0

		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

...
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
...

Which means that it can't be followed by any switch that is in L1 state.

This patch fixes it by disabling L1 on 02:04.0, 01:00.0 and 00:01.2.

Quoted 8 lines
                                                   LnkCtl    LnkCtl
          ------DevCap-------  ----LnkCap-------  -Before-  -After--
 00:01.2                                L1 <32us       L1+       L1-
 01:00.0                                L1 <32us       L1+       L1-
 02:04.0                                L1 <32us       L1+       L1-
 04:00.0  L0s <512 L1 <64us             L1 <64us       L1+       L1-

Solves an issue that enabling ASPM on Dell Latitude 5495 causes system freeze.

Quoted 57 lines

---
drivers/pci/pcie/aspm.c | 23 +++++++++++++++--------

1 file changed, 15 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +15 -8
index 253c30cc1967..893b37669087 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-	u32 latency, l1_switch_latency = 0;
+	u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
 	struct aspm_latency *acceptable;
 	struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
 		    (link->latency_dw.l0s > acceptable->l0s))
 			link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
 		/*
 		 * Check L1 latency.
-		 * Every switch on the path to root complex need 1
-		 * more microsecond for L1. Spec doesn't mention L0s.
+		 *
+		 * PCIe r5.0, sec 5.4.1.2.2 states:
+		 * A Switch is required to initiate an L1 exit transition on its
+		 * Upstream Port Link after no more than 1 μs from the beginning of an
+		 * L1 exit transition on any of its Downstream Port Links.
 		 *
 		 * The exit latencies for L1 substates are not advertised
 		 * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		 * L1 exit latencies advertised by a device include L1
 		 * substate latencies (and hence do not do any check).
 		 */
-		latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-		if ((link->aspm_capable & ASPM_STATE_L1) &&
-		    (latency + l1_switch_latency > acceptable->l1))
-			link->aspm_capable &= ~ASPM_STATE_L1;
-		l1_switch_latency += 1000;
+		if (link->aspm_capable & ASPM_STATE_L1) {
+			latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+			l1_max_latency = max_t(u32, latency, l1_max_latency);
+			if (l1_max_latency + l1_switch_latency > acceptable->l1)
+				link->aspm_capable &= ~ASPM_STATE_L1;
+
+			l1_switch_latency += 1000;
+		}
 
 		link = link->parent;
 	}
Signature · 1 line
2.28.0
#2 lore raw
In reply to Ian Kumlien
Quoted 73 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

   +----------------+
   |                |
   |  Root complex  |
   |                |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 1
           |
   +----------------+
   |     |8 μs|     |
   |     +----+     |
   |    Switch A    |
   |     +----+     |
   |     |8 μs|     |
   +----------------+
           |
           |  Link 2
           |
   +----------------+
   |    |32 μs|     |
   |    +-----+     |
   |    Switch B    |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 3
           |
   +----------------+
   |     |8μs|      |
   |     +---+      |
   |   Endpoint C   |
   |                |
   |                |
   +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
lspci -PP -s 04:00.0
00:01.2/01:00.0/02:04.0/04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

lspci -vvv -s 04:00.0

		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

...
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
...

Which means that it can't be followed by any switch that is in L1 state.

Quoted 8 lines

This patch fixes it by disabling L1 on 02:04.0, 01:00.0 and 00:01.2.

                                                    LnkCtl    LnkCtl
           ------DevCap-------  ----LnkCap-------  -Before-  -After--
  00:01.2                                L1 <32us       L1+       L1-
  01:00.0                                L1 <32us       L1+       L1-
  02:04.0                                L1 <32us       L1+       L1-
  04:00.0  L0s <512 L1 <64us             L1 <64us       L1+       L1-

OK, now we're getting close. We just need to flesh out the justification. We need:

- Tidy subject line. Use "git log --oneline drivers/pci/pcie/aspm.c" and follow the example.

- Description of the problem. I think it's poor bandwidth on your Intel I211 device, but we don't have the complete picture because that NIC is 03:00.0, which doesn't appear above at all.

- Explanation of what's wrong with the "before" ASPM configuration. I want to identify what is wrong on your system. The generic "doesn't match spec" part is good, but step 1 is the specific details, step 2 is the generalization to relate it to the spec.

- Complete "sudo lspci -vv" information for before and after the patch below. https://bugzilla.kernel.org/show_bug.cgi?id=208741
has some of this, but some of the lspci output appears to be copy/pasted and lost all its formatting, and it's not clear how some was collected (what kernel version, with/without patch, etc). Since I'm asking for bugzilla attachments, there's no space constraint, so just attach the complete unedited output for the whole system.

- URL to the bugzilla. Please open a new one with just the relevant problem report ("NIC is slow") and attach (1) "before" lspci output, (2) proposed patch, (3) "after" lspci output. The existing 208741 report is full of distractions and jumps to the conclusion without actually starting with the details of the problem.

Some of this I would normally just do myself, but I can't get the lspci info. It would be really nice if Kai-Heng could also add before/after lspci output from the system he tested.

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +15 -8
index 253c30cc1967..893b37669087 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-	u32 latency, l1_switch_latency = 0;
+	u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
 	struct aspm_latency *acceptable;
 	struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
 		    (link->latency_dw.l0s > acceptable->l0s))
 			link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
 		/*
 		 * Check L1 latency.
-		 * Every switch on the path to root complex need 1
-		 * more microsecond for L1. Spec doesn't mention L0s.
+		 *
+		 * PCIe r5.0, sec 5.4.1.2.2 states:
+		 * A Switch is required to initiate an L1 exit transition on its
+		 * Upstream Port Link after no more than 1 μs from the beginning of an
+		 * L1 exit transition on any of its Downstream Port Links.
 		 *
 		 * The exit latencies for L1 substates are not advertised
 		 * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		 * L1 exit latencies advertised by a device include L1
 		 * substate latencies (and hence do not do any check).
 		 */
-		latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-		if ((link->aspm_capable & ASPM_STATE_L1) &&
-		    (latency + l1_switch_latency > acceptable->l1))
-			link->aspm_capable &= ~ASPM_STATE_L1;
-		l1_switch_latency += 1000;
+		if (link->aspm_capable & ASPM_STATE_L1) {
+			latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+			l1_max_latency = max_t(u32, latency, l1_max_latency);
+			if (l1_max_latency + l1_switch_latency > acceptable->l1)
+				link->aspm_capable &= ~ASPM_STATE_L1;
+
+			l1_switch_latency += 1000;
+		}
 
 		link = link->parent;
 	}
Signature · 1 line
2.28.0
#3 lore raw
Quoted 84 lines · Bjorn Helgaas
Quoted 73 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

   +----------------+
   |                |
   |  Root complex  |
   |                |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 1
           |
   +----------------+
   |     |8 μs|     |
   |     +----+     |
   |    Switch A    |
   |     +----+     |
   |     |8 μs|     |
   +----------------+
           |
           |  Link 2
           |
   +----------------+
   |    |32 μs|     |
   |    +-----+     |
   |    Switch B    |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 3
           |
   +----------------+
   |     |8μs|      |
   |     +---+      |
   |   Endpoint C   |
   |                |
   |                |
   +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
lspci -PP -s 04:00.0
00:01.2/01:00.0/02:04.0/04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

lspci -vvv -s 04:00.0
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
...
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
...

Which means that it can't be followed by any switch that is in L1 state.

Quoted 8 lines

This patch fixes it by disabling L1 on 02:04.0, 01:00.0 and 00:01.2.

                                                    LnkCtl    LnkCtl
           ------DevCap-------  ----LnkCap-------  -Before-  -After--
  00:01.2                                L1 <32us       L1+       L1-
  01:00.0                                L1 <32us       L1+       L1-
  02:04.0                                L1 <32us       L1+       L1-
  04:00.0  L0s <512 L1 <64us             L1 <64us       L1+       L1-
Quoted 5 lines

OK, now we're getting close. We just need to flesh out the justification. We need:

- Tidy subject line. Use "git log --oneline drivers/pci/pcie/aspm.c" and follow the example.

Will do

Quoted 3 lines

- Description of the problem. I think it's poor bandwidth on your Intel I211 device, but we don't have the complete picture because that NIC is 03:00.0, which doesn't appear above at all.

I think we'll use Kai-Hengs issue, since it's actually more related to the change itself...

Mine is a side effect while Kai-Heng is actually hitting an issue caused by the bug.

Quoted 83 lines

- Explanation of what's wrong with the "before" ASPM configuration. I want to identify what is wrong on your system. The generic "doesn't match spec" part is good, but step 1 is the specific details, step 2 is the generalization to relate it to the spec.

- Complete "sudo lspci -vv" information for before and after the patch below. https://bugzilla.kernel.org/show_bug.cgi?id=208741
has some of this, but some of the lspci output appears to be copy/pasted and lost all its formatting, and it's not clear how some was collected (what kernel version, with/without patch, etc). Since I'm asking for bugzilla attachments, there's no space constraint, so just attach the complete unedited output for the whole system.

- URL to the bugzilla. Please open a new one with just the relevant problem report ("NIC is slow") and attach (1) "before" lspci output, (2) proposed patch, (3) "after" lspci output. The existing 208741 report is full of distractions and jumps to the conclusion without actually starting with the details of the problem.

Some of this I would normally just do myself, but I can't get the lspci info. It would be really nice if Kai-Heng could also add before/after lspci output from the system he tested.

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +15 -9
index 253c30cc1967..893b37669087 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
      struct aspm_latency *acceptable;
      struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                  (link->latency_dw.l0s > acceptable->l0s))
                      link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
              /*
               * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 μs from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
               *
               * The exit latencies for L1 substates are not advertised
               * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
               * L1 exit latencies advertised by a device include L1
               * substate latencies (and hence do not do any check).
               */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+
+                     l1_switch_latency += 1000;
+             }
 
              link = link->parent;
      }
--

2.28.0

#4 lore raw
Quoted 101 lines · Ian Kumlien
Quoted 84 lines · Bjorn Helgaas
Quoted 73 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

  +----------------+
  |                |
  |  Root complex  |
  |                |
  |    +-----+     |
  |    |32 μs|     |
  +----------------+
          |
          |  Link 1
          |
  +----------------+
  |     |8 μs|     |
  |     +----+     |
  |    Switch A    |
  |     +----+     |
  |     |8 μs|     |
  +----------------+
          |
          |  Link 2
          |
  +----------------+
  |    |32 μs|     |
  |    +-----+     |
  |    Switch B    |
  |    +-----+     |
  |    |32 μs|     |
  +----------------+
          |
          |  Link 3
          |
  +----------------+
  |     |8μs|      |
  |     +---+      |
  |   Endpoint C   |
  |                |
  |                |
  +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
lspci -PP -s 04:00.0
00:01.2/01:00.0/02:04.0/04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

lspci -vvv -s 04:00.0
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
...
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
...

Which means that it can't be followed by any switch that is in L1 state.

Quoted 8 lines

This patch fixes it by disabling L1 on 02:04.0, 01:00.0 and 00:01.2.

                                                   LnkCtl    LnkCtl
          ------DevCap-------  ----LnkCap-------  -Before-  -After--
 00:01.2                                L1 <32us       L1+       L1-
 01:00.0                                L1 <32us       L1+       L1-
 02:04.0                                L1 <32us       L1+       L1-
 04:00.0  L0s <512 L1 <64us             L1 <64us       L1+       L1-
Quoted 5 lines

OK, now we're getting close. We just need to flesh out the justification. We need:

- Tidy subject line. Use "git log --oneline drivers/pci/pcie/aspm.c" and follow the example.

Will do

Quoted 3 lines

- Description of the problem. I think it's poor bandwidth on your Intel I211 device, but we don't have the complete picture because that NIC is 03:00.0, which doesn't appear above at all.

Quoted 5 lines

I think we'll use Kai-Hengs issue, since it's actually more related to the change itself...

Mine is a side effect while Kai-Heng is actually hitting an issue caused by the bug.

I filed a bug here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

Kai-Heng

Quoted 84 lines
Quoted 84 lines

- Explanation of what's wrong with the "before" ASPM configuration. I want to identify what is wrong on your system. The generic "doesn't match spec" part is good, but step 1 is the specific details, step 2 is the generalization to relate it to the spec.

- Complete "sudo lspci -vv" information for before and after the patch below. https://bugzilla.kernel.org/show_bug.cgi?id=208741
has some of this, but some of the lspci output appears to be copy/pasted and lost all its formatting, and it's not clear how some was collected (what kernel version, with/without patch, etc). Since I'm asking for bugzilla attachments, there's no space constraint, so just attach the complete unedited output for the whole system.

- URL to the bugzilla. Please open a new one with just the relevant problem report ("NIC is slow") and attach (1) "before" lspci output, (2) proposed patch, (3) "after" lspci output. The existing 208741 report is full of distractions and jumps to the conclusion without actually starting with the details of the problem.

Some of this I would normally just do myself, but I can't get the lspci info. It would be really nice if Kai-Heng could also add before/after lspci output from the system he tested.

Quoted 58 lines

---
drivers/pci/pcie/aspm.c | 23 +++++++++++++++--------

1 file changed, 15 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +15 -9
index 253c30cc1967..893b37669087 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
     struct aspm_latency *acceptable;
     struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
             if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                 (link->latency_dw.l0s > acceptable->l0s))
                     link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
             /*
              * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 μs from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
              *
              * The exit latencies for L1 substates are not advertised
              * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              * L1 exit latencies advertised by a device include L1
              * substate latencies (and hence do not do any check).
              */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+
+                     l1_switch_latency += 1000;
+             }
 
             link = link->parent;
     }
--

2.28.0

#5 lore raw
Quoted 102 lines · Kai-Heng Feng
Quoted 102 lines · Ian Kumlien
Quoted 84 lines · Bjorn Helgaas
Quoted 73 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

  +----------------+
  |                |
  |  Root complex  |
  |                |
  |    +-----+     |
  |    |32 μs|     |
  +----------------+
          |
          |  Link 1
          |
  +----------------+
  |     |8 μs|     |
  |     +----+     |
  |    Switch A    |
  |     +----+     |
  |     |8 μs|     |
  +----------------+
          |
          |  Link 2
          |
  +----------------+
  |    |32 μs|     |
  |    +-----+     |
  |    Switch B    |
  |    +-----+     |
  |    |32 μs|     |
  +----------------+
          |
          |  Link 3
          |
  +----------------+
  |     |8μs|      |
  |     +---+      |
  |   Endpoint C   |
  |                |
  |                |
  +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
lspci -PP -s 04:00.0
00:01.2/01:00.0/02:04.0/04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

lspci -vvv -s 04:00.0
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
...
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
...

Which means that it can't be followed by any switch that is in L1 state.

Quoted 8 lines

This patch fixes it by disabling L1 on 02:04.0, 01:00.0 and 00:01.2.

                                                   LnkCtl    LnkCtl
          ------DevCap-------  ----LnkCap-------  -Before-  -After--
 00:01.2                                L1 <32us       L1+       L1-
 01:00.0                                L1 <32us       L1+       L1-
 02:04.0                                L1 <32us       L1+       L1-
 04:00.0  L0s <512 L1 <64us             L1 <64us       L1+       L1-
Quoted 5 lines

OK, now we're getting close. We just need to flesh out the justification. We need:

- Tidy subject line. Use "git log --oneline drivers/pci/pcie/aspm.c" and follow the example.

Will do

Quoted 3 lines

- Description of the problem. I think it's poor bandwidth on your Intel I211 device, but we don't have the complete picture because that NIC is 03:00.0, which doesn't appear above at all.

Quoted 8 lines
Quoted 5 lines

I think we'll use Kai-Hengs issue, since it's actually more related to the change itself...

Mine is a side effect while Kai-Heng is actually hitting an issue caused by the bug.

I filed a bug here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

Thanks!

I'm actually starting to think that reporting what we do with the latency bit could
be beneficial - i.e. report which links have their L1 disabled due to which device...

I also think that this could benefit debugging - I have no clue of how to read the lspci:s - I mean i do see some differences that might be the fix but nothing really specific without a proper message in dmesg....

Björn, what do you think?

Quoted 87 lines

Kai-Heng

Quoted 84 lines
Quoted 84 lines

- Explanation of what's wrong with the "before" ASPM configuration. I want to identify what is wrong on your system. The generic "doesn't match spec" part is good, but step 1 is the specific details, step 2 is the generalization to relate it to the spec.

- Complete "sudo lspci -vv" information for before and after the patch below. https://bugzilla.kernel.org/show_bug.cgi?id=208741
has some of this, but some of the lspci output appears to be copy/pasted and lost all its formatting, and it's not clear how some was collected (what kernel version, with/without patch, etc). Since I'm asking for bugzilla attachments, there's no space constraint, so just attach the complete unedited output for the whole system.

- URL to the bugzilla. Please open a new one with just the relevant problem report ("NIC is slow") and attach (1) "before" lspci output, (2) proposed patch, (3) "after" lspci output. The existing 208741 report is full of distractions and jumps to the conclusion without actually starting with the details of the problem.

Some of this I would normally just do myself, but I can't get the lspci info. It would be really nice if Kai-Heng could also add before/after lspci output from the system he tested.

Quoted 58 lines

---
drivers/pci/pcie/aspm.c | 23 +++++++++++++++--------

1 file changed, 15 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +15 -9
index 253c30cc1967..893b37669087 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
     struct aspm_latency *acceptable;
     struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
             if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                 (link->latency_dw.l0s > acceptable->l0s))
                     link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
             /*
              * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 μs from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
              *
              * The exit latencies for L1 substates are not advertised
              * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              * L1 exit latencies advertised by a device include L1
              * substate latencies (and hence do not do any check).
              */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+
+                     l1_switch_latency += 1000;
+             }
 
             link = link->parent;
     }
--

2.28.0

#6 lore raw
Quoted 9 lines · Ian Kumlien

I'm actually starting to think that reporting what we do with the latency bit could
be beneficial - i.e. report which links have their L1 disabled due to which device...

I also think that this could benefit debugging - I have no clue of how to read the lspci:s - I mean i do see some differences that might be the fix but nothing really specific without a proper message in dmesg....

Yeah, might be worth doing. Probably pci_dbg() since I think it would be too chatty to be enabled all the time.

#7 lore raw
Quoted 8 lines · Bjorn Helgaas
Quoted 5 lines · Ian Kumlien

I'm actually starting to think that reporting what we do with the latency bit could
be beneficial - i.e. report which links have their L1 disabled due to which device...

Quoted 7 lines
Quoted 4 lines

I also think that this could benefit debugging - I have no clue of how to read the lspci:s - I mean i do see some differences that might be the fix but nothing really specific without a proper message in dmesg....

Yeah, might be worth doing. Probably pci_dbg() since I think it would be too chatty to be enabled all the time.

OK, will look in to it =)

#8 lore raw
Quoted 15 lines · Ian Kumlien
Quoted 13 lines · Bjorn Helgaas
Quoted 9 lines · Ian Kumlien

I'm actually starting to think that reporting what we do with the latency bit could
be beneficial - i.e. report which links have their L1 disabled due to which device...

I also think that this could benefit debugging - I have no clue of how to read the lspci:s - I mean i do see some differences that might be the fix but nothing really specific without a proper message in dmesg....

Quoted 4 lines
Quoted 2 lines

Yeah, might be worth doing. Probably pci_dbg() since I think it would be too chatty to be enabled all the time.

OK, will look in to it =)

Just did some very basic hack, since i think it's much better to get the information in dmesg than have to boot will full debug.

I assume that there is a niftier way to do it - but i wanted some feedback basically...

My current output is:
dmesg |grep latency

[    0.817872] pci 0000:04:00.0: ASPM latency exceeded, disabling:

L1:0000:01:00.0-0000:00:01.2

drivers/pci/pcie/aspm.c +2 -1
index 253c30cc1967..5a5146f47aae 100644
@@ -434,7 +434,8 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-       u32 latency, l1_switch_latency = 0;
+       u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
+       bool aspm_disable = 0;
        struct aspm_latency *acceptable;
        struct pcie_link_state *link;
 
@@ -446,6 +447,16 @@ static void pcie_aspm_check_latency(struct

pci_dev *endpoint)

        link = endpoint->bus->self->link_state;
        acceptable = &link->acceptable[PCI_FUNC(endpoint->devfn)];

+#define aspm_info(device, type, down, up) \
+ if (!aspm_disable) \
+ { \
+ pr_cont("pci %s: ASPM latency exceeded, disabling: %s:%s-%s", \
+ pci_name(device), type, pci_name(down), pci_name(up)); \
+ aspm_disable = 1; \
+ } \
+ else \
+ pr_cont(", %s:%s-%s", type, pci_name(down), pci_name(up));
+

        while (link) {
                /* Check upstream direction L0s latency */
                if ((link->aspm_capable & ASPM_STATE_L0S_UP) &&
@@ -456,10 +467,14 @@ static void pcie_aspm_check_latency(struct
pci_dev *endpoint)
                if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                    (link->latency_dw.l0s > acceptable->l0s))
                        link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
                /*
                 * Check L1 latency.
-                * Every switch on the path to root complex need 1
-                * more microsecond for L1. Spec doesn't mention L0s.
+                *
+                * PCIe r5.0, sec 5.4.1.2.2 states:
+                * A Switch is required to initiate an L1 exit transition on its
+                * Upstream Port Link after no more than 1 μs from the

beginning of an
+ * L1 exit transition on any of its Downstream Port Links.
*
* The exit latencies for L1 substates are not advertised
* by a device. Since the spec also doesn't mention a way
@@ -469,14 +484,22 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
* L1 exit latencies advertised by a device include L1
* substate latencies (and hence do not do any check). */
- latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
- if ((link->aspm_capable & ASPM_STATE_L1) &&
- (latency + l1_switch_latency > acceptable->l1))
- link->aspm_capable &= ~ASPM_STATE_L1;
- l1_switch_latency += 1000;
+ if (link->aspm_capable & ASPM_STATE_L1) {
+ latency = max_t(u32, link->latency_up.l1,
link->latency_dw.l1);

+                       l1_max_latency = max_t(u32, latency, l1_max_latency);
+                       if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                       {

+ aspm_info(endpoint, "L1",
link->downstream, link->pdev);
+ link->aspm_capable &= ~ASPM_STATE_L1;
+ }
+ l1_switch_latency += 1000;
+ }

                link = link->parent;
        }

+ if (aspm_disable)
+ pr_cont("\n");
+#undef aspm_info
}

for L1 and L0s, which we haven't discussed that much yet:

drivers/pci/pcie/aspm.c +2 -1
index 5a5146f47aae..b01d393e742d 100644
@@ -434,7 +434,8 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-       u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
+       u32 latency, l1_max_latency = 0, l1_switch_latency = 0,
+               l0s_latency_up = 0, l0s_latency_dw = 0;
        bool aspm_disable = 0;
        struct aspm_latency *acceptable;
        struct pcie_link_state *link;
@@ -459,14 +460,24 @@ static void pcie_aspm_check_latency(struct

pci_dev *endpoint)

        while (link) {
                /* Check upstream direction L0s latency */

- if ((link->aspm_capable & ASPM_STATE_L0S_UP) &&
- (link->latency_up.l0s > acceptable->l0s))
- link->aspm_capable &= ~ASPM_STATE_L0S_UP;
+ if (link->aspm_capable & ASPM_STATE_L0S_UP) {

+                       l0s_latency_up += link->latency_up.l0s;
+                       if (l0s_latency_up > acceptable->l0s)
+                       {

+ aspm_info(endpoint, "L0s-up",
link->downstream, link->pdev);
+ link->aspm_capable &= ~ASPM_STATE_L0S_UP;
+ }
+ }

                /* Check downstream direction L0s latency */

- if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
- (link->latency_dw.l0s > acceptable->l0s))
- link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+ if (link->aspm_capable & ASPM_STATE_L0S_DW) {

+                       l0s_latency_dw += link->latency_dw.l0s;
+                       if (l0s_latency_dw > acceptable->l0s)
+                       {

+ aspm_info(endpoint, "L0s-dw",
link->downstream, link->pdev);
+ link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+ }
+ }

/*
* Check L1 latency.

#9 lore raw
In reply to Ian Kumlien
Quoted 20 lines · Ian Kumlien
Quoted 18 lines · Kai-Heng Feng
Quoted 18 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

OK, now we're getting close. We just need to flesh out the justification. We need:

- Tidy subject line. Use "git log --oneline drivers/pci/pcie/aspm.c" and follow the example.

Will do

Quoted 3 lines

- Description of the problem. I think it's poor bandwidth on your Intel I211 device, but we don't have the complete picture because that NIC is 03:00.0, which doesn't appear above at all.

I think we'll use Kai-Hengs issue, since it's actually more related to the change itself...

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Mine is a side effect while Kai-Heng is actually hitting an issue caused by the bug.

I filed a bug here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

Thanks!

Sigh. I feel like I'm just not getting anywhere here. I still do not have a "before" and "after" set of lspci output.

Kai-Heng's bugzilla has two sets of output, but one is a working config with CONFIG_PCIEASPM_DEFAULT=y and the other is a working config with Ian's patch applied and CONFIG_PCIEASPM_POWERSAVE=y.

Comparing them doesn't show the effect of Ian's patch; it shows the combined effect of Ian's patch and the CONFIG_PCIEASPM_POWERSAVE=y change. I'm not really interested in spending a few hours trying to reverse-engineer the important changes.

Can you please, please, collect these on your system, Ian? I assume that you can easily collect it once without your patch, when you see poor I211 NIC performance but the system is otherwise working. And you can collect it again *with* your patch. Same Kconfig, same *everything* except adding your patch.

Bjorn

#10 lore raw
Quoted 42 lines · Bjorn Helgaas
Quoted 20 lines · Ian Kumlien
Quoted 18 lines · Kai-Heng Feng
Quoted 18 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

OK, now we're getting close. We just need to flesh out the justification. We need:

- Tidy subject line. Use "git log --oneline drivers/pci/pcie/aspm.c" and follow the example.

Will do

Quoted 3 lines

- Description of the problem. I think it's poor bandwidth on your Intel I211 device, but we don't have the complete picture because that NIC is 03:00.0, which doesn't appear above at all.

I think we'll use Kai-Hengs issue, since it's actually more related to the change itself...

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Mine is a side effect while Kai-Heng is actually hitting an issue caused by the bug.

I filed a bug here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

Thanks!

Sigh. I feel like I'm just not getting anywhere here. I still do not have a "before" and "after" set of lspci output.

Kai-Heng's bugzilla has two sets of output, but one is a working config with CONFIG_PCIEASPM_DEFAULT=y and the other is a working config with Ian's patch applied and CONFIG_PCIEASPM_POWERSAVE=y.

Comparing them doesn't show the effect of Ian's patch; it shows the combined effect of Ian's patch and the CONFIG_PCIEASPM_POWERSAVE=y change. I'm not really interested in spending a few hours trying to reverse-engineer the important changes.

Quoted 5 lines

Can you please, please, collect these on your system, Ian? I assume that you can easily collect it once without your patch, when you see poor I211 NIC performance but the system is otherwise working. And you can collect it again *with* your patch. Same Kconfig, same *everything* except adding your patch.

Yeah I can do that, but I would like the changes output from the latest patch suggestion
running on Kai-Heng's system so we can actually see what it does...

Quoted 1 line

Bjorn

#11 lore raw
Quoted 50 lines · Ian Kumlien
Quoted 42 lines · Bjorn Helgaas
Quoted 20 lines · Ian Kumlien
Quoted 18 lines · Kai-Heng Feng
Quoted 18 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

OK, now we're getting close. We just need to flesh out the justification. We need:

- Tidy subject line. Use "git log --oneline drivers/pci/pcie/aspm.c" and follow the example.

Will do

Quoted 3 lines

- Description of the problem. I think it's poor bandwidth on your Intel I211 device, but we don't have the complete picture because that NIC is 03:00.0, which doesn't appear above at all.

I think we'll use Kai-Hengs issue, since it's actually more related to the change itself...

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Mine is a side effect while Kai-Heng is actually hitting an issue caused by the bug.

I filed a bug here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

Thanks!

Sigh. I feel like I'm just not getting anywhere here. I still do not have a "before" and "after" set of lspci output.

Kai-Heng's bugzilla has two sets of output, but one is a working config with CONFIG_PCIEASPM_DEFAULT=y and the other is a working config with Ian's patch applied and CONFIG_PCIEASPM_POWERSAVE=y.

Comparing them doesn't show the effect of Ian's patch; it shows the combined effect of Ian's patch and the CONFIG_PCIEASPM_POWERSAVE=y change. I'm not really interested in spending a few hours trying to reverse-engineer the important changes.

Quoted 5 lines

Can you please, please, collect these on your system, Ian? I assume that you can easily collect it once without your patch, when you see poor I211 NIC performance but the system is otherwise working. And you can collect it again *with* your patch. Same Kconfig, same *everything* except adding your patch.

Quoted 3 lines

Yeah I can do that, but I would like the changes output from the latest patch suggestion
running on Kai-Heng's system so we can actually see what it does...

Is:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

More to your liking?

Quoted 1 line
Quoted 1 line

Bjorn

#12 lore raw
Quoted 9 lines · Ian Kumlien
Quoted 8 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

Can you please, please, collect these on your system, Ian? I assume that you can easily collect it once without your patch, when you see poor I211 NIC performance but the system is otherwise working. And you can collect it again *with* your patch. Same Kconfig, same *everything* except adding your patch.

Quoted 6 lines
Quoted 3 lines

Yeah I can do that, but I would like the changes output from the latest patch suggestion
running on Kai-Heng's system so we can actually see what it does...

Is:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

That's a great start. Can you attach the patch to the bugzilla too, please, so it is self-contained?

And also the analysis of the path from Root Port to Endpoint, with the exit latencies of each link, the acceptable latency of the endpoint and

(1) the computation done by the existing code that results in "latency < acceptable" that means we can enable ASPM, and

(2) the correct computation per spec that results in "latency > acceptable" so we cannot enable ASPM?

This analysis will be the core of the commit log, and the bugzilla with lspci info is the supporting evidence.

Bjorn

#13 lore raw
Quoted 25 lines · Bjorn Helgaas
Quoted 9 lines · Ian Kumlien
Quoted 8 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

Can you please, please, collect these on your system, Ian? I assume that you can easily collect it once without your patch, when you see poor I211 NIC performance but the system is otherwise working. And you can collect it again *with* your patch. Same Kconfig, same *everything* except adding your patch.

Quoted 6 lines
Quoted 3 lines

Yeah I can do that, but I would like the changes output from the latest patch suggestion
running on Kai-Heng's system so we can actually see what it does...

Is:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

That's a great start. Can you attach the patch to the bugzilla too, please, so it is self-contained?

And also the analysis of the path from Root Port to Endpoint, with the exit latencies of each link, the acceptable latency of the endpoint and

Quoted 8 lines

(1) the computation done by the existing code that results in "latency < acceptable" that means we can enable ASPM, and

(2) the correct computation per spec that results in "latency > acceptable" so we cannot enable ASPM?

This analysis will be the core of the commit log, and the bugzilla with lspci info is the supporting evidence.

Ok, will do, there will be some bio-latency though

Were you ok with the pr_cont output per endpoint?

Quoted 1 line

Bjorn

#14 lore raw
Quoted 32 lines · Ian Kumlien
Quoted 31 lines · Bjorn Helgaas
Quoted 9 lines · Ian Kumlien
Quoted 8 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

Can you please, please, collect these on your system, Ian? I assume that you can easily collect it once without your patch, when you see poor I211 NIC performance but the system is otherwise working. And you can collect it again *with* your patch. Same Kconfig, same *everything* except adding your patch.

Quoted 6 lines
Quoted 3 lines

Yeah I can do that, but I would like the changes output from the latest patch suggestion
running on Kai-Heng's system so we can actually see what it does...

Is:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

That's a great start. Can you attach the patch to the bugzilla too, please, so it is self-contained?

And also the analysis of the path from Root Port to Endpoint, with the exit latencies of each link, the acceptable latency of the endpoint and

(1) the computation done by the existing code that results in "latency < acceptable" that means we can enable ASPM, and

(2) the correct computation per spec that results in "latency > acceptable" so we cannot enable ASPM?

Quoted 6 lines
Quoted 2 lines

This analysis will be the core of the commit log, and the bugzilla with lspci info is the supporting evidence.

Ok, will do, there will be some bio-latency though

Were you ok with the pr_cont output per endpoint?

Haven't looked at that yet. Will respond to that patch when I do.

#15 lore raw

PATCH 1/3 PCI/ASPM: Use the path max in L1 ASPM latency check

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

   +----------------+
   |                |
   |  Root complex  |
   |                |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 1
           |
   +----------------+
   |     |8 μs|     |
   |     +----+     |
   |    Switch A    |
   |     +----+     |
   |     |8 μs|     |
   +----------------+
           |
           |  Link 2
           |
   +----------------+
   |    |32 μs|     |
   |    +-----+     |
   |    Switch B    |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 3
           |
   +----------------+
   |     |8μs|      |
   |     +---+      |
   |   Endpoint C   |
   |                |
   |                |
   +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -8
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-	u32 latency, l1_switch_latency = 0;
+	u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
 	struct aspm_latency *acceptable;
 	struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
 		    (link->latency_dw.l0s > acceptable->l0s))
 			link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
 		/*
 		 * Check L1 latency.
-		 * Every switch on the path to root complex need 1
-		 * more microsecond for L1. Spec doesn't mention L0s.
+		 *
+		 * PCIe r5.0, sec 5.4.1.2.2 states:
+		 * A Switch is required to initiate an L1 exit transition on its
+		 * Upstream Port Link after no more than 1 μs from the beginning of an
+		 * L1 exit transition on any of its Downstream Port Links.
 		 *
 		 * The exit latencies for L1 substates are not advertised
 		 * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		 * L1 exit latencies advertised by a device include L1
 		 * substate latencies (and hence do not do any check).
 		 */
-		latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-		if ((link->aspm_capable & ASPM_STATE_L1) &&
-		    (latency + l1_switch_latency > acceptable->l1))
-			link->aspm_capable &= ~ASPM_STATE_L1;
-		l1_switch_latency += 1000;
+		if (link->aspm_capable & ASPM_STATE_L1) {
+			latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+			l1_max_latency = max_t(u32, latency, l1_max_latency);
+			if (l1_max_latency + l1_switch_latency > acceptable->l1)
+				link->aspm_capable &= ~ASPM_STATE_L1;
+			l1_switch_latency += 1000;
+		}
 
 		link = link->parent;
 	}
Signature · 1 line
2.29.1
#16 lore raw

PATCH 2/3 PCI/ASPM: Fix L0s max latency check

From what I have been able to figure out, it seems like LOs path latency is cumulative, so the max path latency should be the sum of all links maximum latency.

---
 drivers/pci/pcie/aspm.c | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
drivers/pci/pcie/aspm.c +12 -7
index c03ead0f1013..dbe3ce60c1ff 100644
@@ -434,7 +434,8 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-	u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
+	u32 latency, l1_max_latency = 0, l1_switch_latency = 0,
+		l0s_latency_up = 0, l0s_latency_dw = 0;
 	struct aspm_latency *acceptable;
 	struct pcie_link_state *link;
 
@@ -448,14 +449,18 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 
 	while (link) {
 		/* Check upstream direction L0s latency */
-		if ((link->aspm_capable & ASPM_STATE_L0S_UP) &&
-		    (link->latency_up.l0s > acceptable->l0s))
-			link->aspm_capable &= ~ASPM_STATE_L0S_UP;
+		if (link->aspm_capable & ASPM_STATE_L0S_UP) {
+			l0s_latency_up += link->latency_up.l0s;
+			if (l0s_latency_up > acceptable->l0s)
+				link->aspm_capable &= ~ASPM_STATE_L0S_UP;
+		}
 
 		/* Check downstream direction L0s latency */
-		if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
-		    (link->latency_dw.l0s > acceptable->l0s))
-			link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+		if (link->aspm_capable & ASPM_STATE_L0S_DW) {
+			l0s_latency_dw += link->latency_dw.l0s;
+			if (l0s_latency_dw > acceptable->l0s)
+				link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+		}
 
 		/*
 		 * Check L1 latency.
Signature · 1 line
2.29.1
#17 lore raw

Re: [PATCH 2/3] PCI/ASPM: Fix L0s max latency check

*bump*

Quoted 52 lines · Ian Kumlien

From what I have been able to figure out, it seems like LOs path latency is cumulative, so the max path latency should be the sum of all links maximum latency.

---
 drivers/pci/pcie/aspm.c | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
drivers/pci/pcie/aspm.c +12 -8
index c03ead0f1013..dbe3ce60c1ff 100644
@@ -434,7 +434,8 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-       u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
+       u32 latency, l1_max_latency = 0, l1_switch_latency = 0,
+               l0s_latency_up = 0, l0s_latency_dw = 0;
        struct aspm_latency *acceptable;
        struct pcie_link_state *link;
 
@@ -448,14 +449,18 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 
        while (link) {
                /* Check upstream direction L0s latency */
-               if ((link->aspm_capable & ASPM_STATE_L0S_UP) &&
-                   (link->latency_up.l0s > acceptable->l0s))
-                       link->aspm_capable &= ~ASPM_STATE_L0S_UP;
+               if (link->aspm_capable & ASPM_STATE_L0S_UP) {
+                       l0s_latency_up += link->latency_up.l0s;
+                       if (l0s_latency_up > acceptable->l0s)
+                               link->aspm_capable &= ~ASPM_STATE_L0S_UP;
+               }
 
                /* Check downstream direction L0s latency */
-               if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
-                   (link->latency_dw.l0s > acceptable->l0s))
-                       link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+               if (link->aspm_capable & ASPM_STATE_L0S_DW) {
+                       l0s_latency_dw += link->latency_dw.l0s;
+                       if (l0s_latency_dw > acceptable->l0s)
+                               link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+               }
 
                /*
                 * Check L1 latency.
--

2.29.1

#18 lore raw

RFC PATCH 3/3 PCI/ASPM: Print L1/L0s latency messages per endpoint

In reply to Ian Kumlien

Print information on where along the path we disable L1 or L0s per endpoint. I think the infromation could be useful in the normal dmesg.

---
 drivers/pci/pcie/aspm.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
drivers/pci/pcie/aspm.c +23 -0
index dbe3ce60c1ff..8bec24119f3e 100644
@@ -436,6 +436,7 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
 	u32 latency, l1_max_latency = 0, l1_switch_latency = 0,
 		l0s_latency_up = 0, l0s_latency_dw = 0;
+	bool aspm_disable = 0;
 	struct aspm_latency *acceptable;
 	struct pcie_link_state *link;
 
@@ -447,19 +448,35 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 	link = endpoint->bus->self->link_state;
 	acceptable = &link->acceptable[PCI_FUNC(endpoint->devfn)];
 
+#define aspm_info(device, type, down, up) \
+	if (!aspm_disable) \
+	{ \
+		pr_cont("pci %s: ASPM latency exceeded, disabling: %s:%s-%s", \
+			pci_name(device), type, pci_name(down), pci_name(up)); \
+		aspm_disable = 1; \
+	} \
+	else \
+		pr_cont(", %s:%s-%s", type, pci_name(down), pci_name(up));
+
 	while (link) {
 		/* Check upstream direction L0s latency */
 		if (link->aspm_capable & ASPM_STATE_L0S_UP) {
 			l0s_latency_up += link->latency_up.l0s;
 			if (l0s_latency_up > acceptable->l0s)
+			{
 				link->aspm_capable &= ~ASPM_STATE_L0S_UP;
+				aspm_info(endpoint, "L0s-up", link->downstream, link->pdev);
+			}
 		}
 
 		/* Check downstream direction L0s latency */
 		if (link->aspm_capable & ASPM_STATE_L0S_DW) {
 			l0s_latency_dw += link->latency_dw.l0s;
 			if (l0s_latency_dw > acceptable->l0s)
+			{
 				link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+				aspm_info(endpoint, "L0s-dw", link->downstream, link->pdev);
+			}
 		}
 
 		/*
@@ -482,12 +499,18 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 			latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
 			l1_max_latency = max_t(u32, latency, l1_max_latency);
 			if (l1_max_latency + l1_switch_latency > acceptable->l1)
+			{
 				link->aspm_capable &= ~ASPM_STATE_L1;
+				aspm_info(endpoint, "L1", link->downstream, link->pdev);
+			}
 			l1_switch_latency += 1000;
 		}
 
 		link = link->parent;
 	}
+	if (aspm_disable)
+		pr_cont("\n");
+#undef aspm_info
 }
 
 /*
Signature · 1 line
2.29.1
#19 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

In reply to Ian Kumlien

Is this ok? Or what's missing?

Quoted 164 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

   +----------------+
   |                |
   |  Root complex  |
   |                |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 1
           |
   +----------------+
   |     |8 μs|     |
   |     +----+     |
   |    Switch A    |
   |     +----+     |
   |     |8 μs|     |
   +----------------+
           |
           |  Link 2
           |
   +----------------+
   |    |32 μs|     |
   |    +-----+     |
   |    Switch B    |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 3
           |
   +----------------+
   |     |8μs|      |
   |     +---+      |
   |   Endpoint C   |
   |                |
   |                |
   +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-       u32 latency, l1_switch_latency = 0;
+       u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
        struct aspm_latency *acceptable;
        struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
                if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                    (link->latency_dw.l0s > acceptable->l0s))
                        link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
                /*
                 * Check L1 latency.
-                * Every switch on the path to root complex need 1
-                * more microsecond for L1. Spec doesn't mention L0s.
+                *
+                * PCIe r5.0, sec 5.4.1.2.2 states:
+                * A Switch is required to initiate an L1 exit transition on its
+                * Upstream Port Link after no more than 1 μs from the beginning of an
+                * L1 exit transition on any of its Downstream Port Links.
                 *
                 * The exit latencies for L1 substates are not advertised
                 * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
                 * L1 exit latencies advertised by a device include L1
                 * substate latencies (and hence do not do any check).
                 */
-               latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-               if ((link->aspm_capable & ASPM_STATE_L1) &&
-                   (latency + l1_switch_latency > acceptable->l1))
-                       link->aspm_capable &= ~ASPM_STATE_L1;
-               l1_switch_latency += 1000;
+               if (link->aspm_capable & ASPM_STATE_L1) {
+                       latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                       l1_max_latency = max_t(u32, latency, l1_max_latency);
+                       if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                               link->aspm_capable &= ~ASPM_STATE_L1;
+                       l1_switch_latency += 1000;
+               }
 
                link = link->parent;
        }
--

2.29.1

#20 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

I was hoping that this patch would get in to 5.10 since it'll be a LTS release and all... but it doesn't seem like it.

Any thoughts?

Quoted 168 lines · Ian Kumlien

Is this ok? Or what's missing?

Quoted 164 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

   +----------------+
   |                |
   |  Root complex  |
   |                |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 1
           |
   +----------------+
   |     |8 μs|     |
   |     +----+     |
   |    Switch A    |
   |     +----+     |
   |     |8 μs|     |
   +----------------+
           |
           |  Link 2
           |
   +----------------+
   |    |32 μs|     |
   |    +-----+     |
   |    Switch B    |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 3
           |
   +----------------+
   |     |8μs|      |
   |     +---+      |
   |   Endpoint C   |
   |                |
   |                |
   +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-       u32 latency, l1_switch_latency = 0;
+       u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
        struct aspm_latency *acceptable;
        struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
                if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                    (link->latency_dw.l0s > acceptable->l0s))
                        link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
                /*
                 * Check L1 latency.
-                * Every switch on the path to root complex need 1
-                * more microsecond for L1. Spec doesn't mention L0s.
+                *
+                * PCIe r5.0, sec 5.4.1.2.2 states:
+                * A Switch is required to initiate an L1 exit transition on its
+                * Upstream Port Link after no more than 1 μs from the beginning of an
+                * L1 exit transition on any of its Downstream Port Links.
                 *
                 * The exit latencies for L1 substates are not advertised
                 * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
                 * L1 exit latencies advertised by a device include L1
                 * substate latencies (and hence do not do any check).
                 */
-               latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-               if ((link->aspm_capable & ASPM_STATE_L1) &&
-                   (latency + l1_switch_latency > acceptable->l1))
-                       link->aspm_capable &= ~ASPM_STATE_L1;
-               l1_switch_latency += 1000;
+               if (link->aspm_capable & ASPM_STATE_L1) {
+                       latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                       l1_max_latency = max_t(u32, latency, l1_max_latency);
+                       if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                               link->aspm_capable &= ~ASPM_STATE_L1;
+                       l1_switch_latency += 1000;
+               }
 
                link = link->parent;
        }
--

2.29.1

#21 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

In reply to Ian Kumlien
Quoted 24 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.
...

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 7 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

There are two paths here that share a Link:

00:01.2 --- 01:00.0 -- 02:03.0 --- 03:00.0 I211 NIC 00:01.2 --- 01:00.0 -- 02:04.0 --- 04:00.x multifunction Realtek

1) The path to the I211 NIC includes four Ports and two Links (the connection between 01:00.0 and 02:03.0 is internal Switch routing, not a Link).

The Ports advertise L1 exit latencies of <32us, <32us, <32us, <16us. If both Links are in L1 and 03:00.0 initiates L1 exit at T, 01:00.0 initiates L1 exit at T + 1. A TLP from 03:00.0 may see up to 1 + 32 = 33us of L1 exit latency.

The NIC can tolerate up to 64us of L1 exit latency, so it is safe to enable L1 for both Links.

2) The path to the Realtek device is similar except that the Realtek L1 exit latency is <64us. If both Links are in L1 and 04:00.x initiates L1 exit at T, 01:00.0 again initiates L1 exit at T + 1, but a TLP from 04:00.x may see up to 1 + 64 = 65us of L1 exit latency.

The Realtek device can only tolerate 64us of latency, so it is not safe to enable L1 for both Links. It should be safe to enable L1 on the shared link because the exit latency for that link would be <32us.

Quoted 11 lines

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

[Nit: I don't think we should add 1 for the 02:04.0 -- 01:00.0 piece because that's internal Switch routing, not a Link. But even without that extra microsecond, this path does exceed the acceptable latency since 1 + 64 = 65us, and 04:00.0 can only tolerate 64us.]

Quoted 4 lines

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

The "lspci without my patches" [1] shows L1 enabled for the shared Link from 00:01.2 --- 01:00.0 and for the Link to 03:00.0 (I211), but not for the Link to 04:00.x (Realtek).

Per my analysis above, that looks like it *should* be a safe configuration. 03:00.0 can tolerate 64us, actual is <33us. 04:00.0 can tolerate 64us, actual should be <32us since only the shared Link is in L1.

However, the commit log at [2] shows L1 *enabled* for both the shared Link from 00:01.2 --- 01:00.0 and the 02:04.0 --- 04:00.x Link, and that would definitely be a problem.

Can you explain the differences between [1] and [2]?

Quoted 3 lines

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

I started working through this info, too, but there's not enough information to tell what difference this patch makes. The attachments compare:

1) CONFIG_PCIEASPM_DEFAULT=y without the patch [3] and
2) CONFIG_PCIEASPM_POWERSAVE=y *with* the patch [4]

Obviously CONFIG_PCIEASPM_POWERSAVE=y will configure things differently than CONFIG_PCIEASPM_DEFAULT=y, so we can't tell what changes are due to the config change and what are due to the patch.

The lspci *with* the patch ([4]) shows L0s and L1 enabled at almost every possible place. Here are the Links, how they're configured, and my analysis of the exit latencies vs acceptable latencies:

  00:01.1 --- 01:00.0      L1+ (                  L1 <64us vs unl)
  00:01.2 --- 02:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.3 --- 03:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.4 --- 04:00.0      L1+ (                  L1 <64us vs unl)

00:08.1 --- 05:00.x L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl) 00:08.2 --- 06:00.0 L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl)

So I can't tell what change prevents the freeze. I would expect the patch would cause us to *disable* L0s or L1 somewhere.

The only place [4] shows ASPM disabled is for 05:00.1. The spec says we should program the same value in all functions of a multi-function device. This is a non-ARI device, so "only capabilities enabled in all functions are enabled for the component as a whole." That would mean that L0s and L1 are effectively disabled for 05:00.x even though 05:00.0 claims they're enabled. But the latencies say ASPM L0s and L1 should be safe to be enabled. This looks like another bug that's probably unrelated.

The patch might be correct; I haven't actually analyzed the code. But the commit log doesn't make sense to me yet.

[1] https://bugzilla.kernel.org/attachment.cgi?id=293047
[2] https://lore.kernel.org/linux-pci/[email protected]/
[3] https://bugzilla.kernel.org/attachment.cgi?id=292955
[4] https://bugzilla.kernel.org/attachment.cgi?id=292957

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -8
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-	u32 latency, l1_switch_latency = 0;
+	u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
 	struct aspm_latency *acceptable;
 	struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
 		    (link->latency_dw.l0s > acceptable->l0s))
 			link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
 		/*
 		 * Check L1 latency.
-		 * Every switch on the path to root complex need 1
-		 * more microsecond for L1. Spec doesn't mention L0s.
+		 *
+		 * PCIe r5.0, sec 5.4.1.2.2 states:
+		 * A Switch is required to initiate an L1 exit transition on its
+		 * Upstream Port Link after no more than 1 μs from the beginning of an
+		 * L1 exit transition on any of its Downstream Port Links.
 		 *
 		 * The exit latencies for L1 substates are not advertised
 		 * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		 * L1 exit latencies advertised by a device include L1
 		 * substate latencies (and hence do not do any check).
 		 */
-		latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-		if ((link->aspm_capable & ASPM_STATE_L1) &&
-		    (latency + l1_switch_latency > acceptable->l1))
-			link->aspm_capable &= ~ASPM_STATE_L1;
-		l1_switch_latency += 1000;
+		if (link->aspm_capable & ASPM_STATE_L1) {
+			latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+			l1_max_latency = max_t(u32, latency, l1_max_latency);
+			if (l1_max_latency + l1_switch_latency > acceptable->l1)
+				link->aspm_capable &= ~ASPM_STATE_L1;
+			l1_switch_latency += 1000;
+		}
 
 		link = link->parent;
 	}
Signature · 1 line
2.29.1
#22 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 56 lines · Bjorn Helgaas
Quoted 24 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.
...

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 7 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

There are two paths here that share a Link:

00:01.2 --- 01:00.0 -- 02:03.0 --- 03:00.0 I211 NIC 00:01.2 --- 01:00.0 -- 02:04.0 --- 04:00.x multifunction Realtek

1) The path to the I211 NIC includes four Ports and two Links (the connection between 01:00.0 and 02:03.0 is internal Switch routing, not a Link).

The Ports advertise L1 exit latencies of <32us, <32us, <32us, <16us. If both Links are in L1 and 03:00.0 initiates L1 exit at T, 01:00.0 initiates L1 exit at T + 1. A TLP from 03:00.0 may see up to 1 + 32 = 33us of L1 exit latency.

The NIC can tolerate up to 64us of L1 exit latency, so it is safe to enable L1 for both Links.

2) The path to the Realtek device is similar except that the Realtek L1 exit latency is <64us. If both Links are in L1 and 04:00.x initiates L1 exit at T, 01:00.0 again initiates L1 exit at T + 1, but a TLP from 04:00.x may see up to 1 + 64 = 65us of L1 exit latency.

Quoted 4 lines

The Realtek device can only tolerate 64us of latency, so it is not safe to enable L1 for both Links. It should be safe to enable L1 on the shared link because the exit latency for that link would be <32us.

04:00.0:
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us

So maximum latency for the entire link has to be <64 us For the device to leave L1 ASPM takes <64us

So the device itself is the slowest entry along the link, which means that nothing
else along that path can have ASPM enabled

Quoted 12 lines
Quoted 11 lines

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

Quoted 4 lines

[Nit: I don't think we should add 1 for the 02:04.0 -- 01:00.0 piece because that's internal Switch routing, not a Link. But even without that extra microsecond, this path does exceed the acceptable latency since 1 + 64 = 65us, and 04:00.0 can only tolerate 64us.]

It does report L1 ASPM on both ends, so the links will be counted as such in the code.

I also assume that it can power down individual ports... and enter rest state if no links are up.

Quoted 5 lines
Quoted 4 lines

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

Quoted 8 lines

The "lspci without my patches" [1] shows L1 enabled for the shared Link from 00:01.2 --- 01:00.0 and for the Link to 03:00.0 (I211), but not for the Link to 04:00.x (Realtek).

Per my analysis above, that looks like it *should* be a safe configuration. 03:00.0 can tolerate 64us, actual is <33us. 04:00.0 can tolerate 64us, actual should be <32us since only the shared Link is in L1.

See above.

Quoted 5 lines

However, the commit log at [2] shows L1 *enabled* for both the shared Link from 00:01.2 --- 01:00.0 and the 02:04.0 --- 04:00.x Link, and that would definitely be a problem.

Can you explain the differences between [1] and [2]?

I don't understand which sections you're referring to.

Quoted 29 lines
Quoted 3 lines

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

I started working through this info, too, but there's not enough information to tell what difference this patch makes. The attachments compare:

1) CONFIG_PCIEASPM_DEFAULT=y without the patch [3] and
2) CONFIG_PCIEASPM_POWERSAVE=y *with* the patch [4]

Obviously CONFIG_PCIEASPM_POWERSAVE=y will configure things differently than CONFIG_PCIEASPM_DEFAULT=y, so we can't tell what changes are due to the config change and what are due to the patch.

The lspci *with* the patch ([4]) shows L0s and L1 enabled at almost every possible place. Here are the Links, how they're configured, and my analysis of the exit latencies vs acceptable latencies:

  00:01.1 --- 01:00.0      L1+ (                  L1 <64us vs unl)
  00:01.2 --- 02:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.3 --- 03:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.4 --- 04:00.0      L1+ (                  L1 <64us vs unl)

00:08.1 --- 05:00.x L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl) 00:08.2 --- 06:00.0 L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl)

So I can't tell what change prevents the freeze. I would expect the patch would cause us to *disable* L0s or L1 somewhere.

Quoted 8 lines

The only place [4] shows ASPM disabled is for 05:00.1. The spec says we should program the same value in all functions of a multi-function device. This is a non-ARI device, so "only capabilities enabled in all functions are enabled for the component as a whole." That would mean that L0s and L1 are effectively disabled for 05:00.x even though 05:00.0 claims they're enabled. But the latencies say ASPM L0s and L1 should be safe to be enabled. This looks like another bug that's probably unrelated.

I don't think it's unrelated, i suspect it's how PCIe works with multiple links...
a device can cause some kind of head of queue stalling - i don't know how but it really looks like it.

Quoted 2 lines

The patch might be correct; I haven't actually analyzed the code. But the commit log doesn't make sense to me yet.

I personally don't think that all this PCI information is required, the linux kernel is
currently doing it wrong according to the spec.

Also, since it's clearly doing the wrong thing, I'm worried that dists will take a kernel enable aspm
and there will be alot of bugreports of non-booting systems or other weird issues... And the culprit
was known all along.

It's been five months...

Quoted 63 lines

[1] https://bugzilla.kernel.org/attachment.cgi?id=293047
[2] https://lore.kernel.org/linux-pci/[email protected]/
[3] https://bugzilla.kernel.org/attachment.cgi?id=292955
[4] https://bugzilla.kernel.org/attachment.cgi?id=292957

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
      struct aspm_latency *acceptable;
      struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                  (link->latency_dw.l0s > acceptable->l0s))
                      link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
              /*
               * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 μs from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
               *
               * The exit latencies for L1 substates are not advertised
               * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
               * L1 exit latencies advertised by a device include L1
               * substate latencies (and hence do not do any check).
               */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+                     l1_switch_latency += 1000;
+             }
 
              link = link->parent;
      }
--

2.29.1

#23 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

[+cc Jesse, Tony, David, Jakub, Heiner, lists in case there's an ASPM issue with I211 or Realtek NICs. Beginning of thread:
https://lore.kernel.org/r/20201024205548.1837770-1-ian.kumlien at gmail.com

Short story: Ian has:

Root Port --- Switch --- I211 NIC
\-- multifunction Realtek NIC, etc

and the I211 performance is poor with ASPM L1 enabled on both links in the path to it. The patch here disables ASPM on the upstream link and fixes the performance, but AFAICT the devices in that path give us no reason to disable L1. If I understand the spec correctly, the Realtek device should not be relevant to the I211 path.]

Quoted 67 lines · Ian Kumlien
Quoted 56 lines · Bjorn Helgaas
Quoted 24 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1?s from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.
...

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 7 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

There are two paths here that share a Link:

00:01.2 --- 01:00.0 -- 02:03.0 --- 03:00.0 I211 NIC 00:01.2 --- 01:00.0 -- 02:04.0 --- 04:00.x multifunction Realtek

1) The path to the I211 NIC includes four Ports and two Links (the connection between 01:00.0 and 02:03.0 is internal Switch routing, not a Link).

The Ports advertise L1 exit latencies of <32us, <32us, <32us, <16us. If both Links are in L1 and 03:00.0 initiates L1 exit@T, 01:00.0 initiates L1 exit at T + 1. A TLP from 03:00.0 may see up to 1 + 32 = 33us of L1 exit latency.

The NIC can tolerate up to 64us of L1 exit latency, so it is safe to enable L1 for both Links.

2) The path to the Realtek device is similar except that the Realtek L1 exit latency is <64us. If both Links are in L1 and 04:00.x initiates L1 exit at T, 01:00.0 again initiates L1 exit at T + 1, but a TLP from 04:00.x may see up to 1 + 64 = 65us of L1 exit latency.

Quoted 4 lines

The Realtek device can only tolerate 64us of latency, so it is not safe to enable L1 for both Links. It should be safe to enable L1 on the shared link because the exit latency for that link would be <32us.

04:00.0:
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us

Quoted 5 lines

So maximum latency for the entire link has to be <64 us For the device to leave L1 ASPM takes <64us

So the device itself is the slowest entry along the link, which means that nothing else along that path can have ASPM enabled

Yes. That's what I said above: "it is not safe to enable L1 for both Links." Unless I'm missing something, we agree on that.

I also said that it should be safe to enable L1 on the shared Link (from 00:01.2 to 01:00.0) because if the downstream Link is always in L0, the exit latency of the shared Link should be <32us, and 04:00.x can tolerate 64us.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

Quoted 7 lines
Quoted 4 lines

[Nit: I don't think we should add 1 for the 02:04.0 -- 01:00.0 piece because that's internal Switch routing, not a Link. But even without that extra microsecond, this path does exceed the acceptable latency since 1 + 64 = 65us, and 04:00.0 can only tolerate 64us.]

It does report L1 ASPM on both ends, so the links will be counted as such in the code.

This is a bit of a tangent and we shouldn't get too wrapped up in it. This is a confusing aspect of PCIe. We're talking about this path:

00:01.2 --- [01:00.0 -- 02:04.0] --- 04:00.x multifunction Realtek

This path only contains two Links. The first one is 00:01.2 --- 01:00.0, and the second one is 02:04.0 --- 04:00.x.

01:00.0 is a Switch Upstream Port and 02:04.0 is a Switch Downstream Port. The connection between them is not a Link; it is some internal wiring of the Switch that is completely opaque to software.

The ASPM information and knobs in 01:00.0 apply to the Link on its upstream side, and the ASPM info and knobs in 02:04.0 apply to the Link on its downstream side.

The example in sec 5.4.1.2.2 contains three Links. The L1 exit latency for the Link is the max of the exit latencies at each end:

Link 1: max(32, 8) = 32us
Link 2: max(8, 32) = 32us
Link 3: max(32, 8) = 32us

The total delay for a TLP starting at the downstream end of Link 3 is 32 + 2 = 32us.

In the path to your 04:00.x Realtek device:

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us Link 2 (from 02:04.0 to 04:00.x): max(32, 64) = 64us

If L1 were enabled on both Links, the exit latency would be 64 + 1 = 65us.

Quoted 2 lines

I also assume that it can power down individual ports... and enter rest state if no links are up.

I don't think this is quite true -- a Link can't enter L1 unless the Ports on both ends have L1 enabled, so I don't think it makes sense to talk about an individual Port being in L1.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

The "lspci without my patches" [1] shows L1 enabled for the shared Link from 00:01.2 --- 01:00.0 and for the Link to 03:00.0 (I211), but not for the Link to 04:00.x (Realtek).

Quoted 6 lines
Quoted 4 lines

Per my analysis above, that looks like it *should* be a safe configuration. 03:00.0 can tolerate 64us, actual is <33us. 04:00.0 can tolerate 64us, actual should be <32us since only the shared Link is in L1.

See above.

As I said above, if we enabled L1 only on the shared Link from 00:01.2 to 01:00.0, the exit latency should be acceptable. In that case, a TLP from 04:00.x would see only 32us of latency:

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us

and 04:00.x can tolerate 64us.

Quoted 7 lines
Quoted 5 lines

However, the commit log at [2] shows L1 *enabled* for both the shared Link from 00:01.2 --- 01:00.0 and the 02:04.0 --- 04:00.x Link, and that would definitely be a problem.

Can you explain the differences between [1] and [2]?

I don't understand which sections you're referring to.

[1] is the "lspci without my patches" attachment of bugzilla #209725, which is supposed to show the problem this patch solves. We're talking about the path to 04:00.x, and [1] show this:

01:00.2 L1+
01:00.0 L1+
02:04.0 L1-
04:00.0 L1-

AFAICT, that should be a legal configuration as far as 04:00.0 is concerned, so it's not a reason for this patch.

[2] is a previous posting of this same patch, and its commit log includes information about the same path to 04:00.x, but the "LnkCtl Before" column shows:

01:00.2 L1+
01:00.0 L1+
02:04.0 L1+
04:00.0 L1+

I don't know why [1] shows L1 disabled on the downstream Link, while
[2] shows L1 *enabled* on the same Link.

Quoted 38 lines
Quoted 29 lines
Quoted 3 lines

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

I started working through this info, too, but there's not enough information to tell what difference this patch makes. The attachments compare:

1) CONFIG_PCIEASPM_DEFAULT=y without the patch [3] and
2) CONFIG_PCIEASPM_POWERSAVE=y *with* the patch [4]

Obviously CONFIG_PCIEASPM_POWERSAVE=y will configure things differently than CONFIG_PCIEASPM_DEFAULT=y, so we can't tell what changes are due to the config change and what are due to the patch.

The lspci *with* the patch ([4]) shows L0s and L1 enabled at almost every possible place. Here are the Links, how they're configured, and my analysis of the exit latencies vs acceptable latencies:

  00:01.1 --- 01:00.0      L1+ (                  L1 <64us vs unl)
  00:01.2 --- 02:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.3 --- 03:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.4 --- 04:00.0      L1+ (                  L1 <64us vs unl)

00:08.1 --- 05:00.x L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl) 00:08.2 --- 06:00.0 L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl)

So I can't tell what change prevents the freeze. I would expect the patch would cause us to *disable* L0s or L1 somewhere.

Quoted 8 lines

The only place [4] shows ASPM disabled is for 05:00.1. The spec says we should program the same value in all functions of a multi-function device. This is a non-ARI device, so "only capabilities enabled in all functions are enabled for the component as a whole." That would mean that L0s and L1 are effectively disabled for 05:00.x even though 05:00.0 claims they're enabled. But the latencies say ASPM L0s and L1 should be safe to be enabled. This looks like another bug that's probably unrelated.

Quoted 3 lines

I don't think it's unrelated, i suspect it's how PCIe works with multiple links... a device can cause some kind of head of queue stalling - i don't know how but it really looks like it.

The text in quotes above is straight out of the spec (PCIe r5.0, sec 7.5.3.7). Either the device works that way or it's not compliant.

The OS configures ASPM based on the requirements and capabilities advertised by the device. If a device has any head of queue stalling or similar issues, those must be comprehended in the numbers advertised by the device. It's not up to the OS to speculate about issues like that.

Quoted 5 lines
Quoted 2 lines

The patch might be correct; I haven't actually analyzed the code. But the commit log doesn't make sense to me yet.

I personally don't think that all this PCI information is required, the linux kernel is currently doing it wrong according to the spec.

We're trying to establish exactly *what* Linux is doing wrong. So far we don't have a good explanation of that.

Based on [1], in the path to 03:00.0, both Links have L1 enabled, with an exit latency of <33us, and 03:00.0 can tolerate 64us. That should work fine.

Also based on [1], in the path to 04:00.x, the upstream Link has L1 enabled and the downstream Link has L1 disabled, for an exit latency of <32us, and 04:00.0 can tolerate 64us. That should also work fine.

(Alternately, disabling L1 on the upstream Link and enabling it on the downstream Link should have an exit latency of <64us and 04:00.0 can tolerate 64us, so that should work fine, too.)

Quoted 4 lines

Also, since it's clearly doing the wrong thing, I'm worried that dists will take a kernel enable aspm and there will be alot of bugreports of non-booting systems or other weird issues... And the culprit was known all along.

There's clearly a problem on your system, but I don't know yet whether Linux is doing something wrong, a device in your system is designed incorrectly, or a device is designed correctly but the instance in your system is defective.

Quoted 1 line

It's been five months...

I apologize for the delay. ASPM is a subtle area of PCIe, the Linux code is complicated, and we have a long history of issues with it. I want to fix the problem, but I want to make sure we do it in a way that matches the spec so the fix applies to all systems. I don't want a magic fix that fixes your system in a way I don't quite understand.

Obviously *you* understand this, so hopefully it's just a matter of pounding it through my thick skull :)

Quoted 63 lines
Quoted 63 lines

[1] https://bugzilla.kernel.org/attachment.cgi?id=293047
[2] https://lore.kernel.org/linux-pci/20201007132808.647589-1-ian.kumlien at gmail.com/
[3] https://bugzilla.kernel.org/attachment.cgi?id=292955
[4] https://bugzilla.kernel.org/attachment.cgi?id=292957

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
      struct aspm_latency *acceptable;
      struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                  (link->latency_dw.l0s > acceptable->l0s))
                      link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
              /*
               * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 ?s from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
               *
               * The exit latencies for L1 substates are not advertised
               * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
               * L1 exit latencies advertised by a device include L1
               * substate latencies (and hence do not do any check).
               */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+                     l1_switch_latency += 1000;
+             }
 
              link = link->parent;
      }
--

2.29.1

#24 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 90 lines · Bjorn Helgaas

[+cc Jesse, Tony, David, Jakub, Heiner, lists in case there's an ASPM issue with I211 or Realtek NICs. Beginning of thread:
https://lore.kernel.org/r/20201024205548.1837770-1-ian.kumlien at gmail.com

Short story: Ian has:

Root Port --- Switch --- I211 NIC
\-- multifunction Realtek NIC, etc

and the I211 performance is poor with ASPM L1 enabled on both links in the path to it. The patch here disables ASPM on the upstream link and fixes the performance, but AFAICT the devices in that path give us no reason to disable L1. If I understand the spec correctly, the Realtek device should not be relevant to the I211 path.]

Quoted 67 lines · Ian Kumlien
Quoted 56 lines · Bjorn Helgaas
Quoted 24 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1?s from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.
...

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 7 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

There are two paths here that share a Link:

00:01.2 --- 01:00.0 -- 02:03.0 --- 03:00.0 I211 NIC 00:01.2 --- 01:00.0 -- 02:04.0 --- 04:00.x multifunction Realtek

1) The path to the I211 NIC includes four Ports and two Links (the connection between 01:00.0 and 02:03.0 is internal Switch routing, not a Link).

The Ports advertise L1 exit latencies of <32us, <32us, <32us, <16us. If both Links are in L1 and 03:00.0 initiates L1 exit@T, 01:00.0 initiates L1 exit at T + 1. A TLP from 03:00.0 may see up to 1 + 32 = 33us of L1 exit latency.

The NIC can tolerate up to 64us of L1 exit latency, so it is safe to enable L1 for both Links.

2) The path to the Realtek device is similar except that the Realtek L1 exit latency is <64us. If both Links are in L1 and 04:00.x initiates L1 exit at T, 01:00.0 again initiates L1 exit at T + 1, but a TLP from 04:00.x may see up to 1 + 64 = 65us of L1 exit latency.

Quoted 4 lines

The Realtek device can only tolerate 64us of latency, so it is not safe to enable L1 for both Links. It should be safe to enable L1 on the shared link because the exit latency for that link would be <32us.

04:00.0:
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us

Quoted 5 lines

So maximum latency for the entire link has to be <64 us For the device to leave L1 ASPM takes <64us

So the device itself is the slowest entry along the link, which means that nothing else along that path can have ASPM enabled

Quoted 7 lines

Yes. That's what I said above: "it is not safe to enable L1 for both Links." Unless I'm missing something, we agree on that.

I also said that it should be safe to enable L1 on the shared Link (from 00:01.2 to 01:00.0) because if the downstream Link is always in L0, the exit latency of the shared Link should be <32us, and 04:00.x can tolerate 64us.

Exit latency of shared link would be max of link, ie 64 + L1-hops, not 32

Quoted 46 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

Quoted 7 lines
Quoted 4 lines

[Nit: I don't think we should add 1 for the 02:04.0 -- 01:00.0 piece because that's internal Switch routing, not a Link. But even without that extra microsecond, this path does exceed the acceptable latency since 1 + 64 = 65us, and 04:00.0 can only tolerate 64us.]

It does report L1 ASPM on both ends, so the links will be counted as such in the code.

This is a bit of a tangent and we shouldn't get too wrapped up in it. This is a confusing aspect of PCIe. We're talking about this path:

00:01.2 --- [01:00.0 -- 02:04.0] --- 04:00.x multifunction Realtek

This path only contains two Links. The first one is 00:01.2 --- 01:00.0, and the second one is 02:04.0 --- 04:00.x.

01:00.0 is a Switch Upstream Port and 02:04.0 is a Switch Downstream Port. The connection between them is not a Link; it is some internal wiring of the Switch that is completely opaque to software.

The ASPM information and knobs in 01:00.0 apply to the Link on its upstream side, and the ASPM info and knobs in 02:04.0 apply to the Link on its downstream side.

The example in sec 5.4.1.2.2 contains three Links. The L1 exit latency for the Link is the max of the exit latencies at each end:

Link 1: max(32, 8) = 32us
Link 2: max(8, 32) = 32us
Link 3: max(32, 8) = 32us

The total delay for a TLP starting at the downstream end of Link 3 is 32 + 2 = 32us.

Quoted 7 lines

In the path to your 04:00.x Realtek device:

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us Link 2 (from 02:04.0 to 04:00.x): max(32, 64) = 64us

If L1 were enabled on both Links, the exit latency would be 64 + 1 = 65us.

So one line to be removed from the changelog, i assume... And yes, the code handles that - first disable is 01:00.0 <-> 00:01.2

Quoted 23 lines
Quoted 2 lines

I also assume that it can power down individual ports... and enter rest state if no links are up.

I don't think this is quite true -- a Link can't enter L1 unless the Ports on both ends have L1 enabled, so I don't think it makes sense to talk about an individual Port being in L1.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

The "lspci without my patches" [1] shows L1 enabled for the shared Link from 00:01.2 --- 01:00.0 and for the Link to 03:00.0 (I211), but not for the Link to 04:00.x (Realtek).

Quoted 6 lines
Quoted 4 lines

Per my analysis above, that looks like it *should* be a safe configuration. 03:00.0 can tolerate 64us, actual is <33us. 04:00.0 can tolerate 64us, actual should be <32us since only the shared Link is in L1.

See above.

Quoted 7 lines

As I said above, if we enabled L1 only on the shared Link from 00:01.2 to 01:00.0, the exit latency should be acceptable. In that case, a TLP from 04:00.x would see only 32us of latency:

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us

and 04:00.x can tolerate 64us.

But, again, you're completely ignoring the full link, ie 04:00.x would also have to power on.

Quoted 12 lines
Quoted 7 lines
Quoted 5 lines

However, the commit log at [2] shows L1 *enabled* for both the shared Link from 00:01.2 --- 01:00.0 and the 02:04.0 --- 04:00.x Link, and that would definitely be a problem.

Can you explain the differences between [1] and [2]?

I don't understand which sections you're referring to.

[1] is the "lspci without my patches" attachment of bugzilla #209725, which is supposed to show the problem this patch solves. We're talking about the path to 04:00.x, and [1] show this:

Quoted 7 lines

01:00.2 L1+
01:00.0 L1+
02:04.0 L1-
04:00.0 L1-

AFAICT, that should be a legal configuration as far as 04:00.0 is concerned, so it's not a reason for this patch.

Actually, no, maximum path latency 64us

04:00.0 wakeup latency == 64us

Again, as stated, it can't be behind any sleeping L1 links

Quoted 11 lines

[2] is a previous posting of this same patch, and its commit log includes information about the same path to 04:00.x, but the "LnkCtl Before" column shows:

01:00.2 L1+
01:00.0 L1+
02:04.0 L1+
04:00.0 L1+

I don't know why [1] shows L1 disabled on the downstream Link, while
[2] shows L1 *enabled* on the same Link.

From the data they look switched.

Quoted 51 lines
Quoted 38 lines
Quoted 29 lines
Quoted 3 lines

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

I started working through this info, too, but there's not enough information to tell what difference this patch makes. The attachments compare:

1) CONFIG_PCIEASPM_DEFAULT=y without the patch [3] and
2) CONFIG_PCIEASPM_POWERSAVE=y *with* the patch [4]

Obviously CONFIG_PCIEASPM_POWERSAVE=y will configure things differently than CONFIG_PCIEASPM_DEFAULT=y, so we can't tell what changes are due to the config change and what are due to the patch.

The lspci *with* the patch ([4]) shows L0s and L1 enabled at almost every possible place. Here are the Links, how they're configured, and my analysis of the exit latencies vs acceptable latencies:

  00:01.1 --- 01:00.0      L1+ (                  L1 <64us vs unl)
  00:01.2 --- 02:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.3 --- 03:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.4 --- 04:00.0      L1+ (                  L1 <64us vs unl)

00:08.1 --- 05:00.x L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl) 00:08.2 --- 06:00.0 L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl)

So I can't tell what change prevents the freeze. I would expect the patch would cause us to *disable* L0s or L1 somewhere.

Quoted 8 lines

The only place [4] shows ASPM disabled is for 05:00.1. The spec says we should program the same value in all functions of a multi-function device. This is a non-ARI device, so "only capabilities enabled in all functions are enabled for the component as a whole." That would mean that L0s and L1 are effectively disabled for 05:00.x even though 05:00.0 claims they're enabled. But the latencies say ASPM L0s and L1 should be safe to be enabled. This looks like another bug that's probably unrelated.

Quoted 3 lines

I don't think it's unrelated, i suspect it's how PCIe works with multiple links... a device can cause some kind of head of queue stalling - i don't know how but it really looks like it.

The text in quotes above is straight out of the spec (PCIe r5.0, sec 7.5.3.7). Either the device works that way or it's not compliant.

The OS configures ASPM based on the requirements and capabilities advertised by the device. If a device has any head of queue stalling or similar issues, those must be comprehended in the numbers advertised by the device. It's not up to the OS to speculate about issues like that.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The patch might be correct; I haven't actually analyzed the code. But the commit log doesn't make sense to me yet.

I personally don't think that all this PCI information is required, the linux kernel is currently doing it wrong according to the spec.

We're trying to establish exactly *what* Linux is doing wrong. So far we don't have a good explanation of that.

Yes we do, linux counts hops + max for "link" while what should be done is counting hops + max for path

Quoted 7 lines

Based on [1], in the path to 03:00.0, both Links have L1 enabled, with an exit latency of <33us, and 03:00.0 can tolerate 64us. That should work fine.

Also based on [1], in the path to 04:00.x, the upstream Link has L1 enabled and the downstream Link has L1 disabled, for an exit latency of <32us, and 04:00.0 can tolerate 64us. That should also work fine.

Again, ignoring the exit latendy for 04:00.0

Quoted 3 lines

(Alternately, disabling L1 on the upstream Link and enabling it on the downstream Link should have an exit latency of <64us and 04:00.0 can tolerate 64us, so that should work fine, too.)

Then nothing else can have L1 aspm enabled

Quoted 9 lines
Quoted 4 lines

Also, since it's clearly doing the wrong thing, I'm worried that dists will take a kernel enable aspm and there will be alot of bugreports of non-booting systems or other weird issues... And the culprit was known all along.

There's clearly a problem on your system, but I don't know yet whether Linux is doing something wrong, a device in your system is designed incorrectly, or a device is designed correctly but the instance in your system is defective.

According to the spec it is, there is a explanation of how to calculate the exit latency
and when you implement that, which i did (before knowing the actual spec) then it works...

Quoted 10 lines
Quoted 1 line

It's been five months...

I apologize for the delay. ASPM is a subtle area of PCIe, the Linux code is complicated, and we have a long history of issues with it. I want to fix the problem, but I want to make sure we do it in a way that matches the spec so the fix applies to all systems. I don't want a magic fix that fixes your system in a way I don't quite understand.

Obviously *you* understand this, so hopefully it's just a matter of pounding it through my thick skull :)

I only understand what I've been forced to understand - and I do leverage the existing code without
knowing what it does underneath, I only look at the links maximum latency and make sure that I keep
the maximum latency along the path and not just link for link

once you realise that the max allowed latency is buffer dependent - then this becomes obviously correct,
and then the pcie spec showed it as being correct as well... so...

Quoted 63 lines
Quoted 63 lines
Quoted 63 lines

[1] https://bugzilla.kernel.org/attachment.cgi?id=293047
[2] https://lore.kernel.org/linux-pci/20201007132808.647589-1-ian.kumlien at gmail.com/
[3] https://bugzilla.kernel.org/attachment.cgi?id=292955
[4] https://bugzilla.kernel.org/attachment.cgi?id=292957

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
      struct aspm_latency *acceptable;
      struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                  (link->latency_dw.l0s > acceptable->l0s))
                      link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
              /*
               * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 ?s from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
               *
               * The exit latencies for L1 substates are not advertised
               * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
               * L1 exit latencies advertised by a device include L1
               * substate latencies (and hence do not do any check).
               */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+                     l1_switch_latency += 1000;
+             }
 
              link = link->parent;
      }
--

2.29.1

#25 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 96 lines · Ian Kumlien
Quoted 95 lines · Bjorn Helgaas

[+cc Jesse, Tony, David, Jakub, Heiner, lists in case there's an ASPM issue with I211 or Realtek NICs. Beginning of thread:
https://lore.kernel.org/r/20201024205548.1837770-1-ian.kumlien at gmail.com

Short story: Ian has:

Root Port --- Switch --- I211 NIC
\-- multifunction Realtek NIC, etc

and the I211 performance is poor with ASPM L1 enabled on both links in the path to it. The patch here disables ASPM on the upstream link and fixes the performance, but AFAICT the devices in that path give us no reason to disable L1. If I understand the spec correctly, the Realtek device should not be relevant to the I211 path.]

Quoted 69 lines · Ian Kumlien
Quoted 58 lines · Bjorn Helgaas
Quoted 25 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1?s from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.
...

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 8 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

There are two paths here that share a Link:

00:01.2 --- 01:00.0 -- 02:03.0 --- 03:00.0 I211 NIC 00:01.2 --- 01:00.0 -- 02:04.0 --- 04:00.x multifunction Realtek

1) The path to the I211 NIC includes four Ports and two Links (the connection between 01:00.0 and 02:03.0 is internal Switch routing, not a Link).

The Ports advertise L1 exit latencies of <32us, <32us, <32us, <16us. If both Links are in L1 and 03:00.0 initiates L1 exit@T, 01:00.0 initiates L1 exit at T + 1. A TLP from 03:00.0 may see up to 1 + 32 = 33us of L1 exit latency.

The NIC can tolerate up to 64us of L1 exit latency, so it is safe to enable L1 for both Links.

2) The path to the Realtek device is similar except that the Realtek L1 exit latency is <64us. If both Links are in L1 and 04:00.x initiates L1 exit at T, 01:00.0 again initiates L1 exit at T + 1, but a TLP from 04:00.x may see up to 1 + 64 = 65us of L1 exit latency.

Quoted 4 lines

The Realtek device can only tolerate 64us of latency, so it is not safe to enable L1 for both Links. It should be safe to enable L1 on the shared link because the exit latency for that link would be <32us.

04:00.0:
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us

Quoted 5 lines

So maximum latency for the entire link has to be <64 us For the device to leave L1 ASPM takes <64us

So the device itself is the slowest entry along the link, which means that nothing else along that path can have ASPM enabled

Yes. That's what I said above: "it is not safe to enable L1 for both Links." Unless I'm missing something, we agree on that.

Quoted 6 lines
Quoted 4 lines

I also said that it should be safe to enable L1 on the shared Link (from 00:01.2 to 01:00.0) because if the downstream Link is always in L0, the exit latency of the shared Link should be <32us, and 04:00.x can tolerate 64us.

Exit latency of shared link would be max of link, ie 64 + L1-hops, not 32

I don't think this is true. The path from 00:01.2 to 04:00.x includes two Links, and they are independent. The exit latency for each Link depends only on the Port at each end:

Link 1 (depends on 00:01.2 and 01:00.0): max(32, 32) = 32us Link 2 (depends on 02:04.0 and 04:00.x): max(32, 64) = 64us

If L1 is enabled for Link 1 and disabled for Link 2, Link 2 will remain in L0 so it has no L1 exit latency, and the exit latency of the entire path should be 32us.

Quoted 84 lines
Quoted 46 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

Quoted 7 lines
Quoted 4 lines

[Nit: I don't think we should add 1 for the 02:04.0 -- 01:00.0 piece because that's internal Switch routing, not a Link. But even without that extra microsecond, this path does exceed the acceptable latency since 1 + 64 = 65us, and 04:00.0 can only tolerate 64us.]

It does report L1 ASPM on both ends, so the links will be counted as such in the code.

This is a bit of a tangent and we shouldn't get too wrapped up in it. This is a confusing aspect of PCIe. We're talking about this path:

00:01.2 --- [01:00.0 -- 02:04.0] --- 04:00.x multifunction Realtek

This path only contains two Links. The first one is 00:01.2 --- 01:00.0, and the second one is 02:04.0 --- 04:00.x.

01:00.0 is a Switch Upstream Port and 02:04.0 is a Switch Downstream Port. The connection between them is not a Link; it is some internal wiring of the Switch that is completely opaque to software.

The ASPM information and knobs in 01:00.0 apply to the Link on its upstream side, and the ASPM info and knobs in 02:04.0 apply to the Link on its downstream side.

The example in sec 5.4.1.2.2 contains three Links. The L1 exit latency for the Link is the max of the exit latencies at each end:

Link 1: max(32, 8) = 32us
Link 2: max(8, 32) = 32us
Link 3: max(32, 8) = 32us

The total delay for a TLP starting at the downstream end of Link 3 is 32 + 2 = 32us.

Quoted 7 lines

In the path to your 04:00.x Realtek device:

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us Link 2 (from 02:04.0 to 04:00.x): max(32, 64) = 64us

If L1 were enabled on both Links, the exit latency would be 64 + 1 = 65us.

So one line to be removed from the changelog, i assume... And yes, the code handles that - first disable is 01:00.0 <-> 00:01.2

Quoted 27 lines
Quoted 2 lines

I also assume that it can power down individual ports... and enter rest state if no links are up.

I don't think this is quite true -- a Link can't enter L1 unless the Ports on both ends have L1 enabled, so I don't think it makes sense to talk about an individual Port being in L1.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

The "lspci without my patches" [1] shows L1 enabled for the shared Link from 00:01.2 --- 01:00.0 and for the Link to 03:00.0 (I211), but not for the Link to 04:00.x (Realtek).

Quoted 6 lines
Quoted 4 lines

Per my analysis above, that looks like it *should* be a safe configuration. 03:00.0 can tolerate 64us, actual is <33us. 04:00.0 can tolerate 64us, actual should be <32us since only the shared Link is in L1.

See above.

As I said above, if we enabled L1 only on the shared Link from 00:01.2 to 01:00.0, the exit latency should be acceptable. In that case, a TLP from 04:00.x would see only 32us of latency:

Quoted 6 lines
Quoted 3 lines

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us

and 04:00.x can tolerate 64us.

But, again, you're completely ignoring the full link, ie 04:00.x would also have to power on.

I think you're using "the full link" to refer to the entire path from 00:01.2 to 04:00.x. In PCIe, a "Link" directly connects two Ports. It doesn't refer to the entire path.

No, if L1 is disabled on 02:04.0 and 04:00.x (as Linux apparently does by default), the Link between them never enters L1, so there is no power-on for this Link.

Quoted 17 lines
Quoted 17 lines
Quoted 7 lines
Quoted 5 lines

However, the commit log at [2] shows L1 *enabled* for both the shared Link from 00:01.2 --- 01:00.0 and the 02:04.0 --- 04:00.x Link, and that would definitely be a problem.

Can you explain the differences between [1] and [2]?

I don't understand which sections you're referring to.

[1] is the "lspci without my patches" attachment of bugzilla #209725, which is supposed to show the problem this patch solves. We're talking about the path to 04:00.x, and [1] show this:

01:00.2 L1+ # <-- my typo here, should be 00:01.2
01:00.0 L1+
02:04.0 L1-
04:00.0 L1-

Quoted 8 lines
Quoted 2 lines

AFAICT, that should be a legal configuration as far as 04:00.0 is concerned, so it's not a reason for this patch.

Actually, no, maximum path latency 64us

04:00.0 wakeup latency == 64us

Again, as stated, it can't be behind any sleeping L1 links

It would be pointless for a device to advertise L1 support if it could never be used. 04:00.0 advertises that it can tolerate L1 latency of 64us and that it can exit L1 in 64us or less. So it *can* be behind a Link in L1 as long as nothing else in the path adds more latency.

Quoted 68 lines
Quoted 11 lines

[2] is a previous posting of this same patch, and its commit log includes information about the same path to 04:00.x, but the "LnkCtl Before" column shows:

01:00.2 L1+ # <-- my typo here, should be 00:01.2
01:00.0 L1+
02:04.0 L1+
04:00.0 L1+

I don't know why [1] shows L1 disabled on the downstream Link, while
[2] shows L1 *enabled* on the same Link.

From the data they look switched.

Quoted 54 lines
Quoted 38 lines
Quoted 29 lines
Quoted 3 lines

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

I started working through this info, too, but there's not enough information to tell what difference this patch makes. The attachments compare:

1) CONFIG_PCIEASPM_DEFAULT=y without the patch [3] and
2) CONFIG_PCIEASPM_POWERSAVE=y *with* the patch [4]

Obviously CONFIG_PCIEASPM_POWERSAVE=y will configure things differently than CONFIG_PCIEASPM_DEFAULT=y, so we can't tell what changes are due to the config change and what are due to the patch.

The lspci *with* the patch ([4]) shows L0s and L1 enabled at almost every possible place. Here are the Links, how they're configured, and my analysis of the exit latencies vs acceptable latencies:

  00:01.1 --- 01:00.0      L1+ (                  L1 <64us vs unl)
  00:01.2 --- 02:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.3 --- 03:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.4 --- 04:00.0      L1+ (                  L1 <64us vs unl)

00:08.1 --- 05:00.x L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl) 00:08.2 --- 06:00.0 L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl)

So I can't tell what change prevents the freeze. I would expect the patch would cause us to *disable* L0s or L1 somewhere.

Quoted 8 lines

The only place [4] shows ASPM disabled is for 05:00.1. The spec says we should program the same value in all functions of a multi-function device. This is a non-ARI device, so "only capabilities enabled in all functions are enabled for the component as a whole." That would mean that L0s and L1 are effectively disabled for 05:00.x even though 05:00.0 claims they're enabled. But the latencies say ASPM L0s and L1 should be safe to be enabled. This looks like another bug that's probably unrelated.

Quoted 3 lines

I don't think it's unrelated, i suspect it's how PCIe works with multiple links... a device can cause some kind of head of queue stalling - i don't know how but it really looks like it.

The text in quotes above is straight out of the spec (PCIe r5.0, sec 7.5.3.7). Either the device works that way or it's not compliant.

The OS configures ASPM based on the requirements and capabilities advertised by the device. If a device has any head of queue stalling or similar issues, those must be comprehended in the numbers advertised by the device. It's not up to the OS to speculate about issues like that.

Quoted 3 lines
Quoted 2 lines

The patch might be correct; I haven't actually analyzed the code. But the commit log doesn't make sense to me yet.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I personally don't think that all this PCI information is required, the linux kernel is currently doing it wrong according to the spec.

We're trying to establish exactly *what* Linux is doing wrong. So far we don't have a good explanation of that.

Yes we do, linux counts hops + max for "link" while what should be done is counting hops + max for path

I think you're saying we need to include L1 exit latency even for Links where L1 is disabled. I don't think we should include those.

Quoted 8 lines
Quoted 7 lines

Based on [1], in the path to 03:00.0, both Links have L1 enabled, with an exit latency of <33us, and 03:00.0 can tolerate 64us. That should work fine.

Also based on [1], in the path to 04:00.x, the upstream Link has L1 enabled and the downstream Link has L1 disabled, for an exit latency of <32us, and 04:00.0 can tolerate 64us. That should also work fine.

Quoted 7 lines

Again, ignoring the exit latendy for 04:00.0

Quoted 3 lines

(Alternately, disabling L1 on the upstream Link and enabling it on the downstream Link should have an exit latency of <64us and 04:00.0 can tolerate 64us, so that should work fine, too.)

Then nothing else can have L1 aspm enabled

Yes, as I said, we should be able to enable L1 on either of the Links in the path to 04:00.x, but not both.

The original problem here is not with the Realtek device at 04:00.x but with the I211 NIC at 03:00.0. So we also need to figure out what the connection is. Does the same I211 performance problem occur if you remove the Realtek device from the system?

03:00.0 can tolerate 64us of latency, so even if L1 is enabled on both Links leading to it, the path exit latency would be <33us, which should be fine.

Quoted 100 lines
Quoted 9 lines
Quoted 4 lines

Also, since it's clearly doing the wrong thing, I'm worried that dists will take a kernel enable aspm and there will be alot of bugreports of non-booting systems or other weird issues... And the culprit was known all along.

There's clearly a problem on your system, but I don't know yet whether Linux is doing something wrong, a device in your system is designed incorrectly, or a device is designed correctly but the instance in your system is defective.

According to the spec it is, there is a explanation of how to calculate the exit latency
and when you implement that, which i did (before knowing the actual spec) then it works...

Quoted 10 lines
Quoted 1 line

It's been five months...

I apologize for the delay. ASPM is a subtle area of PCIe, the Linux code is complicated, and we have a long history of issues with it. I want to fix the problem, but I want to make sure we do it in a way that matches the spec so the fix applies to all systems. I don't want a magic fix that fixes your system in a way I don't quite understand.

Obviously *you* understand this, so hopefully it's just a matter of pounding it through my thick skull :)

I only understand what I've been forced to understand - and I do leverage the existing code without
knowing what it does underneath, I only look at the links maximum latency and make sure that I keep
the maximum latency along the path and not just link for link

once you realise that the max allowed latency is buffer dependent - then this becomes obviously correct,
and then the pcie spec showed it as being correct as well... so...

Quoted 63 lines
Quoted 63 lines
Quoted 63 lines

[1] https://bugzilla.kernel.org/attachment.cgi?id=293047
[2] https://lore.kernel.org/linux-pci/20201007132808.647589-1-ian.kumlien at gmail.com/
[3] https://bugzilla.kernel.org/attachment.cgi?id=292955
[4] https://bugzilla.kernel.org/attachment.cgi?id=292957

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
      struct aspm_latency *acceptable;
      struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                  (link->latency_dw.l0s > acceptable->l0s))
                      link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
              /*
               * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 ?s from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
               *
               * The exit latencies for L1 substates are not advertised
               * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
               * L1 exit latencies advertised by a device include L1
               * substate latencies (and hence do not do any check).
               */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+                     l1_switch_latency += 1000;
+             }
 
              link = link->parent;
      }
--

2.29.1

#26 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 103 lines · Bjorn Helgaas
Quoted 101 lines · Ian Kumlien
Quoted 99 lines · Bjorn Helgaas

[+cc Jesse, Tony, David, Jakub, Heiner, lists in case there's an ASPM issue with I211 or Realtek NICs. Beginning of thread:
https://lore.kernel.org/r/20201024205548.1837770-1-ian.kumlien at gmail.com

Short story: Ian has:

Root Port --- Switch --- I211 NIC
\-- multifunction Realtek NIC, etc

and the I211 performance is poor with ASPM L1 enabled on both links in the path to it. The patch here disables ASPM on the upstream link and fixes the performance, but AFAICT the devices in that path give us no reason to disable L1. If I understand the spec correctly, the Realtek device should not be relevant to the I211 path.]

Quoted 69 lines · Ian Kumlien
Quoted 58 lines · Bjorn Helgaas
Quoted 25 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1?s from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.
...

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 8 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

There are two paths here that share a Link:

00:01.2 --- 01:00.0 -- 02:03.0 --- 03:00.0 I211 NIC 00:01.2 --- 01:00.0 -- 02:04.0 --- 04:00.x multifunction Realtek

1) The path to the I211 NIC includes four Ports and two Links (the connection between 01:00.0 and 02:03.0 is internal Switch routing, not a Link).

The Ports advertise L1 exit latencies of <32us, <32us, <32us, <16us. If both Links are in L1 and 03:00.0 initiates L1 exit@T, 01:00.0 initiates L1 exit at T + 1. A TLP from 03:00.0 may see up to 1 + 32 = 33us of L1 exit latency.

The NIC can tolerate up to 64us of L1 exit latency, so it is safe to enable L1 for both Links.

2) The path to the Realtek device is similar except that the Realtek L1 exit latency is <64us. If both Links are in L1 and 04:00.x initiates L1 exit at T, 01:00.0 again initiates L1 exit at T + 1, but a TLP from 04:00.x may see up to 1 + 64 = 65us of L1 exit latency.

Quoted 4 lines

The Realtek device can only tolerate 64us of latency, so it is not safe to enable L1 for both Links. It should be safe to enable L1 on the shared link because the exit latency for that link would be <32us.

04:00.0:
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us

Quoted 5 lines

So maximum latency for the entire link has to be <64 us For the device to leave L1 ASPM takes <64us

So the device itself is the slowest entry along the link, which means that nothing else along that path can have ASPM enabled

Yes. That's what I said above: "it is not safe to enable L1 for both Links." Unless I'm missing something, we agree on that.

I also said that it should be safe to enable L1 on the shared Link (from 00:01.2 to 01:00.0) because if the downstream Link is always in L0, the exit latency of the shared Link should be <32us, and 04:00.x can tolerate 64us.

Quoted 5 lines
Quoted 1 line

Exit latency of shared link would be max of link, ie 64 + L1-hops, not 32

I don't think this is true. The path from 00:01.2 to 04:00.x includes two Links, and they are independent. The exit latency for each Link depends only on the Port at each end:

The full path is what is important, because that is the actual latency (which the current linux code doesn't do)

Quoted 6 lines

Link 1 (depends on 00:01.2 and 01:00.0): max(32, 32) = 32us Link 2 (depends on 02:04.0 and 04:00.x): max(32, 64) = 64us

If L1 is enabled for Link 1 and disabled for Link 2, Link 2 will remain in L0 so it has no L1 exit latency, and the exit latency of the entire path should be 32us.

My patch disables this so yes.

Quoted 91 lines
Quoted 84 lines
Quoted 46 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

Quoted 7 lines
Quoted 4 lines

[Nit: I don't think we should add 1 for the 02:04.0 -- 01:00.0 piece because that's internal Switch routing, not a Link. But even without that extra microsecond, this path does exceed the acceptable latency since 1 + 64 = 65us, and 04:00.0 can only tolerate 64us.]

It does report L1 ASPM on both ends, so the links will be counted as such in the code.

This is a bit of a tangent and we shouldn't get too wrapped up in it. This is a confusing aspect of PCIe. We're talking about this path:

00:01.2 --- [01:00.0 -- 02:04.0] --- 04:00.x multifunction Realtek

This path only contains two Links. The first one is 00:01.2 --- 01:00.0, and the second one is 02:04.0 --- 04:00.x.

01:00.0 is a Switch Upstream Port and 02:04.0 is a Switch Downstream Port. The connection between them is not a Link; it is some internal wiring of the Switch that is completely opaque to software.

The ASPM information and knobs in 01:00.0 apply to the Link on its upstream side, and the ASPM info and knobs in 02:04.0 apply to the Link on its downstream side.

The example in sec 5.4.1.2.2 contains three Links. The L1 exit latency for the Link is the max of the exit latencies at each end:

Link 1: max(32, 8) = 32us
Link 2: max(8, 32) = 32us
Link 3: max(32, 8) = 32us

The total delay for a TLP starting at the downstream end of Link 3 is 32 + 2 = 32us.

Quoted 7 lines

In the path to your 04:00.x Realtek device:

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us Link 2 (from 02:04.0 to 04:00.x): max(32, 64) = 64us

If L1 were enabled on both Links, the exit latency would be 64 + 1 = 65us.

So one line to be removed from the changelog, i assume... And yes, the code handles that - first disable is 01:00.0 <-> 00:01.2

Quoted 27 lines
Quoted 2 lines

I also assume that it can power down individual ports... and enter rest state if no links are up.

I don't think this is quite true -- a Link can't enter L1 unless the Ports on both ends have L1 enabled, so I don't think it makes sense to talk about an individual Port being in L1.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

The "lspci without my patches" [1] shows L1 enabled for the shared Link from 00:01.2 --- 01:00.0 and for the Link to 03:00.0 (I211), but not for the Link to 04:00.x (Realtek).

Quoted 6 lines
Quoted 4 lines

Per my analysis above, that looks like it *should* be a safe configuration. 03:00.0 can tolerate 64us, actual is <33us. 04:00.0 can tolerate 64us, actual should be <32us since only the shared Link is in L1.

See above.

As I said above, if we enabled L1 only on the shared Link from 00:01.2 to 01:00.0, the exit latency should be acceptable. In that case, a TLP from 04:00.x would see only 32us of latency:

Quoted 6 lines
Quoted 3 lines

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us

and 04:00.x can tolerate 64us.

But, again, you're completely ignoring the full link, ie 04:00.x would also have to power on.

Quoted 7 lines

I think you're using "the full link" to refer to the entire path from 00:01.2 to 04:00.x. In PCIe, a "Link" directly connects two Ports. It doesn't refer to the entire path.

No, if L1 is disabled on 02:04.0 and 04:00.x (as Linux apparently does by default), the Link between them never enters L1, so there is no power-on for this Link.

It doesn't do it by default, my patch does

Quoted 22 lines
Quoted 22 lines
Quoted 12 lines
Quoted 7 lines
Quoted 5 lines

However, the commit log at [2] shows L1 *enabled* for both the shared Link from 00:01.2 --- 01:00.0 and the 02:04.0 --- 04:00.x Link, and that would definitely be a problem.

Can you explain the differences between [1] and [2]?

I don't understand which sections you're referring to.

[1] is the "lspci without my patches" attachment of bugzilla #209725, which is supposed to show the problem this patch solves. We're talking about the path to 04:00.x, and [1] show this:

Quoted 7 lines

01:00.2 L1+ # <-- my typo here, should be 00:01.2
01:00.0 L1+
02:04.0 L1-
04:00.0 L1-

AFAICT, that should be a legal configuration as far as 04:00.0 is concerned, so it's not a reason for this patch.

Actually, no, maximum path latency 64us

Quoted 8 lines
Quoted 3 lines

04:00.0 wakeup latency == 64us

Again, as stated, it can't be behind any sleeping L1 links

It would be pointless for a device to advertise L1 support if it could never be used. 04:00.0 advertises that it can tolerate L1 latency of 64us and that it can exit L1 in 64us or less. So it *can* be behind a Link in L1 as long as nothing else in the path adds more latency.

Yes, as long as nothing along the entire path adds latency - and I didn't make the component
I can only say what it states, and we have to handle it.

Quoted 71 lines
Quoted 71 lines
Quoted 11 lines

[2] is a previous posting of this same patch, and its commit log includes information about the same path to 04:00.x, but the "LnkCtl Before" column shows:

01:00.2 L1+ # <-- my typo here, should be 00:01.2
01:00.0 L1+
02:04.0 L1+
04:00.0 L1+

I don't know why [1] shows L1 disabled on the downstream Link, while
[2] shows L1 *enabled* on the same Link.

From the data they look switched.

Quoted 57 lines
Quoted 38 lines
Quoted 29 lines
Quoted 3 lines

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

I started working through this info, too, but there's not enough information to tell what difference this patch makes. The attachments compare:

1) CONFIG_PCIEASPM_DEFAULT=y without the patch [3] and
2) CONFIG_PCIEASPM_POWERSAVE=y *with* the patch [4]

Obviously CONFIG_PCIEASPM_POWERSAVE=y will configure things differently than CONFIG_PCIEASPM_DEFAULT=y, so we can't tell what changes are due to the config change and what are due to the patch.

The lspci *with* the patch ([4]) shows L0s and L1 enabled at almost every possible place. Here are the Links, how they're configured, and my analysis of the exit latencies vs acceptable latencies:

  00:01.1 --- 01:00.0      L1+ (                  L1 <64us vs unl)
  00:01.2 --- 02:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.3 --- 03:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.4 --- 04:00.0      L1+ (                  L1 <64us vs unl)

00:08.1 --- 05:00.x L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl) 00:08.2 --- 06:00.0 L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl)

So I can't tell what change prevents the freeze. I would expect the patch would cause us to *disable* L0s or L1 somewhere.

Quoted 8 lines

The only place [4] shows ASPM disabled is for 05:00.1. The spec says we should program the same value in all functions of a multi-function device. This is a non-ARI device, so "only capabilities enabled in all functions are enabled for the component as a whole." That would mean that L0s and L1 are effectively disabled for 05:00.x even though 05:00.0 claims they're enabled. But the latencies say ASPM L0s and L1 should be safe to be enabled. This looks like another bug that's probably unrelated.

Quoted 3 lines

I don't think it's unrelated, i suspect it's how PCIe works with multiple links... a device can cause some kind of head of queue stalling - i don't know how but it really looks like it.

The text in quotes above is straight out of the spec (PCIe r5.0, sec 7.5.3.7). Either the device works that way or it's not compliant.

The OS configures ASPM based on the requirements and capabilities advertised by the device. If a device has any head of queue stalling or similar issues, those must be comprehended in the numbers advertised by the device. It's not up to the OS to speculate about issues like that.

Quoted 5 lines
Quoted 2 lines

The patch might be correct; I haven't actually analyzed the code. But the commit log doesn't make sense to me yet.

I personally don't think that all this PCI information is required, the linux kernel is currently doing it wrong according to the spec.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

We're trying to establish exactly *what* Linux is doing wrong. So far we don't have a good explanation of that.

Yes we do, linux counts hops + max for "link" while what should be done is counting hops + max for path

I think you're saying we need to include L1 exit latency even for Links where L1 is disabled. I don't think we should include those.

Nope, the code does not do that, it only adds the l1 latency on L1 enabled hops

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

Based on [1], in the path to 03:00.0, both Links have L1 enabled, with an exit latency of <33us, and 03:00.0 can tolerate 64us. That should work fine.

Also based on [1], in the path to 04:00.x, the upstream Link has L1 enabled and the downstream Link has L1 disabled, for an exit latency of <32us, and 04:00.0 can tolerate 64us. That should also work fine.

Again, ignoring the exit latendy for 04:00.0

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

(Alternately, disabling L1 on the upstream Link and enabling it on the downstream Link should have an exit latency of <64us and 04:00.0 can tolerate 64us, so that should work fine, too.)

Then nothing else can have L1 aspm enabled

Yes, as I said, we should be able to enable L1 on either of the Links in the path to 04:00.x, but not both.

The code works backwards and disables the first hop that exceeds the latency requirements -
we could argue that it should try to be smarter about it and try to disable a minimum amount of links
while still retaining the minimum latency but... It is what it is and it works when patched.

Quoted 4 lines

The original problem here is not with the Realtek device at 04:00.x but with the I211 NIC at 03:00.0. So we also need to figure out what the connection is. Does the same I211 performance problem occur if you remove the Realtek device from the system?

It's mounted on the motherboard, so no I can't remove it.

Quoted 3 lines

03:00.0 can tolerate 64us of latency, so even if L1 is enabled on both Links leading to it, the path exit latency would be <33us, which should be fine.

Yes, it "should be" but due to broken ASPM latency calculations we have some kind of
side effect that triggers a racecondition/sideeffect/bug that causes it to misbehave.

Since fixing the latency calculation fixes it, I'll leave the rest to someone with a logic
analyzer and a die-hard-fetish for pcie links - I can't debug it.

Quoted 100 lines
Quoted 100 lines
Quoted 9 lines
Quoted 4 lines

Also, since it's clearly doing the wrong thing, I'm worried that dists will take a kernel enable aspm and there will be alot of bugreports of non-booting systems or other weird issues... And the culprit was known all along.

There's clearly a problem on your system, but I don't know yet whether Linux is doing something wrong, a device in your system is designed incorrectly, or a device is designed correctly but the instance in your system is defective.

According to the spec it is, there is a explanation of how to calculate the exit latency
and when you implement that, which i did (before knowing the actual spec) then it works...

Quoted 10 lines
Quoted 1 line

It's been five months...

I apologize for the delay. ASPM is a subtle area of PCIe, the Linux code is complicated, and we have a long history of issues with it. I want to fix the problem, but I want to make sure we do it in a way that matches the spec so the fix applies to all systems. I don't want a magic fix that fixes your system in a way I don't quite understand.

Obviously *you* understand this, so hopefully it's just a matter of pounding it through my thick skull :)

I only understand what I've been forced to understand - and I do leverage the existing code without
knowing what it does underneath, I only look at the links maximum latency and make sure that I keep
the maximum latency along the path and not just link for link

once you realise that the max allowed latency is buffer dependent - then this becomes obviously correct,
and then the pcie spec showed it as being correct as well... so...

Quoted 63 lines
Quoted 63 lines
Quoted 63 lines

[1] https://bugzilla.kernel.org/attachment.cgi?id=293047
[2] https://lore.kernel.org/linux-pci/20201007132808.647589-1-ian.kumlien at gmail.com/
[3] https://bugzilla.kernel.org/attachment.cgi?id=292955
[4] https://bugzilla.kernel.org/attachment.cgi?id=292957

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
      struct aspm_latency *acceptable;
      struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                  (link->latency_dw.l0s > acceptable->l0s))
                      link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
              /*
               * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 ?s from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
               *
               * The exit latencies for L1 substates are not advertised
               * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
               * L1 exit latencies advertised by a device include L1
               * substate latencies (and hence do not do any check).
               */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+                     l1_switch_latency += 1000;
+             }
 
              link = link->parent;
      }
--

2.29.1

#27 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 103 lines · Ian Kumlien
Quoted 102 lines · Bjorn Helgaas
Quoted 101 lines · Ian Kumlien
Quoted 99 lines · Bjorn Helgaas

[+cc Jesse, Tony, David, Jakub, Heiner, lists in case there's an ASPM issue with I211 or Realtek NICs. Beginning of thread:
https://lore.kernel.org/r/20201024205548.1837770-1-ian.kumlien at gmail.com

Short story: Ian has:

Root Port --- Switch --- I211 NIC
\-- multifunction Realtek NIC, etc

and the I211 performance is poor with ASPM L1 enabled on both links in the path to it. The patch here disables ASPM on the upstream link and fixes the performance, but AFAICT the devices in that path give us no reason to disable L1. If I understand the spec correctly, the Realtek device should not be relevant to the I211 path.]

Quoted 69 lines · Ian Kumlien
Quoted 58 lines · Bjorn Helgaas
Quoted 25 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1?s from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.
...

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 8 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

There are two paths here that share a Link:

00:01.2 --- 01:00.0 -- 02:03.0 --- 03:00.0 I211 NIC 00:01.2 --- 01:00.0 -- 02:04.0 --- 04:00.x multifunction Realtek

1) The path to the I211 NIC includes four Ports and two Links (the connection between 01:00.0 and 02:03.0 is internal Switch routing, not a Link).

The Ports advertise L1 exit latencies of <32us, <32us, <32us, <16us. If both Links are in L1 and 03:00.0 initiates L1 exit@T, 01:00.0 initiates L1 exit at T + 1. A TLP from 03:00.0 may see up to 1 + 32 = 33us of L1 exit latency.

The NIC can tolerate up to 64us of L1 exit latency, so it is safe to enable L1 for both Links.

2) The path to the Realtek device is similar except that the Realtek L1 exit latency is <64us. If both Links are in L1 and 04:00.x initiates L1 exit at T, 01:00.0 again initiates L1 exit at T + 1, but a TLP from 04:00.x may see up to 1 + 64 = 65us of L1 exit latency.

Quoted 4 lines

The Realtek device can only tolerate 64us of latency, so it is not safe to enable L1 for both Links. It should be safe to enable L1 on the shared link because the exit latency for that link would be <32us.

04:00.0:
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us

Quoted 5 lines

So maximum latency for the entire link has to be <64 us For the device to leave L1 ASPM takes <64us

So the device itself is the slowest entry along the link, which means that nothing else along that path can have ASPM enabled

Yes. That's what I said above: "it is not safe to enable L1 for both Links." Unless I'm missing something, we agree on that.

I also said that it should be safe to enable L1 on the shared Link (from 00:01.2 to 01:00.0) because if the downstream Link is always in L0, the exit latency of the shared Link should be <32us, and 04:00.x can tolerate 64us.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Exit latency of shared link would be max of link, ie 64 + L1-hops, not 32

I don't think this is true. The path from 00:01.2 to 04:00.x includes two Links, and they are independent. The exit latency for each Link depends only on the Port at each end:

The full path is what is important, because that is the actual latency (which the current linux code doesn't do)

I think you're saying we need to include the 04:00.x exit latency of 64us even though L1 is not enabled for 04:00.x. I disagree; the L1 exit latency of Ports where L1 is disabled is irrelevant.

Quoted 104 lines
Quoted 6 lines

Link 1 (depends on 00:01.2 and 01:00.0): max(32, 32) = 32us Link 2 (depends on 02:04.0 and 04:00.x): max(32, 64) = 64us

If L1 is enabled for Link 1 and disabled for Link 2, Link 2 will remain in L0 so it has no L1 exit latency, and the exit latency of the entire path should be 32us.

My patch disables this so yes.

Quoted 95 lines
Quoted 84 lines
Quoted 46 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

Quoted 7 lines
Quoted 4 lines

[Nit: I don't think we should add 1 for the 02:04.0 -- 01:00.0 piece because that's internal Switch routing, not a Link. But even without that extra microsecond, this path does exceed the acceptable latency since 1 + 64 = 65us, and 04:00.0 can only tolerate 64us.]

It does report L1 ASPM on both ends, so the links will be counted as such in the code.

This is a bit of a tangent and we shouldn't get too wrapped up in it. This is a confusing aspect of PCIe. We're talking about this path:

00:01.2 --- [01:00.0 -- 02:04.0] --- 04:00.x multifunction Realtek

This path only contains two Links. The first one is 00:01.2 --- 01:00.0, and the second one is 02:04.0 --- 04:00.x.

01:00.0 is a Switch Upstream Port and 02:04.0 is a Switch Downstream Port. The connection between them is not a Link; it is some internal wiring of the Switch that is completely opaque to software.

The ASPM information and knobs in 01:00.0 apply to the Link on its upstream side, and the ASPM info and knobs in 02:04.0 apply to the Link on its downstream side.

The example in sec 5.4.1.2.2 contains three Links. The L1 exit latency for the Link is the max of the exit latencies at each end:

Link 1: max(32, 8) = 32us
Link 2: max(8, 32) = 32us
Link 3: max(32, 8) = 32us

The total delay for a TLP starting at the downstream end of Link 3 is 32 + 2 = 32us.

Quoted 7 lines

In the path to your 04:00.x Realtek device:

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us Link 2 (from 02:04.0 to 04:00.x): max(32, 64) = 64us

If L1 were enabled on both Links, the exit latency would be 64 + 1 = 65us.

So one line to be removed from the changelog, i assume... And yes, the code handles that - first disable is 01:00.0 <-> 00:01.2

Quoted 27 lines
Quoted 2 lines

I also assume that it can power down individual ports... and enter rest state if no links are up.

I don't think this is quite true -- a Link can't enter L1 unless the Ports on both ends have L1 enabled, so I don't think it makes sense to talk about an individual Port being in L1.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

The "lspci without my patches" [1] shows L1 enabled for the shared Link from 00:01.2 --- 01:00.0 and for the Link to 03:00.0 (I211), but not for the Link to 04:00.x (Realtek).

Quoted 6 lines
Quoted 4 lines

Per my analysis above, that looks like it *should* be a safe configuration. 03:00.0 can tolerate 64us, actual is <33us. 04:00.0 can tolerate 64us, actual should be <32us since only the shared Link is in L1.

See above.

As I said above, if we enabled L1 only on the shared Link from 00:01.2 to 01:00.0, the exit latency should be acceptable. In that case, a TLP from 04:00.x would see only 32us of latency:

Quoted 6 lines
Quoted 3 lines

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us

and 04:00.x can tolerate 64us.

But, again, you're completely ignoring the full link, ie 04:00.x would also have to power on.

I think you're using "the full link" to refer to the entire path from 00:01.2 to 04:00.x. In PCIe, a "Link" directly connects two Ports. It doesn't refer to the entire path.

Quoted 5 lines
Quoted 3 lines

No, if L1 is disabled on 02:04.0 and 04:00.x (as Linux apparently does by default), the Link between them never enters L1, so there is no power-on for this Link.

It doesn't do it by default, my patch does

I'm relying on [1], your "lspci without my patches" attachment named "lspci-5.9-mainline.txt", which shows:

02:04.0 LnkCtl: ASPM Disabled
04:00.0 LnkCtl: ASPM Disabled

so I assumed that was what Linux did by default.

Quoted 149 lines
Quoted 22 lines
Quoted 22 lines
Quoted 12 lines
Quoted 7 lines
Quoted 5 lines

However, the commit log at [2] shows L1 *enabled* for both the shared Link from 00:01.2 --- 01:00.0 and the 02:04.0 --- 04:00.x Link, and that would definitely be a problem.

Can you explain the differences between [1] and [2]?

I don't understand which sections you're referring to.

[1] is the "lspci without my patches" attachment of bugzilla #209725, which is supposed to show the problem this patch solves. We're talking about the path to 04:00.x, and [1] show this:

Quoted 7 lines

01:00.2 L1+ # <-- my typo here, should be 00:01.2
01:00.0 L1+
02:04.0 L1-
04:00.0 L1-

AFAICT, that should be a legal configuration as far as 04:00.0 is concerned, so it's not a reason for this patch.

Actually, no, maximum path latency 64us

Quoted 8 lines
Quoted 3 lines

04:00.0 wakeup latency == 64us

Again, as stated, it can't be behind any sleeping L1 links

It would be pointless for a device to advertise L1 support if it could never be used. 04:00.0 advertises that it can tolerate L1 latency of 64us and that it can exit L1 in 64us or less. So it *can* be behind a Link in L1 as long as nothing else in the path adds more latency.

Yes, as long as nothing along the entire path adds latency - and I didn't make the component
I can only say what it states, and we have to handle it.

Quoted 76 lines
Quoted 76 lines
Quoted 11 lines

[2] is a previous posting of this same patch, and its commit log includes information about the same path to 04:00.x, but the "LnkCtl Before" column shows:

01:00.2 L1+ # <-- my typo here, should be 00:01.2
01:00.0 L1+
02:04.0 L1+
04:00.0 L1+

I don't know why [1] shows L1 disabled on the downstream Link, while
[2] shows L1 *enabled* on the same Link.

From the data they look switched.

Quoted 62 lines
Quoted 42 lines
Quoted 32 lines
Quoted 3 lines

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

I started working through this info, too, but there's not enough information to tell what difference this patch makes. The attachments compare:

1) CONFIG_PCIEASPM_DEFAULT=y without the patch [3] and
2) CONFIG_PCIEASPM_POWERSAVE=y *with* the patch [4]

Obviously CONFIG_PCIEASPM_POWERSAVE=y will configure things differently than CONFIG_PCIEASPM_DEFAULT=y, so we can't tell what changes are due to the config change and what are due to the patch.

The lspci *with* the patch ([4]) shows L0s and L1 enabled at almost every possible place. Here are the Links, how they're configured, and my analysis of the exit latencies vs acceptable latencies:

  00:01.1 --- 01:00.0      L1+ (                  L1 <64us vs unl)
  00:01.2 --- 02:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.3 --- 03:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.4 --- 04:00.0      L1+ (                  L1 <64us vs unl)

00:08.1 --- 05:00.x L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl) 00:08.2 --- 06:00.0 L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl)

So I can't tell what change prevents the freeze. I would expect the patch would cause us to *disable* L0s or L1 somewhere.

Quoted 9 lines

The only place [4] shows ASPM disabled is for 05:00.1. The spec says we should program the same value in all functions of a multi-function device. This is a non-ARI device, so "only capabilities enabled in all functions are enabled for the component as a whole." That would mean that L0s and L1 are effectively disabled for 05:00.x even though 05:00.0 claims they're enabled. But the latencies say ASPM L0s and L1 should be safe to be enabled. This looks like another bug that's probably unrelated.

Quoted 3 lines

I don't think it's unrelated, i suspect it's how PCIe works with multiple links... a device can cause some kind of head of queue stalling - i don't know how but it really looks like it.

The text in quotes above is straight out of the spec (PCIe r5.0, sec 7.5.3.7). Either the device works that way or it's not compliant.

The OS configures ASPM based on the requirements and capabilities advertised by the device. If a device has any head of queue stalling or similar issues, those must be comprehended in the numbers advertised by the device. It's not up to the OS to speculate about issues like that.

Quoted 6 lines
Quoted 3 lines

The patch might be correct; I haven't actually analyzed the code. But the commit log doesn't make sense to me yet.

I personally don't think that all this PCI information is required, the linux kernel is currently doing it wrong according to the spec.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

We're trying to establish exactly *what* Linux is doing wrong. So far we don't have a good explanation of that.

Yes we do, linux counts hops + max for "link" while what should be done is counting hops + max for path

I think you're saying we need to include L1 exit latency even for Links where L1 is disabled. I don't think we should include those.

Nope, the code does not do that, it only adds the l1 latency on L1 enabled hops

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

Based on [1], in the path to 03:00.0, both Links have L1 enabled, with an exit latency of <33us, and 03:00.0 can tolerate 64us. That should work fine.

Also based on [1], in the path to 04:00.x, the upstream Link has L1 enabled and the downstream Link has L1 disabled, for an exit latency of <32us, and 04:00.0 can tolerate 64us. That should also work fine.

Again, ignoring the exit latency for 04:00.0

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

(Alternately, disabling L1 on the upstream Link and enabling it on the downstream Link should have an exit latency of <64us and 04:00.0 can tolerate 64us, so that should work fine, too.)

Then nothing else can have L1 aspm enabled

Yes, as I said, we should be able to enable L1 on either of the Links in the path to 04:00.x, but not both.

The code works backwards and disables the first hop that exceeds the latency requirements -
we could argue that it should try to be smarter about it and try to disable a minimum amount of links
while still retaining the minimum latency but... It is what it is and it works when patched.

Quoted 6 lines
Quoted 4 lines

The original problem here is not with the Realtek device at 04:00.x but with the I211 NIC at 03:00.0. So we also need to figure out what the connection is. Does the same I211 performance problem occur if you remove the Realtek device from the system?

It's mounted on the motherboard, so no I can't remove it.

If you're interested, you could probably unload the Realtek drivers, remove the devices, and set the PCI_EXP_LNKCTL_LD (Link Disable) bit in 02:04.0, e.g.,

# RT=/sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:04.0

  # echo 1 > $RT/0000:04:00.0/remove
  # echo 1 > $RT/0000:04:00.1/remove
  # echo 1 > $RT/0000:04:00.2/remove
  # echo 1 > $RT/0000:04:00.4/remove
  # echo 1 > $RT/0000:04:00.7/remove
  # setpci -s02:04.0 CAP_EXP+0x10.w=0x0010

That should take 04:00.x out of the picture.

Quoted 113 lines
Quoted 3 lines

03:00.0 can tolerate 64us of latency, so even if L1 is enabled on both Links leading to it, the path exit latency would be <33us, which should be fine.

Yes, it "should be" but due to broken ASPM latency calculations we have some kind of
side effect that triggers a racecondition/sideeffect/bug that causes it to misbehave.

Since fixing the latency calculation fixes it, I'll leave the rest to someone with a logic
analyzer and a die-hard-fetish for pcie links - I can't debug it.

Quoted 100 lines
Quoted 100 lines
Quoted 9 lines
Quoted 4 lines

Also, since it's clearly doing the wrong thing, I'm worried that dists will take a kernel enable aspm and there will be alot of bugreports of non-booting systems or other weird issues... And the culprit was known all along.

There's clearly a problem on your system, but I don't know yet whether Linux is doing something wrong, a device in your system is designed incorrectly, or a device is designed correctly but the instance in your system is defective.

According to the spec it is, there is a explanation of how to calculate the exit latency
and when you implement that, which i did (before knowing the actual spec) then it works...

Quoted 10 lines
Quoted 1 line

It's been five months...

I apologize for the delay. ASPM is a subtle area of PCIe, the Linux code is complicated, and we have a long history of issues with it. I want to fix the problem, but I want to make sure we do it in a way that matches the spec so the fix applies to all systems. I don't want a magic fix that fixes your system in a way I don't quite understand.

Obviously *you* understand this, so hopefully it's just a matter of pounding it through my thick skull :)

I only understand what I've been forced to understand - and I do leverage the existing code without
knowing what it does underneath, I only look at the links maximum latency and make sure that I keep
the maximum latency along the path and not just link for link

once you realise that the max allowed latency is buffer dependent - then this becomes obviously correct,
and then the pcie spec showed it as being correct as well... so...

Quoted 63 lines
Quoted 63 lines
Quoted 63 lines

[1] https://bugzilla.kernel.org/attachment.cgi?id=293047
[2] https://lore.kernel.org/linux-pci/20201007132808.647589-1-ian.kumlien at gmail.com/
[3] https://bugzilla.kernel.org/attachment.cgi?id=292955
[4] https://bugzilla.kernel.org/attachment.cgi?id=292957

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
      struct aspm_latency *acceptable;
      struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                  (link->latency_dw.l0s > acceptable->l0s))
                      link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
              /*
               * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 ?s from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
               *
               * The exit latencies for L1 substates are not advertised
               * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
               * L1 exit latencies advertised by a device include L1
               * substate latencies (and hence do not do any check).
               */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+                     l1_switch_latency += 1000;
+             }
 
              link = link->parent;
      }
--

2.29.1

#28 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 111 lines · Bjorn Helgaas
Quoted 109 lines · Ian Kumlien
Quoted 107 lines · Bjorn Helgaas
Quoted 96 lines · Ian Kumlien
Quoted 95 lines · Bjorn Helgaas

[+cc Jesse, Tony, David, Jakub, Heiner, lists in case there's an ASPM issue with I211 or Realtek NICs. Beginning of thread:
https://lore.kernel.org/r/20201024205548.1837770-1-ian.kumlien at gmail.com

Short story: Ian has:

Root Port --- Switch --- I211 NIC
\-- multifunction Realtek NIC, etc

and the I211 performance is poor with ASPM L1 enabled on both links in the path to it. The patch here disables ASPM on the upstream link and fixes the performance, but AFAICT the devices in that path give us no reason to disable L1. If I understand the spec correctly, the Realtek device should not be relevant to the I211 path.]

Quoted 69 lines · Ian Kumlien
Quoted 58 lines · Bjorn Helgaas
Quoted 25 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1?s from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.
...

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 8 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

There are two paths here that share a Link:

00:01.2 --- 01:00.0 -- 02:03.0 --- 03:00.0 I211 NIC 00:01.2 --- 01:00.0 -- 02:04.0 --- 04:00.x multifunction Realtek

1) The path to the I211 NIC includes four Ports and two Links (the connection between 01:00.0 and 02:03.0 is internal Switch routing, not a Link).

The Ports advertise L1 exit latencies of <32us, <32us, <32us, <16us. If both Links are in L1 and 03:00.0 initiates L1 exit@T, 01:00.0 initiates L1 exit at T + 1. A TLP from 03:00.0 may see up to 1 + 32 = 33us of L1 exit latency.

The NIC can tolerate up to 64us of L1 exit latency, so it is safe to enable L1 for both Links.

2) The path to the Realtek device is similar except that the Realtek L1 exit latency is <64us. If both Links are in L1 and 04:00.x initiates L1 exit at T, 01:00.0 again initiates L1 exit at T + 1, but a TLP from 04:00.x may see up to 1 + 64 = 65us of L1 exit latency.

Quoted 4 lines

The Realtek device can only tolerate 64us of latency, so it is not safe to enable L1 for both Links. It should be safe to enable L1 on the shared link because the exit latency for that link would be <32us.

04:00.0:
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us

Quoted 5 lines

So maximum latency for the entire link has to be <64 us For the device to leave L1 ASPM takes <64us

So the device itself is the slowest entry along the link, which means that nothing else along that path can have ASPM enabled

Yes. That's what I said above: "it is not safe to enable L1 for both Links." Unless I'm missing something, we agree on that.

Quoted 6 lines
Quoted 4 lines

I also said that it should be safe to enable L1 on the shared Link (from 00:01.2 to 01:00.0) because if the downstream Link is always in L0, the exit latency of the shared Link should be <32us, and 04:00.x can tolerate 64us.

Exit latency of shared link would be max of link, ie 64 + L1-hops, not 32

I don't think this is true. The path from 00:01.2 to 04:00.x includes two Links, and they are independent. The exit latency for each Link depends only on the Port at each end:

Quoted 6 lines
Quoted 2 lines

The full path is what is important, because that is the actual latency (which the current linux code doesn't do)

I think you're saying we need to include the 04:00.x exit latency of 64us even though L1 is not enabled for 04:00.x. I disagree; the L1 exit latency of Ports where L1 is disabled is irrelevant.

I will redo the without patch and look again, I know that I have to wait a while for it to happen.

Witch patch 3 i get:
dec 14 13:44:40 localhost kernel: pci 0000:04:00.0: ASPM latency exceeded, disabling: L1:0000:01:00.0-0000:00:01.2

And it should only check links that has L1 aspm enabled, as per the original code.

Quoted 110 lines
Quoted 104 lines
Quoted 6 lines

Link 1 (depends on 00:01.2 and 01:00.0): max(32, 32) = 32us Link 2 (depends on 02:04.0 and 04:00.x): max(32, 64) = 64us

If L1 is enabled for Link 1 and disabled for Link 2, Link 2 will remain in L0 so it has no L1 exit latency, and the exit latency of the entire path should be 32us.

My patch disables this so yes.

Quoted 95 lines
Quoted 84 lines
Quoted 46 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

Quoted 7 lines
Quoted 4 lines

[Nit: I don't think we should add 1 for the 02:04.0 -- 01:00.0 piece because that's internal Switch routing, not a Link. But even without that extra microsecond, this path does exceed the acceptable latency since 1 + 64 = 65us, and 04:00.0 can only tolerate 64us.]

It does report L1 ASPM on both ends, so the links will be counted as such in the code.

This is a bit of a tangent and we shouldn't get too wrapped up in it. This is a confusing aspect of PCIe. We're talking about this path:

00:01.2 --- [01:00.0 -- 02:04.0] --- 04:00.x multifunction Realtek

This path only contains two Links. The first one is 00:01.2 --- 01:00.0, and the second one is 02:04.0 --- 04:00.x.

01:00.0 is a Switch Upstream Port and 02:04.0 is a Switch Downstream Port. The connection between them is not a Link; it is some internal wiring of the Switch that is completely opaque to software.

The ASPM information and knobs in 01:00.0 apply to the Link on its upstream side, and the ASPM info and knobs in 02:04.0 apply to the Link on its downstream side.

The example in sec 5.4.1.2.2 contains three Links. The L1 exit latency for the Link is the max of the exit latencies at each end:

Link 1: max(32, 8) = 32us
Link 2: max(8, 32) = 32us
Link 3: max(32, 8) = 32us

The total delay for a TLP starting at the downstream end of Link 3 is 32 + 2 = 32us.

Quoted 7 lines

In the path to your 04:00.x Realtek device:

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us Link 2 (from 02:04.0 to 04:00.x): max(32, 64) = 64us

If L1 were enabled on both Links, the exit latency would be 64 + 1 = 65us.

So one line to be removed from the changelog, i assume... And yes, the code handles that - first disable is 01:00.0 <-> 00:01.2

Quoted 27 lines
Quoted 2 lines

I also assume that it can power down individual ports... and enter rest state if no links are up.

I don't think this is quite true -- a Link can't enter L1 unless the Ports on both ends have L1 enabled, so I don't think it makes sense to talk about an individual Port being in L1.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

The "lspci without my patches" [1] shows L1 enabled for the shared Link from 00:01.2 --- 01:00.0 and for the Link to 03:00.0 (I211), but not for the Link to 04:00.x (Realtek).

Quoted 6 lines
Quoted 4 lines

Per my analysis above, that looks like it *should* be a safe configuration. 03:00.0 can tolerate 64us, actual is <33us. 04:00.0 can tolerate 64us, actual should be <32us since only the shared Link is in L1.

See above.

As I said above, if we enabled L1 only on the shared Link from 00:01.2 to 01:00.0, the exit latency should be acceptable. In that case, a TLP from 04:00.x would see only 32us of latency:

Quoted 6 lines
Quoted 3 lines

Link 1 (from 00:01.2 to 01:00.0): max(32, 32) = 32us

and 04:00.x can tolerate 64us.

But, again, you're completely ignoring the full link, ie 04:00.x would also have to power on.

I think you're using "the full link" to refer to the entire path from 00:01.2 to 04:00.x. In PCIe, a "Link" directly connects two Ports. It doesn't refer to the entire path.

Quoted 5 lines
Quoted 3 lines

No, if L1 is disabled on 02:04.0 and 04:00.x (as Linux apparently does by default), the Link between them never enters L1, so there is no power-on for this Link.

It doesn't do it by default, my patch does

Quoted 7 lines

I'm relying on [1], your "lspci without my patches" attachment named "lspci-5.9-mainline.txt", which shows:

02:04.0 LnkCtl: ASPM Disabled
04:00.0 LnkCtl: ASPM Disabled

so I assumed that was what Linux did by default.

Interesting, they are disabled.

Quoted 161 lines
Quoted 149 lines
Quoted 22 lines
Quoted 22 lines
Quoted 12 lines
Quoted 7 lines
Quoted 5 lines

However, the commit log at [2] shows L1 *enabled* for both the shared Link from 00:01.2 --- 01:00.0 and the 02:04.0 --- 04:00.x Link, and that would definitely be a problem.

Can you explain the differences between [1] and [2]?

I don't understand which sections you're referring to.

[1] is the "lspci without my patches" attachment of bugzilla #209725, which is supposed to show the problem this patch solves. We're talking about the path to 04:00.x, and [1] show this:

Quoted 7 lines

01:00.2 L1+ # <-- my typo here, should be 00:01.2
01:00.0 L1+
02:04.0 L1-
04:00.0 L1-

AFAICT, that should be a legal configuration as far as 04:00.0 is concerned, so it's not a reason for this patch.

Actually, no, maximum path latency 64us

Quoted 8 lines
Quoted 3 lines

04:00.0 wakeup latency == 64us

Again, as stated, it can't be behind any sleeping L1 links

It would be pointless for a device to advertise L1 support if it could never be used. 04:00.0 advertises that it can tolerate L1 latency of 64us and that it can exit L1 in 64us or less. So it *can* be behind a Link in L1 as long as nothing else in the path adds more latency.

Yes, as long as nothing along the entire path adds latency - and I didn't make the component
I can only say what it states, and we have to handle it.

Quoted 76 lines
Quoted 76 lines
Quoted 11 lines

[2] is a previous posting of this same patch, and its commit log includes information about the same path to 04:00.x, but the "LnkCtl Before" column shows:

01:00.2 L1+ # <-- my typo here, should be 00:01.2
01:00.0 L1+
02:04.0 L1+
04:00.0 L1+

I don't know why [1] shows L1 disabled on the downstream Link, while
[2] shows L1 *enabled* on the same Link.

From the data they look switched.

Quoted 62 lines
Quoted 42 lines
Quoted 32 lines
Quoted 3 lines

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

I started working through this info, too, but there's not enough information to tell what difference this patch makes. The attachments compare:

1) CONFIG_PCIEASPM_DEFAULT=y without the patch [3] and
2) CONFIG_PCIEASPM_POWERSAVE=y *with* the patch [4]

Obviously CONFIG_PCIEASPM_POWERSAVE=y will configure things differently than CONFIG_PCIEASPM_DEFAULT=y, so we can't tell what changes are due to the config change and what are due to the patch.

The lspci *with* the patch ([4]) shows L0s and L1 enabled at almost every possible place. Here are the Links, how they're configured, and my analysis of the exit latencies vs acceptable latencies:

  00:01.1 --- 01:00.0      L1+ (                  L1 <64us vs unl)
  00:01.2 --- 02:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.3 --- 03:00.0      L1+ (                  L1 <64us vs 64us)
  00:01.4 --- 04:00.0      L1+ (                  L1 <64us vs unl)

00:08.1 --- 05:00.x L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl) 00:08.2 --- 06:00.0 L0s+ L1+ (L0s <64ns vs 4us, L1 <1us vs unl)

So I can't tell what change prevents the freeze. I would expect the patch would cause us to *disable* L0s or L1 somewhere.

Quoted 9 lines

The only place [4] shows ASPM disabled is for 05:00.1. The spec says we should program the same value in all functions of a multi-function device. This is a non-ARI device, so "only capabilities enabled in all functions are enabled for the component as a whole." That would mean that L0s and L1 are effectively disabled for 05:00.x even though 05:00.0 claims they're enabled. But the latencies say ASPM L0s and L1 should be safe to be enabled. This looks like another bug that's probably unrelated.

Quoted 3 lines

I don't think it's unrelated, i suspect it's how PCIe works with multiple links... a device can cause some kind of head of queue stalling - i don't know how but it really looks like it.

The text in quotes above is straight out of the spec (PCIe r5.0, sec 7.5.3.7). Either the device works that way or it's not compliant.

The OS configures ASPM based on the requirements and capabilities advertised by the device. If a device has any head of queue stalling or similar issues, those must be comprehended in the numbers advertised by the device. It's not up to the OS to speculate about issues like that.

Quoted 6 lines
Quoted 3 lines

The patch might be correct; I haven't actually analyzed the code. But the commit log doesn't make sense to me yet.

I personally don't think that all this PCI information is required, the linux kernel is currently doing it wrong according to the spec.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

We're trying to establish exactly *what* Linux is doing wrong. So far we don't have a good explanation of that.

Yes we do, linux counts hops + max for "link" while what should be done is counting hops + max for path

I think you're saying we need to include L1 exit latency even for Links where L1 is disabled. I don't think we should include those.

Nope, the code does not do that, it only adds the l1 latency on L1 enabled hops

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

Based on [1], in the path to 03:00.0, both Links have L1 enabled, with an exit latency of <33us, and 03:00.0 can tolerate 64us. That should work fine.

Also based on [1], in the path to 04:00.x, the upstream Link has L1 enabled and the downstream Link has L1 disabled, for an exit latency of <32us, and 04:00.0 can tolerate 64us. That should also work fine.

Again, ignoring the exit latency for 04:00.0

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

(Alternately, disabling L1 on the upstream Link and enabling it on the downstream Link should have an exit latency of <64us and 04:00.0 can tolerate 64us, so that should work fine, too.)

Then nothing else can have L1 aspm enabled

Yes, as I said, we should be able to enable L1 on either of the Links in the path to 04:00.x, but not both.

The code works backwards and disables the first hop that exceeds the latency requirements -
we could argue that it should try to be smarter about it and try to disable a minimum amount of links
while still retaining the minimum latency but... It is what it is and it works when patched.

Quoted 6 lines
Quoted 4 lines

The original problem here is not with the Realtek device at 04:00.x but with the I211 NIC at 03:00.0. So we also need to figure out what the connection is. Does the same I211 performance problem occur if you remove the Realtek device from the system?

It's mounted on the motherboard, so no I can't remove it.

If you're interested, you could probably unload the Realtek drivers, remove the devices, and set the PCI_EXP_LNKCTL_LD (Link Disable) bit in 02:04.0, e.g.,

# RT=/sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:04.0

Quoted 8 lines
  # echo 1 > $RT/0000:04:00.0/remove
  # echo 1 > $RT/0000:04:00.1/remove
  # echo 1 > $RT/0000:04:00.2/remove
  # echo 1 > $RT/0000:04:00.4/remove
  # echo 1 > $RT/0000:04:00.7/remove
  # setpci -s02:04.0 CAP_EXP+0x10.w=0x0010

That should take 04:00.x out of the picture.

Didn't actually change the behaviour, I'm suspecting an errata for AMD pcie...

So did this, with unpatched kernel:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.56 MBytes  38.2 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  4.47 MBytes  37.5 Mbits/sec    0   96.2 KBytes
[  5]   2.00-3.00   sec  4.85 MBytes  40.7 Mbits/sec    0   50.9 KBytes
[  5]   3.00-4.00   sec  4.23 MBytes  35.4 Mbits/sec    0   70.7 KBytes
[  5]   4.00-5.00   sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
[  5]   5.00-6.00   sec  4.23 MBytes  35.4 Mbits/sec    0   45.2 KBytes
[  5]   6.00-7.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   7.00-8.00   sec  3.98 MBytes  33.4 Mbits/sec    0   36.8 KBytes
[  5]   8.00-9.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   9.00-10.00  sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  43.2 MBytes  36.2 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec                  receiver

and:
echo 0 > /sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/link/l1_aspm

and:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   951 Mbits/sec  153    772 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  276    550 KBytes
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec  123    625 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   31    687 KBytes
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0    679 KBytes
[  5]   5.00-6.00   sec   110 MBytes   923 Mbits/sec  136    577 KBytes
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec  214    645 KBytes
[  5]   7.00-8.00   sec   110 MBytes   923 Mbits/sec   32    628 KBytes
[  5]   8.00-9.00   sec   110 MBytes   923 Mbits/sec   81    537 KBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec   10    577 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec  1056             sender
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec                  receiver

But this only confirms that the fix i experience is a side effect.

The original code is still wrong :)

Quoted 113 lines
Quoted 113 lines
Quoted 3 lines

03:00.0 can tolerate 64us of latency, so even if L1 is enabled on both Links leading to it, the path exit latency would be <33us, which should be fine.

Yes, it "should be" but due to broken ASPM latency calculations we have some kind of
side effect that triggers a racecondition/sideeffect/bug that causes it to misbehave.

Since fixing the latency calculation fixes it, I'll leave the rest to someone with a logic
analyzer and a die-hard-fetish for pcie links - I can't debug it.

Quoted 100 lines
Quoted 100 lines
Quoted 9 lines
Quoted 4 lines

Also, since it's clearly doing the wrong thing, I'm worried that dists will take a kernel enable aspm and there will be alot of bugreports of non-booting systems or other weird issues... And the culprit was known all along.

There's clearly a problem on your system, but I don't know yet whether Linux is doing something wrong, a device in your system is designed incorrectly, or a device is designed correctly but the instance in your system is defective.

According to the spec it is, there is a explanation of how to calculate the exit latency
and when you implement that, which i did (before knowing the actual spec) then it works...

Quoted 10 lines
Quoted 1 line

It's been five months...

I apologize for the delay. ASPM is a subtle area of PCIe, the Linux code is complicated, and we have a long history of issues with it. I want to fix the problem, but I want to make sure we do it in a way that matches the spec so the fix applies to all systems. I don't want a magic fix that fixes your system in a way I don't quite understand.

Obviously *you* understand this, so hopefully it's just a matter of pounding it through my thick skull :)

I only understand what I've been forced to understand - and I do leverage the existing code without
knowing what it does underneath, I only look at the links maximum latency and make sure that I keep
the maximum latency along the path and not just link for link

once you realise that the max allowed latency is buffer dependent - then this becomes obviously correct,
and then the pcie spec showed it as being correct as well... so...

Quoted 63 lines
Quoted 63 lines
Quoted 63 lines

[1] https://bugzilla.kernel.org/attachment.cgi?id=293047
[2] https://lore.kernel.org/linux-pci/20201007132808.647589-1-ian.kumlien at gmail.com/
[3] https://bugzilla.kernel.org/attachment.cgi?id=292955
[4] https://bugzilla.kernel.org/attachment.cgi?id=292957

Quoted 58 lines
---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +14 -9
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-     u32 latency, l1_switch_latency = 0;
+     u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
      struct aspm_latency *acceptable;
      struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
              if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
                  (link->latency_dw.l0s > acceptable->l0s))
                      link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
              /*
               * Check L1 latency.
-              * Every switch on the path to root complex need 1
-              * more microsecond for L1. Spec doesn't mention L0s.
+              *
+              * PCIe r5.0, sec 5.4.1.2.2 states:
+              * A Switch is required to initiate an L1 exit transition on its
+              * Upstream Port Link after no more than 1 ?s from the beginning of an
+              * L1 exit transition on any of its Downstream Port Links.
               *
               * The exit latencies for L1 substates are not advertised
               * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
               * L1 exit latencies advertised by a device include L1
               * substate latencies (and hence do not do any check).
               */
-             latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-             if ((link->aspm_capable & ASPM_STATE_L1) &&
-                 (latency + l1_switch_latency > acceptable->l1))
-                     link->aspm_capable &= ~ASPM_STATE_L1;
-             l1_switch_latency += 1000;
+             if (link->aspm_capable & ASPM_STATE_L1) {
+                     latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+                     l1_max_latency = max_t(u32, latency, l1_max_latency);
+                     if (l1_max_latency + l1_switch_latency > acceptable->l1)
+                             link->aspm_capable &= ~ASPM_STATE_L1;
+                     l1_switch_latency += 1000;
+             }
 
              link = link->parent;
      }
--

2.29.1

#29 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 35 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

If you're interested, you could probably unload the Realtek drivers, remove the devices, and set the PCI_EXP_LNKCTL_LD (Link Disable) bit in 02:04.0, e.g.,

# RT=/sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:04.0

Quoted 8 lines
  # echo 1 > $RT/0000:04:00.0/remove
  # echo 1 > $RT/0000:04:00.1/remove
  # echo 1 > $RT/0000:04:00.2/remove
  # echo 1 > $RT/0000:04:00.4/remove
  # echo 1 > $RT/0000:04:00.7/remove
  # setpci -s02:04.0 CAP_EXP+0x10.w=0x0010

That should take 04:00.x out of the picture.

Didn't actually change the behaviour, I'm suspecting an errata for AMD pcie...

So did this, with unpatched kernel:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.56 MBytes  38.2 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  4.47 MBytes  37.5 Mbits/sec    0   96.2 KBytes
[  5]   2.00-3.00   sec  4.85 MBytes  40.7 Mbits/sec    0   50.9 KBytes
[  5]   3.00-4.00   sec  4.23 MBytes  35.4 Mbits/sec    0   70.7 KBytes
[  5]   4.00-5.00   sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
[  5]   5.00-6.00   sec  4.23 MBytes  35.4 Mbits/sec    0   45.2 KBytes
[  5]   6.00-7.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   7.00-8.00   sec  3.98 MBytes  33.4 Mbits/sec    0   36.8 KBytes
[  5]   8.00-9.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   9.00-10.00  sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  43.2 MBytes  36.2 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec                  receiver
Quoted 2 lines

and:
echo 0 > /sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/link/l1_aspm

BTW, thanks a lot for testing out the "l1_aspm" sysfs file. I'm very pleased that it seems to be working as intended.

Quoted 17 lines

and:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   951 Mbits/sec  153    772 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  276    550 KBytes
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec  123    625 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   31    687 KBytes
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0    679 KBytes
[  5]   5.00-6.00   sec   110 MBytes   923 Mbits/sec  136    577 KBytes
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec  214    645 KBytes
[  5]   7.00-8.00   sec   110 MBytes   923 Mbits/sec   32    628 KBytes
[  5]   8.00-9.00   sec   110 MBytes   923 Mbits/sec   81    537 KBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec   10    577 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec  1056             sender
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec                  receiver
Quoted 3 lines

But this only confirms that the fix i experience is a side effect.

The original code is still wrong :)

What exactly is this machine? Brand, model, config? Maybe you could add this and a dmesg log to the buzilla? It seems like other people should be seeing the same problem, so I'm hoping to grub around on the web to see if there are similar reports involving these devices.

https://bugzilla.kernel.org/show_bug.cgi?id=209725

Here's one that is superficially similar:
https://linux-hardware.org/index.php?probe=e5f24075e5&log=lspci_all
in that it has a RP -- switch -- I211 path. Interestingly, the switch here advertises <64us L1 exit latency instead of the <32us latency your switch advertises. Of course, I can't tell if it's exactly the same switch.

Bjorn

#30 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 37 lines · Bjorn Helgaas
Quoted 35 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

If you're interested, you could probably unload the Realtek drivers, remove the devices, and set the PCI_EXP_LNKCTL_LD (Link Disable) bit in 02:04.0, e.g.,

# RT=/sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:04.0

Quoted 8 lines
  # echo 1 > $RT/0000:04:00.0/remove
  # echo 1 > $RT/0000:04:00.1/remove
  # echo 1 > $RT/0000:04:00.2/remove
  # echo 1 > $RT/0000:04:00.4/remove
  # echo 1 > $RT/0000:04:00.7/remove
  # setpci -s02:04.0 CAP_EXP+0x10.w=0x0010

That should take 04:00.x out of the picture.

Didn't actually change the behaviour, I'm suspecting an errata for AMD pcie...

So did this, with unpatched kernel:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.56 MBytes  38.2 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  4.47 MBytes  37.5 Mbits/sec    0   96.2 KBytes
[  5]   2.00-3.00   sec  4.85 MBytes  40.7 Mbits/sec    0   50.9 KBytes
[  5]   3.00-4.00   sec  4.23 MBytes  35.4 Mbits/sec    0   70.7 KBytes
[  5]   4.00-5.00   sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
[  5]   5.00-6.00   sec  4.23 MBytes  35.4 Mbits/sec    0   45.2 KBytes
[  5]   6.00-7.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   7.00-8.00   sec  3.98 MBytes  33.4 Mbits/sec    0   36.8 KBytes
[  5]   8.00-9.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   9.00-10.00  sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  43.2 MBytes  36.2 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec                  receiver
Quoted 5 lines
Quoted 2 lines

and:
echo 0 > /sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/link/l1_aspm

BTW, thanks a lot for testing out the "l1_aspm" sysfs file. I'm very pleased that it seems to be working as intended.

It was nice to find it for easy disabling :)

Quoted 17 lines
Quoted 17 lines

and:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   951 Mbits/sec  153    772 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  276    550 KBytes
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec  123    625 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   31    687 KBytes
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0    679 KBytes
[  5]   5.00-6.00   sec   110 MBytes   923 Mbits/sec  136    577 KBytes
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec  214    645 KBytes
[  5]   7.00-8.00   sec   110 MBytes   923 Mbits/sec   32    628 KBytes
[  5]   8.00-9.00   sec   110 MBytes   923 Mbits/sec   81    537 KBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec   10    577 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec  1056             sender
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec                  receiver
Quoted 8 lines
Quoted 3 lines

But this only confirms that the fix i experience is a side effect.

The original code is still wrong :)

What exactly is this machine? Brand, model, config? Maybe you could add this and a dmesg log to the buzilla? It seems like other people should be seeing the same problem, so I'm hoping to grub around on the web to see if there are similar reports involving these devices.

ASUS Pro WS X570-ACE with AMD Ryzen 9 3900X

Quoted 8 lines

https://bugzilla.kernel.org/show_bug.cgi?id=209725

Here's one that is superficially similar:
https://linux-hardware.org/index.php?probe=e5f24075e5&log=lspci_all
in that it has a RP -- switch -- I211 path. Interestingly, the switch here advertises <64us L1 exit latency instead of the <32us latency your switch advertises. Of course, I can't tell if it's exactly the same switch.

Same chipset it seems

I'm running bios version:

        Version: 2206
        Release Date: 08/13/2020

ANd latest is:
Version 3003
2020/12/07

Will test upgrading that as well, but it could be that they report the incorrect latency of the switch - I don't know how many things AGESA changes but... It's been updated twice since my upgrade.

Quoted 1 line

Bjorn

#31 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 65 lines · Ian Kumlien
Quoted 37 lines · Bjorn Helgaas
Quoted 35 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

If you're interested, you could probably unload the Realtek drivers, remove the devices, and set the PCI_EXP_LNKCTL_LD (Link Disable) bit in 02:04.0, e.g.,

# RT=/sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:04.0

Quoted 8 lines
  # echo 1 > $RT/0000:04:00.0/remove
  # echo 1 > $RT/0000:04:00.1/remove
  # echo 1 > $RT/0000:04:00.2/remove
  # echo 1 > $RT/0000:04:00.4/remove
  # echo 1 > $RT/0000:04:00.7/remove
  # setpci -s02:04.0 CAP_EXP+0x10.w=0x0010

That should take 04:00.x out of the picture.

Didn't actually change the behaviour, I'm suspecting an errata for AMD pcie...

So did this, with unpatched kernel:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.56 MBytes  38.2 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  4.47 MBytes  37.5 Mbits/sec    0   96.2 KBytes
[  5]   2.00-3.00   sec  4.85 MBytes  40.7 Mbits/sec    0   50.9 KBytes
[  5]   3.00-4.00   sec  4.23 MBytes  35.4 Mbits/sec    0   70.7 KBytes
[  5]   4.00-5.00   sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
[  5]   5.00-6.00   sec  4.23 MBytes  35.4 Mbits/sec    0   45.2 KBytes
[  5]   6.00-7.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   7.00-8.00   sec  3.98 MBytes  33.4 Mbits/sec    0   36.8 KBytes
[  5]   8.00-9.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   9.00-10.00  sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  43.2 MBytes  36.2 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec                  receiver
Quoted 5 lines
Quoted 2 lines

and:
echo 0 > /sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/link/l1_aspm

BTW, thanks a lot for testing out the "l1_aspm" sysfs file. I'm very pleased that it seems to be working as intended.

It was nice to find it for easy disabling :)

Quoted 19 lines
Quoted 19 lines

and:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   951 Mbits/sec  153    772 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  276    550 KBytes
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec  123    625 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   31    687 KBytes
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0    679 KBytes
[  5]   5.00-6.00   sec   110 MBytes   923 Mbits/sec  136    577 KBytes
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec  214    645 KBytes
[  5]   7.00-8.00   sec   110 MBytes   923 Mbits/sec   32    628 KBytes
[  5]   8.00-9.00   sec   110 MBytes   923 Mbits/sec   81    537 KBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec   10    577 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec  1056             sender
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec                  receiver

But this only confirms that the fix i experience is a side effect.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

The original code is still wrong :)

What exactly is this machine? Brand, model, config? Maybe you could add this and a dmesg log to the buzilla? It seems like other people should be seeing the same problem, so I'm hoping to grub around on the web to see if there are similar reports involving these devices.

ASUS Pro WS X570-ACE with AMD Ryzen 9 3900X

Possible similar issues:

https://forums.unraid.net/topic/94274-hardware-upgrade-woes/
https://forums.servethehome.com/index.php?threads/upgraded-my-home-server-from-intel-to-amd-virtual-disk-stuck-in-degraded-unhealty-state.25535/ (Windows)

Quoted 15 lines
Quoted 8 lines

https://bugzilla.kernel.org/show_bug.cgi?id=209725

Here's one that is superficially similar:
https://linux-hardware.org/index.php?probe=e5f24075e5&log=lspci_all
in that it has a RP -- switch -- I211 path. Interestingly, the switch here advertises <64us L1 exit latency instead of the <32us latency your switch advertises. Of course, I can't tell if it's exactly the same switch.

Same chipset it seems

I'm running bios version:

        Version: 2206
        Release Date: 08/13/2020
Quoted 7 lines

ANd latest is:
Version 3003
2020/12/07

Will test upgrading that as well, but it could be that they report the incorrect latency of the switch - I don't know how many things AGESA changes but... It's been updated twice since my upgrade.

I wouldn't be surprised if the advertised exit latencies are writable by the BIOS because it probably depends on electrical characteristics outside the switch. If so, it's possible ASUS just screwed it up.

#32 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 74 lines · Bjorn Helgaas
Quoted 72 lines · Ian Kumlien
Quoted 37 lines · Bjorn Helgaas
Quoted 35 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

If you're interested, you could probably unload the Realtek drivers, remove the devices, and set the PCI_EXP_LNKCTL_LD (Link Disable) bit in 02:04.0, e.g.,

# RT=/sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:04.0

Quoted 8 lines
  # echo 1 > $RT/0000:04:00.0/remove
  # echo 1 > $RT/0000:04:00.1/remove
  # echo 1 > $RT/0000:04:00.2/remove
  # echo 1 > $RT/0000:04:00.4/remove
  # echo 1 > $RT/0000:04:00.7/remove
  # setpci -s02:04.0 CAP_EXP+0x10.w=0x0010

That should take 04:00.x out of the picture.

Didn't actually change the behaviour, I'm suspecting an errata for AMD pcie...

So did this, with unpatched kernel:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.56 MBytes  38.2 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  4.47 MBytes  37.5 Mbits/sec    0   96.2 KBytes
[  5]   2.00-3.00   sec  4.85 MBytes  40.7 Mbits/sec    0   50.9 KBytes
[  5]   3.00-4.00   sec  4.23 MBytes  35.4 Mbits/sec    0   70.7 KBytes
[  5]   4.00-5.00   sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
[  5]   5.00-6.00   sec  4.23 MBytes  35.4 Mbits/sec    0   45.2 KBytes
[  5]   6.00-7.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   7.00-8.00   sec  3.98 MBytes  33.4 Mbits/sec    0   36.8 KBytes
[  5]   8.00-9.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   9.00-10.00  sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  43.2 MBytes  36.2 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec                  receiver
Quoted 5 lines
Quoted 2 lines

and:
echo 0 > /sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/link/l1_aspm

BTW, thanks a lot for testing out the "l1_aspm" sysfs file. I'm very pleased that it seems to be working as intended.

It was nice to find it for easy disabling :)

Quoted 25 lines
Quoted 17 lines

and:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   951 Mbits/sec  153    772 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  276    550 KBytes
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec  123    625 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   31    687 KBytes
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0    679 KBytes
[  5]   5.00-6.00   sec   110 MBytes   923 Mbits/sec  136    577 KBytes
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec  214    645 KBytes
[  5]   7.00-8.00   sec   110 MBytes   923 Mbits/sec   32    628 KBytes
[  5]   8.00-9.00   sec   110 MBytes   923 Mbits/sec   81    537 KBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec   10    577 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec  1056             sender
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec                  receiver
Quoted 3 lines

But this only confirms that the fix i experience is a side effect.

The original code is still wrong :)

What exactly is this machine? Brand, model, config? Maybe you could add this and a dmesg log to the buzilla? It seems like other people should be seeing the same problem, so I'm hoping to grub around on the web to see if there are similar reports involving these devices.

Quoted 6 lines
Quoted 1 line

ASUS Pro WS X570-ACE with AMD Ryzen 9 3900X

Possible similar issues:

https://forums.unraid.net/topic/94274-hardware-upgrade-woes/
https://forums.servethehome.com/index.php?threads/upgraded-my-home-server-from-intel-to-amd-virtual-disk-stuck-in-degraded-unhealty-state.25535/ (Windows)

Could be, I suspect that we need a workaround (is there a quirk for "reporting wrong latency"?) and the patches.

Quoted 19 lines
Quoted 19 lines
Quoted 8 lines

https://bugzilla.kernel.org/show_bug.cgi?id=209725

Here's one that is superficially similar:
https://linux-hardware.org/index.php?probe=e5f24075e5&log=lspci_all
in that it has a RP -- switch -- I211 path. Interestingly, the switch here advertises <64us L1 exit latency instead of the <32us latency your switch advertises. Of course, I can't tell if it's exactly the same switch.

Same chipset it seems

I'm running bios version:

        Version: 2206
        Release Date: 08/13/2020

ANd latest is:
Version 3003
2020/12/07

Quoted 7 lines
Quoted 3 lines

Will test upgrading that as well, but it could be that they report the incorrect latency of the switch - I don't know how many things AGESA changes but... It's been updated twice since my upgrade.

I wouldn't be surprised if the advertised exit latencies are writable by the BIOS because it probably depends on electrical characteristics outside the switch. If so, it's possible ASUS just screwed it up.

Not surprisingly, nothing changed.
(There was a lot of "stability improvements")

#33 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 75 lines · Ian Kumlien
Quoted 74 lines · Bjorn Helgaas
Quoted 72 lines · Ian Kumlien
Quoted 36 lines · Bjorn Helgaas
Quoted 35 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

If you're interested, you could probably unload the Realtek drivers, remove the devices, and set the PCI_EXP_LNKCTL_LD (Link Disable) bit in 02:04.0, e.g.,

# RT=/sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:04.0

Quoted 8 lines
  # echo 1 > $RT/0000:04:00.0/remove
  # echo 1 > $RT/0000:04:00.1/remove
  # echo 1 > $RT/0000:04:00.2/remove
  # echo 1 > $RT/0000:04:00.4/remove
  # echo 1 > $RT/0000:04:00.7/remove
  # setpci -s02:04.0 CAP_EXP+0x10.w=0x0010

That should take 04:00.x out of the picture.

Didn't actually change the behaviour, I'm suspecting an errata for AMD pcie...

So did this, with unpatched kernel:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.56 MBytes  38.2 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  4.47 MBytes  37.5 Mbits/sec    0   96.2 KBytes
[  5]   2.00-3.00   sec  4.85 MBytes  40.7 Mbits/sec    0   50.9 KBytes
[  5]   3.00-4.00   sec  4.23 MBytes  35.4 Mbits/sec    0   70.7 KBytes
[  5]   4.00-5.00   sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
[  5]   5.00-6.00   sec  4.23 MBytes  35.4 Mbits/sec    0   45.2 KBytes
[  5]   6.00-7.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   7.00-8.00   sec  3.98 MBytes  33.4 Mbits/sec    0   36.8 KBytes
[  5]   8.00-9.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   9.00-10.00  sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  43.2 MBytes  36.2 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec                  receiver
Quoted 5 lines
Quoted 2 lines

and:
echo 0 > /sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/link/l1_aspm

BTW, thanks a lot for testing out the "l1_aspm" sysfs file. I'm very pleased that it seems to be working as intended.

It was nice to find it for easy disabling :)

Quoted 17 lines
Quoted 17 lines

and:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   951 Mbits/sec  153    772 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  276    550 KBytes
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec  123    625 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   31    687 KBytes
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0    679 KBytes
[  5]   5.00-6.00   sec   110 MBytes   923 Mbits/sec  136    577 KBytes
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec  214    645 KBytes
[  5]   7.00-8.00   sec   110 MBytes   923 Mbits/sec   32    628 KBytes
[  5]   8.00-9.00   sec   110 MBytes   923 Mbits/sec   81    537 KBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec   10    577 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec  1056             sender
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec                  receiver
Quoted 8 lines
Quoted 3 lines

But this only confirms that the fix i experience is a side effect.

The original code is still wrong :)

What exactly is this machine? Brand, model, config? Maybe you could add this and a dmesg log to the buzilla? It seems like other people should be seeing the same problem, so I'm hoping to grub around on the web to see if there are similar reports involving these devices.

ASUS Pro WS X570-ACE with AMD Ryzen 9 3900X

Quoted 7 lines
Quoted 4 lines

Possible similar issues:

https://forums.unraid.net/topic/94274-hardware-upgrade-woes/
https://forums.servethehome.com/index.php?threads/upgraded-my-home-server-from-intel-to-amd-virtual-disk-stuck-in-degraded-unhealty-state.25535/ (Windows)

Could be, I suspect that we need a workaround (is there a quirk for "reporting wrong latency"?) and the patches.

I don't think there's currently a quirk mechanism that would work for correcting latencies, but there should be, and we could add one if we can figure out for sure what's wrong.

I found this:

https://www.reddit.com/r/VFIO/comments/hgk3cz/x570_pcieclassic_pci_bridge_woes/

which looks like it should be the same hardware (if you can collect a dmesg log or "lspci -nnvv" output we could tell for sure) and is interesting because it includes some lspci output that shows different L1 exit latencies than what you see.

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 8 lines

https://bugzilla.kernel.org/show_bug.cgi?id=209725

Here's one that is superficially similar:
https://linux-hardware.org/index.php?probe=e5f24075e5&log=lspci_all
in that it has a RP -- switch -- I211 path. Interestingly, the switch here advertises <64us L1 exit latency instead of the <32us latency your switch advertises. Of course, I can't tell if it's exactly the same switch.

Same chipset it seems

I'm running bios version:

        Version: 2206
        Release Date: 08/13/2020

ANd latest is:
Version 3003
2020/12/07

Will test upgrading that as well, but it could be that they report the incorrect latency of the switch - I don't know how many things AGESA changes but... It's been updated twice since my upgrade.

Quoted 6 lines
Quoted 3 lines

I wouldn't be surprised if the advertised exit latencies are writable by the BIOS because it probably depends on electrical characteristics outside the switch. If so, it's possible ASUS just screwed it up.

Not surprisingly, nothing changed.
(There was a lot of "stability improvements")

I wouldn't be totally surprised if ASUS didn't test that I211 NIC under Linux, but I'm sure it must work well under Windows. If you happen to have Windows, a free trial version of AIDA64 should be able to give us the equivalent of "lspci -vv".

Bjorn

#34 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 89 lines · Bjorn Helgaas
Quoted 75 lines · Ian Kumlien
Quoted 74 lines · Bjorn Helgaas
Quoted 72 lines · Ian Kumlien
Quoted 36 lines · Bjorn Helgaas
Quoted 35 lines · Ian Kumlien
Quoted 5 lines · Bjorn Helgaas

If you're interested, you could probably unload the Realtek drivers, remove the devices, and set the PCI_EXP_LNKCTL_LD (Link Disable) bit in 02:04.0, e.g.,

# RT=/sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:04.0

Quoted 8 lines
  # echo 1 > $RT/0000:04:00.0/remove
  # echo 1 > $RT/0000:04:00.1/remove
  # echo 1 > $RT/0000:04:00.2/remove
  # echo 1 > $RT/0000:04:00.4/remove
  # echo 1 > $RT/0000:04:00.7/remove
  # setpci -s02:04.0 CAP_EXP+0x10.w=0x0010

That should take 04:00.x out of the picture.

Didn't actually change the behaviour, I'm suspecting an errata for AMD pcie...

So did this, with unpatched kernel:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.56 MBytes  38.2 Mbits/sec    0   67.9 KBytes
[  5]   1.00-2.00   sec  4.47 MBytes  37.5 Mbits/sec    0   96.2 KBytes
[  5]   2.00-3.00   sec  4.85 MBytes  40.7 Mbits/sec    0   50.9 KBytes
[  5]   3.00-4.00   sec  4.23 MBytes  35.4 Mbits/sec    0   70.7 KBytes
[  5]   4.00-5.00   sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
[  5]   5.00-6.00   sec  4.23 MBytes  35.4 Mbits/sec    0   45.2 KBytes
[  5]   6.00-7.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   7.00-8.00   sec  3.98 MBytes  33.4 Mbits/sec    0   36.8 KBytes
[  5]   8.00-9.00   sec  4.23 MBytes  35.4 Mbits/sec    0   36.8 KBytes
[  5]   9.00-10.00  sec  4.23 MBytes  35.4 Mbits/sec    0   48.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  43.2 MBytes  36.2 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec                  receiver
Quoted 5 lines
Quoted 2 lines

and:
echo 0 > /sys/devices/pci0000:00/0000:00:01.2/0000:01:00.0/link/l1_aspm

BTW, thanks a lot for testing out the "l1_aspm" sysfs file. I'm very pleased that it seems to be working as intended.

It was nice to find it for easy disabling :)

Quoted 17 lines
Quoted 17 lines

and:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   951 Mbits/sec  153    772 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  276    550 KBytes
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec  123    625 KBytes
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   31    687 KBytes
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0    679 KBytes
[  5]   5.00-6.00   sec   110 MBytes   923 Mbits/sec  136    577 KBytes
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec  214    645 KBytes
[  5]   7.00-8.00   sec   110 MBytes   923 Mbits/sec   32    628 KBytes
[  5]   8.00-9.00   sec   110 MBytes   923 Mbits/sec   81    537 KBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec   10    577 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec  1056             sender
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec                  receiver
Quoted 8 lines
Quoted 3 lines

But this only confirms that the fix i experience is a side effect.

The original code is still wrong :)

What exactly is this machine? Brand, model, config? Maybe you could add this and a dmesg log to the buzilla? It seems like other people should be seeing the same problem, so I'm hoping to grub around on the web to see if there are similar reports involving these devices.

ASUS Pro WS X570-ACE with AMD Ryzen 9 3900X

Quoted 7 lines
Quoted 4 lines

Possible similar issues:

https://forums.unraid.net/topic/94274-hardware-upgrade-woes/
https://forums.servethehome.com/index.php?threads/upgraded-my-home-server-from-intel-to-amd-virtual-disk-stuck-in-degraded-unhealty-state.25535/ (Windows)

Could be, I suspect that we need a workaround (is there a quirk for "reporting wrong latency"?) and the patches.

I don't think there's currently a quirk mechanism that would work for correcting latencies, but there should be, and we could add one if we can figure out for sure what's wrong.

Quoted 8 lines

I found this:

https://www.reddit.com/r/VFIO/comments/hgk3cz/x570_pcieclassic_pci_bridge_woes/

which looks like it should be the same hardware (if you can collect a dmesg log or "lspci -nnvv" output we could tell for sure) and is interesting because it includes some lspci output that shows different L1 exit latencies than what you see.

I'll send both of them separately to you, no reason to push that to everyone i assume.. =)

Quoted 27 lines
Quoted 27 lines
Quoted 26 lines
Quoted 15 lines
Quoted 8 lines

https://bugzilla.kernel.org/show_bug.cgi?id=209725

Here's one that is superficially similar:
https://linux-hardware.org/index.php?probe=e5f24075e5&log=lspci_all
in that it has a RP -- switch -- I211 path. Interestingly, the switch here advertises <64us L1 exit latency instead of the <32us latency your switch advertises. Of course, I can't tell if it's exactly the same switch.

Same chipset it seems

I'm running bios version:

        Version: 2206
        Release Date: 08/13/2020
Quoted 7 lines

ANd latest is:
Version 3003
2020/12/07

Will test upgrading that as well, but it could be that they report the incorrect latency of the switch - I don't know how many things AGESA changes but... It's been updated twice since my upgrade.

I wouldn't be surprised if the advertised exit latencies are writable by the BIOS because it probably depends on electrical characteristics outside the switch. If so, it's possible ASUS just screwed it up.

Quoted 7 lines
Quoted 2 lines

Not surprisingly, nothing changed.
(There was a lot of "stability improvements")

I wouldn't be totally surprised if ASUS didn't test that I211 NIC under Linux, but I'm sure it must work well under Windows. If you happen to have Windows, a free trial version of AIDA64 should be able to give us the equivalent of "lspci -vv".

I don't have windows, haven't had windows at home since '98 ;)

I'll check with some friends that dualboot om systems that might be similar - will see what i can get

Quoted 1 line

Bjorn

#35 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

In reply to Ian Kumlien
Quoted 76 lines · Ian Kumlien

Make pcie_aspm_check_latency comply with the PCIe spec, specifically:
"5.4.1.2.2. Exit from the L1 State"

Which makes it clear that each switch is required to initiate a transition within 1μs from receiving it, accumulating this latency and then we have to wait for the slowest link along the path before entering L0 state from L1.

The current code doesn't take the maximum latency into account.

From the example:

   +----------------+
   |                |
   |  Root complex  |
   |                |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 1
           |
   +----------------+
   |     |8 μs|     |
   |     +----+     |
   |    Switch A    |
   |     +----+     |
   |     |8 μs|     |
   +----------------+
           |
           |  Link 2
           |
   +----------------+
   |    |32 μs|     |
   |    +-----+     |
   |    Switch B    |
   |    +-----+     |
   |    |32 μs|     |
   +----------------+
           |
           |  Link 3
           |
   +----------------+
   |     |8μs|      |
   |     +---+      |
   |   Endpoint C   |
   |                |
   |                |
   +----------------+

Links 1, 2 and 3 are all in L1 state - endpoint C initiates the transition to L0 at time T. Since switch B takes 32 μs to exit L1 on it's ports, Link 3 will transition to L0 at T+32 (longest time considering T+8 for endpoint C and T+32 for switch B).

Switch B is required to initiate a transition from the L1 state on it's upstream port after no more than 1 μs from the beginning of the transition from L1 state on the downstream port. Therefore, transition from L1 to L0 will begin on link 2 at T+1, this will cascade up the path.

The path will exit L1 at T+34.

On my specific system:
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)

            Exit latency       Acceptable latency
Tree:       L1       L0s       L1       L0s
----------  -------  -----     -------  ------
00:01.2     <32 us   -
| 01:00.0   <32 us   -
|- 02:03.0  <32 us   -
| \03:00.0  <16 us   <2us      <64 us   <512ns
|
\- 02:04.0  <32 us   -
  \04:00.0  <64 us   unlimited <64 us   <512ns
Quoted 3 lines

04:00.0's latency is the same as the maximum it allows so as we walk the path the first switchs startup latency will pass the acceptable latency limit for the link, and as a side-effect it fixes my issues with 03:00.0.

I don't think this is quite right. We're looking at the path to 04:00.0, which includes two Links and one Switch:

The upstream Link:
00:01.2 AMD Root Port L1 Exit Latency <32us
01:00.0 AMD Switch Upstream Port L1 Exit Latency <32us

The downstream Link:
02:04.0 AMD Switch Downstream Port L1 Exit Latency <32us
04:00.0 Realtek Endpoint L1 Exit Latency <64us, Acceptable 64us

If both Links are in L1 and 04:00.0 needs to use the Link at time T, I think the following events are relevant:

  T        04:00.0 initiates L1 exit on downstream Link
  T+1us    01:00.0 initiates L1 exit on upstream Link
  T+33us   upstream Link is in L0 (32us after initiating exit)
  T+64us   downstream Link is in L0 (64us after initiating exit)

The upstream Link's L1 exit latency is completely covered by the downstream Link's, so 04:00.0 *should* only see its own exit latency (64us), which it claims to be able to tolerate.

This patch computes "l1_max_latency + l1_switch_latency", which is 64us + 1us in this case. I don't think it's correct to add the 1us here because that delay is only relevant to the upstream Link. We should add it to the *upstream Link's* exit latency, but even with that, its exit latency is only 33us from 04:00.0's point of view.

Quoted 67 lines

Without this patch, 03:00.0 misbehaves and only gives me ~40 mbit/s over links with 6 or more hops. With this patch I'm back to a maximum of ~933 mbit/s.

The original code path did:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 32 +1 -> ok
01:00.0-00:01.2 max latency 32 +2 -> ok

And thus didn't see any L1 ASPM latency issues.

The new code does:
04:00:0-02:04.0 max latency 64 -> ok
02:04.0-01:00.0 max latency 64 +1 -> latency exceeded 01:00.0-00:01.2 max latency 64 +2 -> latency exceeded

It correctly identifies the issue.

For reference, pcie information:
https://bugzilla.kernel.org/show_bug.cgi?id=209725

Kai-Heng Feng has a machine that will not boot with ASPM without this patch, information is documented here:
https://bugzilla.kernel.org/show_bug.cgi?id=209671

---
 drivers/pci/pcie/aspm.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/pci/pcie/aspm.c +7 -5
index 253c30cc1967..c03ead0f1013 100644
@@ -434,7 +434,7 @@ static void pcie_get_aspm_reg(struct pci_dev *pdev,
 
 static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 {
-	u32 latency, l1_switch_latency = 0;
+	u32 latency, l1_max_latency = 0, l1_switch_latency = 0;
 	struct aspm_latency *acceptable;
 	struct pcie_link_state *link;
 
@@ -456,10 +456,14 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
 		    (link->latency_dw.l0s > acceptable->l0s))
 			link->aspm_capable &= ~ASPM_STATE_L0S_DW;
+
 		/*
 		 * Check L1 latency.
-		 * Every switch on the path to root complex need 1
-		 * more microsecond for L1. Spec doesn't mention L0s.
+		 *
+		 * PCIe r5.0, sec 5.4.1.2.2 states:
+		 * A Switch is required to initiate an L1 exit transition on its
+		 * Upstream Port Link after no more than 1 μs from the beginning of an
+		 * L1 exit transition on any of its Downstream Port Links.
 		 *
 		 * The exit latencies for L1 substates are not advertised
 		 * by a device.  Since the spec also doesn't mention a way
@@ -469,11 +473,13 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint)
 		 * L1 exit latencies advertised by a device include L1
 		 * substate latencies (and hence do not do any check).
 		 */
-		latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
-		if ((link->aspm_capable & ASPM_STATE_L1) &&

drivers/pci/pcie/aspm.c static void pcie_aspm_check_latency(struct pci_dev *endpoint) · line 482

Quoted 10 lines
drivers/pci/pcie/aspm.c +7 -3
-		    (latency + l1_switch_latency > acceptable->l1))
-			link->aspm_capable &= ~ASPM_STATE_L1;
-		l1_switch_latency += 1000;
+		if (link->aspm_capable & ASPM_STATE_L1) {
+			latency = max_t(u32, link->latency_up.l1, link->latency_dw.l1);
+			l1_max_latency = max_t(u32, latency, l1_max_latency);
+			if (l1_max_latency + l1_switch_latency > acceptable->l1)
+				link->aspm_capable &= ~ASPM_STATE_L1;
+			l1_switch_latency += 1000;
+		}

This is pretty subtle but I *think* the existing code is actually correct. The exit latency of a downstream Link overlaps all except 1us of the latency of the next upstream Link, so I don't think we have to add the total Switch delay to the max Link exit latency. We only have to add the Switch delays downstream of Link X to Link X's exit latency.

Also, I think we should accumulate the 1us Switch latency for *all* Switches as the existing code does. For the case where a Switch's Upstream Port is L1-capable but the Downstream Port is not, this patch doesn't add any l1_switch_latency. That assumes the Switch can start the upstream L1 exit instantly upon receipt of a TLP at the Downstream Port. I think we should assume it takes the same time (up to 1us) to start that exit as it would if the Downstream Port were in L1.

drivers/pci/pcie/aspm.c static void pcie_aspm_check_latency(struct pci_dev *endpoint) · line 485

Quoted 2 lines
drivers/pci/pcie/aspm.c +0 -0
 		link = link->parent;
 	}

My guess is the real problem is the Switch is advertising incorrect exit latencies. If the Switch advertised "<64us" exit latency for its Upstream Port, we'd compute "64us exit latency + 1us Switch delay = 65us", which is more than either 03:00.0 or 04:00.0 can tolerate, so we would disable L1 on that upstream Link.

Working around this would require some sort of quirk to override the values read from the Switch, which is a little messy. Here's what I'm thinking (not asking you to do this; just trying to think of an approach):

- Configure common clock earlier, in pci_configure_device(), because this changes the "read-only" L1 exit latencies in Link Capabilities.

- Cache Link Capabilities in the pci_dev.

- Add a quirk to override the cached values for the Switch based on Vendor/Device ID and probably DMI motherboard/BIOS info.

Bjorn

#36 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Sorry about the late reply, been trying to figure out what goes wrong since this email...

And yes, I think you're right - the fact that it fixed my system was basically too good to be true =)

Quoted 10 lines · Bjorn Helgaas

My guess is the real problem is the Switch is advertising incorrect exit latencies. If the Switch advertised "<64us" exit latency for its Upstream Port, we'd compute "64us exit latency + 1us Switch delay = 65us", which is more than either 03:00.0 or 04:00.0 can tolerate, so we would disable L1 on that upstream Link.

Working around this would require some sort of quirk to override the values read from the Switch, which is a little messy. Here's what I'm thinking (not asking you to do this; just trying to think of an approach):

The question is if it's the device or if it's the bridge...

Ie, if the device can't quite handle it or if the bridge/switch/port advertises the wrong latency
I have a friend with a similar motherboard and he has the same latency values - but his kernel doesn't apply ASPM

I also want to check L0s since it seems to be enabled...

Quoted 8 lines

- Configure common clock earlier, in pci_configure_device(), because this changes the "read-only" L1 exit latencies in Link Capabilities.

- Cache Link Capabilities in the pci_dev.

- Add a quirk to override the cached values for the Switch based on Vendor/Device ID and probably DMI motherboard/BIOS info.

I can't say and I actually think it depends on the actual culprit which we haven't quite identified yet...

Quoted 1 line

Bjorn

#37 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 6 lines · Ian Kumlien

Sorry about the late reply, been trying to figure out what goes wrong since this email...

And yes, I think you're right - the fact that it fixed my system was basically too good to be true =)

So, finally had some time to look at this again...

I played some with:

drivers/pci/pcie/aspm.c +0 -0
index ac0557a305af..fdf252eee206 100644
@@ -392,13 +392,13 @@ static void pcie_aspm_check_latency(struct

pci_dev *endpoint)

        while (link) {
                /* Check upstream direction L0s latency */

- if ((link->aspm_capable & ASPM_STATE_L0S_UP) &&
- (link->latency_up.l0s > acceptable->l0s))
+ if ((link->aspm_capable & ASPM_STATE_L0S_UP) /* &&
+ (link->latency_up.l0s > acceptable->l0s)*/)

                        link->aspm_capable &= ~ASPM_STATE_L0S_UP;

                /* Check downstream direction L0s latency */

- if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
- (link->latency_dw.l0s > acceptable->l0s))
+ if ((link->aspm_capable & ASPM_STATE_L0S_DW) /* &&
+ (link->latency_dw.l0s > acceptable->l0s)*/)

                        link->aspm_capable &= ~ASPM_STATE_L0S_DW;
                /*
                 * Check L1 latency.

---

Which does perform better but doesn't solve all the issues...

Home machine:

Latency:       3.364 ms
Download:    640.170 Mbit/s
Upload:      918.865 Mbit/s

My test server:

Latency:       4.549 ms
Download:    945.137 Mbit/s
Upload:      957.848 Mbit/s

But iperf3 still gets bogged down...

[  5]   0.00-1.00   sec  4.66 MBytes  39.0 Mbits/sec    0   82.0 KBytes
[  5]   1.00-2.00   sec  4.60 MBytes  38.6 Mbits/sec    0   79.2 KBytes
[  5]   2.00-3.00   sec  4.47 MBytes  37.5 Mbits/sec    0   56.6 KBytes

And with L1 ASPM disabled as usual:

[  5]   0.00-1.00   sec   112 MBytes   938 Mbits/sec  439    911 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  492    888 KBytes
[  5]   2.00-3.00   sec   110 MBytes   923 Mbits/sec  370   1.07 MBytes

And just for reference, bredbandskollen again with L1 ASPM disabled:

Latency:       2.281 ms
Download:    742.136 Mbit/s
Upload:      938.053 Mbit/s

Anyway, we started to look at the PCIe bridges etc, but i think it's the network card
that is at fault, either with advertised latencies (should be lower) or some bug since
other cards and peripherals connected to the system works just fine...

So, L0s actually seems to have somewhat of an impact - which I found surprising sice
both machines are ~6 hops away - however latency differs (measured with tcp)

Can we collect L1 ASPM latency numbers for:
Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

Quoted 34 lines · Bjorn Helgaas
Quoted 10 lines

My guess is the real problem is the Switch is advertising incorrect exit latencies. If the Switch advertised "<64us" exit latency for its Upstream Port, we'd compute "64us exit latency + 1us Switch delay = 65us", which is more than either 03:00.0 or 04:00.0 can tolerate, so we would disable L1 on that upstream Link.

Working around this would require some sort of quirk to override the values read from the Switch, which is a little messy. Here's what I'm thinking (not asking you to do this; just trying to think of an approach):

The question is if it's the device or if it's the bridge...

Ie, if the device can't quite handle it or if the bridge/switch/port advertises the wrong latency
I have a friend with a similar motherboard and he has the same latency values - but his kernel doesn't apply ASPM

I also want to check L0s since it seems to be enabled...

Quoted 8 lines

- Configure common clock earlier, in pci_configure_device(), because this changes the "read-only" L1 exit latencies in Link Capabilities.

- Cache Link Capabilities in the pci_dev.

- Add a quirk to override the cached values for the Switch based on Vendor/Device ID and probably DMI motherboard/BIOS info.

I can't say and I actually think it depends on the actual culprit which we haven't quite identified yet...

Quoted 1 line

Bjorn

#38 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 68 lines · Ian Kumlien
Quoted 6 lines · Ian Kumlien

Sorry about the late reply, been trying to figure out what goes wrong since this email...

And yes, I think you're right - the fact that it fixed my system was basically too good to be true =)

So, finally had some time to look at this again...

I played some with:

drivers/pci/pcie/aspm.c +4 -4
index ac0557a305af..fdf252eee206 100644
@@ -392,13 +392,13 @@ static void pcie_aspm_check_latency(struct
 pci_dev *endpoint)
 
        while (link) {
                /* Check upstream direction L0s latency */
-               if ((link->aspm_capable & ASPM_STATE_L0S_UP) &&
-                   (link->latency_up.l0s > acceptable->l0s))
+               if ((link->aspm_capable & ASPM_STATE_L0S_UP) /* &&
+                   (link->latency_up.l0s > acceptable->l0s)*/)
                        link->aspm_capable &= ~ASPM_STATE_L0S_UP;
 
                /* Check downstream direction L0s latency */
-               if ((link->aspm_capable & ASPM_STATE_L0S_DW) &&
-                   (link->latency_dw.l0s > acceptable->l0s))
+               if ((link->aspm_capable & ASPM_STATE_L0S_DW) /* &&
+                   (link->latency_dw.l0s > acceptable->l0s)*/)
                        link->aspm_capable &= ~ASPM_STATE_L0S_DW;
                /*

* Check L1 latency.
---

Which does perform better but doesn't solve all the issues...

Home machine:

Latency:       3.364 ms
Download:    640.170 Mbit/s
Upload:      918.865 Mbit/s

My test server:

Latency:       4.549 ms
Download:    945.137 Mbit/s
Upload:      957.848 Mbit/s

But iperf3 still gets bogged down...

[  5]   0.00-1.00   sec  4.66 MBytes  39.0 Mbits/sec    0   82.0 KBytes
[  5]   1.00-2.00   sec  4.60 MBytes  38.6 Mbits/sec    0   79.2 KBytes
[  5]   2.00-3.00   sec  4.47 MBytes  37.5 Mbits/sec    0   56.6 KBytes

And with L1 ASPM disabled as usual:

[  5]   0.00-1.00   sec   112 MBytes   938 Mbits/sec  439    911 KBytes
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec  492    888 KBytes
[  5]   2.00-3.00   sec   110 MBytes   923 Mbits/sec  370   1.07 MBytes

And just for reference, bredbandskollen again with L1 ASPM disabled:

Latency:       2.281 ms
Download:    742.136 Mbit/s
Upload:      938.053 Mbit/s

Anyway, we started to look at the PCIe bridges etc, but i think it's the network card that is at fault, either with advertised latencies (should be lower) or some bug since other cards and peripherals connected to the system works just fine...

Quoted 6 lines

So, L0s actually seems to have somewhat of an impact - which I found surprising sice both machines are ~6 hops away - however latency differs (measured with tcp)

Can we collect L1 ASPM latency numbers for:
Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

I think the most useful information would be the ASPM configuration of the tree rooted at 00:01.2 under Windows, since there the NIC should be supported and have good performance.

Bjorn

#39 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 2 lines · Bjorn Helgaas

On Wed, Feb 24, 2021 at 11:19:55PM +0100, Ian Kumlien wrote:

[... 8<...]

Quoted 3 lines

I think the most useful information would be the ASPM configuration of the tree rooted at 00:01.2 under Windows, since there the NIC should be supported and have good performance.

So the AMD bios patches to fix USB issues seems to have fixed this issue as well!

There are still further fixes to bioses available, but currently that is in beta form

#40 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 4 lines · Ian Kumlien
Quoted 2 lines · Bjorn Helgaas

On Wed, Feb 24, 2021 at 11:19:55PM +0100, Ian Kumlien wrote:

Quoted 8 lines

[... 8<...]

Quoted 3 lines

I think the most useful information would be the ASPM configuration of the tree rooted at 00:01.2 under Windows, since there the NIC should be supported and have good performance.

So the AMD bios patches to fix USB issues seems to have fixed this issue as well!

Really? That's amazing! I assume they did this by changing the exit or acceptable latency values?

It would be really interesting to see the "lspci -vv" output after the BIOS update.

Thanks a lot for following up on this!

Bjorn

#41 lore raw

Re: [PATCH 1/3] PCI/ASPM: Use the path max in L1 ASPM latency check

Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Ian Kumlien
Quoted 2 lines · Bjorn Helgaas

On Wed, Feb 24, 2021 at 11:19:55PM +0100, Ian Kumlien wrote:

[... 8<...]

Quoted 3 lines

I think the most useful information would be the ASPM configuration of the tree rooted at 00:01.2 under Windows, since there the NIC should be supported and have good performance.

Quoted 8 lines
Quoted 2 lines

So the AMD bios patches to fix USB issues seems to have fixed this issue as well!

Really? That's amazing! I assume they did this by changing the exit or acceptable latency values?

It would be really interesting to see the "lspci -vv" output after the BIOS update.

First time I looked there was no difference...

for x in 00:01.2 01:00.0 02:03.0 03:00.0 ; do lspci -vvvv -s $x |grep Latency ; done
Latency: 0, Cache Line Size: 64 bytes
LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <32us
Latency: 0, Cache Line Size: 64 bytes
LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <32us
Latency: 0, Cache Line Size: 64 bytes
LnkCap: Port #3, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <32us
Latency: 0, Cache Line Size: 64 bytes
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us

I think they actually fixed a issue behind the scenes

Quoted 1 line

Thanks a lot for following up on this!

Sorry about the delay - work + looking at a switch with a dodgy lldp bug :/

Quoted 1 line

Bjorn