Skip to contentSkip to conversation

Disabling ASPM L1 sub-states selectively from drivers?

DISCUSSION Heiner Kallweit 2 participants 3 messages Feb 22, 2019 – Feb 27, 2019 PCI

lore
Share & tools
lore
#0 lore raw

I face the issue that a PCIe network chip misses RX packets if ASPM L1 sub-states are enabled. It seems that the RX FIFO is too small to buffer all incoming packets during ASPM exit latency.

So far pci_disable_link_state() only allows to disable L1 completely. Would it make sense to extend this function to allow disabling L1 sub-states selectively? Looking at pcie_config_aspm_link() this seems to be possible.

maintainer
#1 lore raw

Hi Heiner,

Quoted 9 lines · Heiner Kallweit

I face the issue that a PCIe network chip misses RX packets if ASPM L1 sub-states are enabled. It seems that the RX FIFO is too small to buffer all incoming packets during ASPM exit latency.

So far pci_disable_link_state() only allows to disable L1 completely. Would it make sense to extend this function to allow disabling L1 sub-states selectively? Looking at pcie_config_aspm_link() this seems to be possible.

We could certainly explore the option of selectively disabling L1 substates.

But before we do that, let's look at a couple things, because there are some Linux issues in that area, and it's possible we could make a generic fix that wouldn't require disabling the substates completely.

One problem is the ASPM L1.2 state depends on LTR information, and we don't support LTR correctly. There are a couple patches in -next to fix some problems, but we still don't handle cases where the BIOS doesn't program the LTR latencies and the LTR_L1.2_Threshold.

Can you open a report at bugzilla.kernel.org and attach the complete dmesg log (booted with "pci=earlydump") and the "sudo lspci -vvvxx" output?

Have you figured out which L1 substate specifically causes problems? If not, maybe we can use setpci to fiddle with things manually and narrow it down.

Bjorn

#2 lore raw
Quoted 19 lines · Bjorn Helgaas

Hi Heiner,

Quoted 9 lines · Heiner Kallweit

I face the issue that a PCIe network chip misses RX packets if ASPM L1 sub-states are enabled. It seems that the RX FIFO is too small to buffer all incoming packets during ASPM exit latency.

So far pci_disable_link_state() only allows to disable L1 completely. Would it make sense to extend this function to allow disabling L1 sub-states selectively? Looking at pcie_config_aspm_link() this seems to be possible.

We could certainly explore the option of selectively disabling L1 substates.

But before we do that, let's look at a couple things, because there are some Linux issues in that area, and it's possible we could make a generic fix that wouldn't require disabling the substates completely.

Quoted 9 lines

One problem is the ASPM L1.2 state depends on LTR information, and we don't support LTR correctly. There are a couple patches in -next to fix some problems, but we still don't handle cases where the BIOS doesn't program the LTR latencies and the LTR_L1.2_Threshold.

Can you open a report at bugzilla.kernel.org and attach the complete dmesg log (booted with "pci=earlydump") and the "sudo lspci -vvvxx" output?

I have no system where I can reproduce the issue, it was reported by a user: https://bugzilla.redhat.com/show_bug.cgi?id=1671958
In comments 61 and 65 you find the lspci -vv output for the network chip.

Quoted 4 lines

Have you figured out which L1 substate specifically causes problems? If not, maybe we can use setpci to fiddle with things manually and narrow it down.

So far all I can say is that with ASPM disabled completely the issue doesn't occur. I'd have to see how to disable L1.2 only with setpci and then let the user test.

Quoted 2 lines

Bjorn

Heiner