PCI: Re-evaluate DEV3 14-Bit Tag Requester Enable on link mode changes
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v3 Latest5 patches 4 repliesIn review
- v24 patches 0 repliesSuperseded
Changes in v3 from the cover letter
* Add patch-5 to handle hot-swap scenario
Patches in v3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/5 | PCI: Add DEV3 14-Bit Tag Requester register definitionsinclude/ | +2 | 0 replies | |
| 2/5 | PCI: Move __pcie_update_link_speed() out of linedrivers/ drivers/ | +16 | 0 replies | |
| 3/5 | PCI: Save and restore the Device 3 Control registerdrivers/ drivers/ | +96 | 1 reply | |
| 4/5 | PCI: Clear stale 14-Bit Tag Requester Enable when a link leaves Flit Modedrivers/ drivers/ drivers/ | +142 | 1 reply | |
| 5/5 | PCI: pciehp: Clear stale 14-Bit Tag Requester Enable on hot adddrivers/ | +8 | 1 reply |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.
Cover letter [PATCH V3 0/5] PCI: Re-evaluate DEV3 14-Bit Tag Requester Enable on link mode changes
DEV3_CTL.14-Bit Tag Requester Enable is only meaningful while the link operates in Flit Mode. In Non-Flit Mode the upper tag bits are not transmitted on the wire, so a requester that still has it set emits requests whose completions it can no longer match: the completer does answer, but the completion comes back with the upper tag bits zero. The requester therefore logs an Unexpected Completion for it and, once the timer expires, a Completion Timeout as well. The completer side needs no handling, since a completer only reflects the Tag field of the request it answers and the spec accordingly defines no completer enable.
This happens today on the very first config read a Port issues after the link below it comes back in a different mode, and there are two ways to get there: a reset of any flavour, and a hot swap that replaces a card whose link ran in Flit Mode with one that trains in Non-Flit Mode. Two things are missing in the core:
- DEV3_CTL is not part of the state the core saves and restores, so its contents are lost across a reset even for the devices that do get saved and restored, and
- nothing re-evaluates the 14-Bit Tag Requester Enable when the link mode changes. This matters most for the Port: it is the requester for the config read the core issues to the device below as soon as the link is back, but it is not itself saved and restored on these paths, so it keeps its stale enable.
The series:
1/5 adds the DEV3_CAP.14-Bit Tag Requester Supported and
DEV3_CTL.14-Bit Tag Requester Enable definitions, both bit 2 of their respective register.
2/5 moves __pcie_update_link_speed() out of line, which 4/5 needs. Pure
refactor.
3/5 saves DEV3_CTL and, on restore, drops 14-Bit Tag Requester Enable
from the value written to hardware when the live LNKSTA2.Flit_Mode and DEV3_STA.Segment Captured say Flit Mode is gone. This covers the devices reached via pci_dev_restore(). No other DEV3_CTL bit is modified.
4/5 adds pci_bridge_refresh_14bit_tag(), which fixes the bridge itself
and then walks its subordinate bus, and calls it from pci_bridge_wait_for_secondary_bus() (before the first downstream config read after SBR, DPC release, AER bus reset, slot reset or a bridge D3cold->D0 resume) and from __pcie_update_link_speed() (which covers retrain, bwctrl and hotplug paths that never reach pci_bridge_wait_for_secondary_bus()).
5/5 calls it from pciehp_check_link_status() as well. A surprise hot
swap resets nothing and the new card has no saved state, so the only thing standing between the Port and a failed enumeration is a fixup before pci_bus_check_dev() probes the card.
3/5 and 4/5 are independent of each other; each is useful on its own.
One deliberate asymmetry: the helper only ever clears the enable, it never sets it. That matches how the core treats 10-Bit Tags, which it also never enables on its own, and it means a Port that has been through a Non-Flit link does not get 14-Bit Tags back when a Flit Mode card is plugged in later. Restoring that automatically needs a policy for checking every completer below the Port, which seems better left to a separate series. The saved DEV3_CTL value is deliberately left untouched in 3/5 so that the originally programmed value remains available if we do want it back.
Tested on an arm64 platform with a Root Port whose link comes back in Non-Flit Mode after a reset. Resetting the subordinate bus from sysfs (reset_subordinate, i.e. pci_reset_bridge() -> pci_try_reset_slot()) previously left the Root Port with 14-Bit Tag Requester Enable set, and the first config read to the device below failed with a Completion Timeout plus an Unexpected Completion. With this series the Root Port is fixed up first,
pcieport 0002:80:00.0: cleared 14-Bit Tag Requester Enable: flit mode no longer active (DEV3_STA=0x00000008)
and the reset completes. Note that DEV3_STA.Segment Captured was still set at that point while LNKSTA2.Flit_Mode had already dropped, which is why both are consulted.
V3:
* Add patch-5 to handle hot-swap scenario
V2:
* Split the V1 monolithic patch into 4 patches
Vidya Sagar (5):
PCI: Add DEV3 14-Bit Tag Requester register definitions
PCI: Move __pcie_update_link_speed() out of line
PCI: Save and restore the Device 3 Control register
PCI: Clear stale 14-Bit Tag Requester Enable when a link leaves Flit
Mode
PCI: pciehp: Clear stale 14-Bit Tag Requester Enable on hot add
drivers/pci/hotplug/pciehp_hpc.c | 8 ++ drivers/pci/pci.c | 211 +++++++++++++++++++++++++++++++ drivers/pci/pci.h | 17 +-- drivers/pci/probe.c | 36 ++++++ include/uapi/linux/pci_regs.h | 2 + 5 files changed, 262 insertions(+), 12 deletions(-)
Signature · 1 line
2.43.0