PCI: Add save and restore capability for TPH config space
PATCH v2 1/1 PCI: Add save and restore capability for TPH config space
From: paul luse <[email protected]>
The PCI subsystem does not currently save and restore the configuration space for the TPH (Transaction Layer Packet Processing Hints) capability, leading to the possibility of configuration loss in some scenarios. For more information please refer to PCIe r6.0 sec 6.17.
This was discovered working on the SPDK Project (Storage Performance Development Kit, see https://spdk.io/) where we typically unbind devices from their native kernel driver and bind to VFIO for use with our own user space drivers. The process of binding to VFIO results in the loss of TPH settings due to the resulting PCI reset.
--- drivers/pci/pci.c | 2 + drivers/pci/pci.h | 10 ++++ drivers/pci/pcie/Kconfig | 8 +++ drivers/pci/pcie/Makefile | 1 + drivers/pci/pcie/tph.c | 93 +++++++++++++++++++++++++++++++++++ drivers/pci/probe.c | 1 + include/uapi/linux/pci_regs.h | 2 + 7 files changed, 117 insertions(+) create mode 100644 drivers/pci/pcie/tph.c
drivers/pci/pci.c +2 -0
@@ -1670,6 +1670,7 @@ int pci_save_state(struct pci_dev *dev)pci_save_dpc_state (dev);pci_save_aer_state (dev);pci_save_ptm_state (dev); +pci_save_tph_state (dev);return pci_save_vc_state (dev); }EXPORT_SYMBOL (pci_save_state);
@@ -1782,6 +1783,7 @@ void pci_restore_state(struct pci_dev *dev)pci_restore_rebar_state (dev);pci_restore_dpc_state (dev);pci_restore_ptm_state (dev); +pci_restore_tph_state (dev);pci_aer_clear_status (dev);pci_restore_aer_state (dev);
drivers/pci/pci.h +10 -0
@@ -514,6 +514,16 @@ static inline void pci_restore_ptm_state(struct pci_dev *dev) { }static inline void pci_disable_ptm (struct pci_dev *dev) { }#endif +#ifdef CONFIG_PCIE_TPH +void pci_save_tph_state (struct pci_dev *dev); +void pci_restore_tph_state (struct pci_dev *dev); +void pci_tph_init (struct pci_dev *dev); +#else +static inline void pci_save_tph_state (struct pci_dev *dev) { } +static inline void pci_restore_tph_state (struct pci_dev *dev) { } +static inline void pci_tph_init (struct pci_dev *dev) { } +#endif +unsigned long pci_cardbus_resource_alignment (struct resource *); static inline resource_size_t pci_resource_alignment(struct pci_dev *dev,
drivers/pci/pcie/Kconfig +8 -0
@@ -142,3 +142,11 @@ config PCIE_EDR the PCI Firmware Specification r3.2. Enable this if you want to support hybrid DPC model which uses both firmware and OS to implement DPC. + +config PCIE_TPH + bool "PCI TPH (Transaction Layer Packet Processing Hints) capability support" + help + This enables PCI Express TPH (Transaction Layer Packet Processing Hints) by + making sure they are saved and restored across resets. Enable this if your + hardware uses the PCI Express TPH capabilities. For more information please + refer to PCIe r6.0 sec 6.17.
drivers/pci/pcie/Makefile +1 -0
@@ -13,3 +13,4 @@ obj-$(CONFIG_PCIE_PME) += pme.o obj-$(CONFIG_PCIE_DPC) += dpc.o obj-$(CONFIG_PCIE_PTM) += ptm.o obj-$(CONFIG_PCIE_EDR) += edr.o +obj-$(CONFIG_PCIE_TPH) += tph.o
drivers/pci/pcie/tph.c new file +93 -0
93 lines not on this page · raw message
drivers/pci/probe.c +1 -0
@@ -2464,6 +2464,7 @@ static void pci_init_capabilities(struct pci_dev *dev)pci_aer_init (dev);/* Advanced Error Reporting */ pci_dpc_init (dev);/* Downstream Port Containment */ pci_rcec_init (dev);/* Root Complex Event Collector */ +pci_tph_init (dev);/* Transaction Layer Packet Processing Hints */ pcie_report_downtraining (dev);pci_init_reset_methods (dev);
include/uapi/linux/pci_regs.h +2 -0
@@ -1020,6 +1020,8 @@#define PCI_TPH_LOC_MSIX 0x400/* in MSI-X */ #define PCI_TPH_CAP_ST_MASK 0x07FF0000/* ST table mask */ #define PCI_TPH_CAP_ST_SHIFT 16/* ST table shift */ +#define PCI_TPH_CTL 8/* control offset */ +#define PCI_TPH_ST_TBL 0xc/* ST table offset */ #define PCI_TPH_BASE_SIZEOF 0xc/* size with no ST table */ /* Downstream Port Containment */
Signature · 1 line
2.34.3
Quoted 13 lines · Paul Luse
From: paul luse <[email protected]>
The PCI subsystem does not currently save and restore the configuration space for the TPH (Transaction Layer Packet Processing Hints) capability, leading to the possibility of configuration loss in some scenarios. For more information please refer to PCIe r6.0 sec 6.17.
This was discovered working on the SPDK Project (Storage Performance Development Kit, see https://spdk.io/) where we typically unbind devices from their native kernel driver and bind to VFIO for use with our own user space drivers. The process of binding to VFIO results in the loss of TPH settings due to the resulting PCI reset.
Quoted 3 lines
Signed-off-by Jing Liu <[email protected]>Signed-off-by paul luse <[email protected]>---
Hi Paul,
If you need to submit subsequent versions, could you include a summary of what changed from the previous? You can just jot them down below this '---' line for the reviewers, and any text there would be omitted from the changelog.
As far as I can tell, the only difference is 'tph' is a u16 instead of an int in pci_save_tph_state().
drivers/pci/pcie/Kconfig config PCIE_EDR · line 152
Quoted 7 lines
drivers/pci/pcie/Kconfig +7
-0+config PCIE_TPH + bool "PCI TPH (Transaction Layer Packet Processing Hints) capability support" + help + This enables PCI Express TPH (Transaction Layer Packet Processing Hints) by + making sure they are saved and restored across resets. Enable this if your + hardware uses the PCI Express TPH capabilities. For more information please + refer to PCIe r6.0 sec 6.17.
I'm not sure this needs a config option. Even if hardware isn't supporting TPH, this state save code doesn't takes up much space, and saving this config capability seems to always be the right thing if hardware is supporting TPH.
Otherwise, this looks good to me.
Quoted 12 lines · Keith Busch
drivers/pci/pcie/Kconfig config PCIE_EDR · line 152
Quoted 7 lines · Paul Luse
drivers/pci/pcie/Kconfig +7
-0+config PCIE_TPH + bool "PCI TPH (Transaction Layer Packet Processing Hints) capability support" + help + This enables PCI Express TPH (Transaction Layer Packet Processing Hints) by + making sure they are saved and restored across resets. Enable this if your + hardware uses the PCI Express TPH capabilities. For more information please + refer to PCIe r6.0 sec 6.17.I'm not sure this needs a config option. Even if hardware isn't supporting TPH, this state save code doesn't takes up much space, and saving this config capability seems to always be the right thing if hardware is supporting TPH.
We recently had a complaint from Christoph Hellwig that DOE support should not "bloat every single kernel built with PCI support":
https://lore.kernel.org/linux-pci/YkVBJ+nRA2g%[email protected]/
And OpenWRT folks once tried to make PCI quirks conditional on a config option to reduce kernel footprint by 12 kBytes on space- constrained Mips routers:
https://lore.kernel.org/linux-pci/[email protected]/
Paul has followed your advice and did away with the config option in his most recent version of the patch, but I'm not sure that's actually the right thing to do. I don't think TPH is a mainstream PCIe feature and we need to cater to the needs of low-end devices with PCIe as well.
Thanks,
Lukas
Quoted 23 lines · Lukas Wunner
Quoted 12 lines · Keith Busch
drivers/pci/pcie/Kconfig config PCIE_EDR · line 152
Quoted 7 lines · Paul Luse
drivers/pci/pcie/Kconfig +7
-0+config PCIE_TPH + bool "PCI TPH (Transaction Layer Packet Processing Hints) capability support" + help + This enables PCI Express TPH (Transaction Layer Packet Processing Hints) by + making sure they are saved and restored across resets. Enable this if your + hardware uses the PCI Express TPH capabilities. For more information please + refer to PCIe r6.0 sec 6.17.I'm not sure this needs a config option. Even if hardware isn't supporting TPH, this state save code doesn't takes up much space, and saving this config capability seems to always be the right thing if hardware is supporting TPH.
We recently had a complaint from Christoph Hellwig that DOE support should not "bloat every single kernel built with PCI support":
https://lore.kernel.org/linux-pci/YkVBJ+nRA2g%[email protected]/
And OpenWRT folks once tried to make PCI quirks conditional on a config option to reduce kernel footprint by 12 kBytes on space- constrained Mips routers:
Quoted 7 lines
https://lore.kernel.org/linux-pci/[email protected]/
Paul has followed your advice and did away with the config option in his most recent version of the patch, but I'm not sure that's actually the right thing to do. I don't think TPH is a mainstream PCIe feature and we need to cater to the needs of low-end devices with PCIe as well.
If the added code size from this is not negligable for any enviroment, then okay, fair enough.
Paul, sorry for pointing you in the wrong direction.