PCI: Improve PCIe Capability RMW concurrency control
Subsystems (12)
v4 cover PCI: Improve PCIe Capability RMW concurrency control
PCI Express Capability RMW accessors don't properly protect against concurrent access. Link Control Register is written by a number of things in the kernel in a RMW fashion without any concurrency control. This could in the unlucky case lead to losing one of the updates. One of the most obvious path which can race with most of the other LNKCTL RMW operations seems to be ASPM policy sysfs write which triggers LNKCTL update. Similarly, Root Control Register can be concurrently accessed by AER and PME.
Make pcie_capability_clear_and_set_word() (and other RMW accessors that call it) to use a per device spinlock to protect the RMW operations to the Capability Registers that require locking. Convert open-coded LNKCTL RMW operations to use pcie_capability_clear_and_set_word() to benefit from the locking.
There's also a related series which improves ASPM service driver and device driver coordination by removing out-of-band ASPM state management from device drivers (which will remove some of the code fragments changed by this series but it has higher regression potential which is why it seems prudent to do these changes in two steps):
https://lore.kernel.org/linux-pci/[email protected]/T/#t
v4:
- Rebased on top of pci/main
- Added patch to update documentation
v3:
- Split link retraining change off from ASPM patch & reorder it earlier
- Adjust changelog to take into account the move of link retraining code into PCI core and no longer refer to ASPM (currently in pci/enumeration branch)
- based on top of pci/main
v2:
- Keep the RMW ops caller API the same
- Make pcie_capability_clear_and_set_word() a wrapper that uses locked/unlocked variant based on the capability reg
- Extracted LNKCTL2 changes out from this series to keep this purely a series which fixes something (LNKCTL2 RMW lock is necessary only when PCIe BW control is introduced).
- Added Fixes tags (it's a bit rathole but yeah, they're there now).
- Renamed cap_lock to pcie_cap_lock
- Changed ath1* to clear the ASPMC field before setting it
Ilpo Järvinen (11):
PCI: Add locking to RMW PCI Express Capability Register accessors
PCI: Make link retraining use RMW accessors for changing LNKCTL
PCI: pciehp: Use RMW accessors for changing LNKCTL PCI/ASPM: Use RMW accessors for changing LNKCTL drm/amdgpu: Use RMW accessors for changing LNKCTL drm/radeon: Use RMW accessors for changing LNKCTL net/mlx5: Use RMW accessors for changing LNKCTL
wifi: ath11k: Use RMW accessors for changing LNKCTL
wifi: ath12k: Use RMW accessors for changing LNKCTL
wifi: ath10k: Use RMW accessors for changing LNKCTL
PCI: Document the Capability accessor RMW improvements
Documentation/PCI/pciebus-howto.rst | 14 ++++--- drivers/gpu/drm/amd/amdgpu/cik.c | 36 +++++------------- drivers/gpu/drm/amd/amdgpu/si.c | 36 +++++------------- drivers/gpu/drm/radeon/cik.c | 36 +++++------------- drivers/gpu/drm/radeon/si.c | 37 +++++-------------- .../ethernet/mellanox/mlx5/core/fw_reset.c | 9 +---- drivers/net/wireless/ath/ath10k/pci.c | 9 +++-- drivers/net/wireless/ath/ath11k/pci.c | 10 +++-- drivers/net/wireless/ath/ath12k/pci.c | 10 +++-- drivers/pci/access.c | 20 ++++++++-- drivers/pci/hotplug/pciehp_hpc.c | 12 ++---- drivers/pci/pci.c | 8 +--- drivers/pci/pcie/aspm.c | 31 +++++++--------- drivers/pci/probe.c | 1 + include/linux/pci.h | 34 ++++++++++++++++- 15 files changed, 137 insertions(+), 166 deletions(-)
Signature · 1 line
2.30.2
PATCH v4 1/11 PCI: Add locking to RMW PCI Express Capability Register accessors
Many places in the kernel write the Link Control and Root Control PCI Express Capability Registers without proper concurrency control and this could result in losing the changes one of the writers intended to make.
Add pcie_cap_lock spinlock into the struct pci_dev and use it to protect bit changes made in the RMW capability accessors. Protect only a selected set of registers by differentiating the RMW accessor internally to locked/unlocked variants using a wrapper which has the same signature as pcie_capability_clear_and_set_word(). As the Capability Register (pos) given to the wrapper is always a constant, the compiler should be able to simplify all the dead-code away.
So far only the Link Control Register (ASPM, hotplug, link retraining, various drivers) and the Root Control Register (AER & PME) seem to require RMW locking.
--- drivers/pci/access.c | 20 +++++++++++++++++--- drivers/pci/probe.c | 1 + include/linux/pci.h | 34 ++++++++++++++++++++++++++++++++-- 3 files changed, 50 insertions(+), 5 deletions(-)
drivers/pci/access.c +17 -3
@@ -497,8 +497,8 @@ int pcie_capability_write_dword(struct pci_dev *dev, int pos, u32 val) }EXPORT_SYMBOL (pcie_capability_write_dword); -int pcie_capability_clear_and_set_word (struct pci_dev *dev,int pos, -u16 clear,u16 set) +int pcie_capability_clear_and_set_word_unlocked (struct pci_dev *dev,int pos, +u16 clear,u16 set) {int ret;u16 val;
@@ -512,7 +512,21 @@ int pcie_capability_clear_and_set_word(struct pci_dev *dev, int pos,return ret; } -EXPORT_SYMBOL (pcie_capability_clear_and_set_word); +EXPORT_SYMBOL (pcie_capability_clear_and_set_word_unlocked); + +int pcie_capability_clear_and_set_word_locked (struct pci_dev *dev,int pos, +u16 clear,u16 set) +{ +unsigned long flags; +int ret; + +spin_lock_irqsave (&dev->pcie_cap_lock, flags); + ret =pcie_capability_clear_and_set_word_unlocked (dev, pos, clear, set); +spin_unlock_irqrestore (&dev->pcie_cap_lock, flags); + +return ret; +} +EXPORT_SYMBOL (pcie_capability_clear_and_set_word_locked);int pcie_capability_clear_and_set_dword (struct pci_dev *dev,int pos,u32 clear,u32 set)
drivers/pci/probe.c +1 -0
@@ -2324,6 +2324,7 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus) .end = -1, }; +spin_lock_init (&dev->pcie_cap_lock);#ifdef CONFIG_PCI_MSI raw_spin_lock_init (&dev->msi_lock);#endif
include/linux/pci.h +32 -2
@@ -467,6 +467,7 @@ struct pci_dev {pci_dev_flags_t dev_flags;atomic_t enable_cnt;/* pci_enable_device has been called */ +spinlock_t pcie_cap_lock;/* Protects RMW ops in capability accessors */ u32 saved_config_space[16 ];/* Config space saved at suspend time */ struct hlist_head saved_cap_space;int rom_attr_enabled;/* Display of ROM attribute enabled? */
@@ -1217,11 +1218,40 @@ int pcie_capability_read_word(struct pci_dev *dev, int pos, u16 *val);int pcie_capability_read_dword (struct pci_dev *dev,int pos,u32 *val);int pcie_capability_write_word (struct pci_dev *dev,int pos,u16 val);int pcie_capability_write_dword (struct pci_dev *dev,int pos,u32 val); -int pcie_capability_clear_and_set_word (struct pci_dev *dev,int pos, -u16 clear,u16 set); +int pcie_capability_clear_and_set_word_unlocked (struct pci_dev *dev,int pos, +u16 clear,u16 set); +int pcie_capability_clear_and_set_word_locked (struct pci_dev *dev,int pos, +u16 clear,u16 set);int pcie_capability_clear_and_set_dword (struct pci_dev *dev,int pos,u32 clear,u32 set); +/** +* pcie_capability_clear_and_set_word - RMW accessor for PCI Express Capability Registers +* @dev: PCI device structure of the PCI Express device +* @pos: PCI Express Capability Register +* @clear: Clear bitmask +* @set: Set bitmask +* +* Perform a Read-Modify-Write (RMW) operation using @clear and @set +* bitmasks on PCI Express Capability Register at @pos. Certain PCI Express +* Capability Registers are accessed concurrently in RMW fashion, hence +* require locking which is handled transparently to the caller. +*/ +static inline int pcie_capability_clear_and_set_word (struct pci_dev *dev, +int pos, +u16 clear,u16 set) +{ +switch (pos) { +case PCI_EXP_LNKCTL : +case PCI_EXP_RTCTL : +return pcie_capability_clear_and_set_word_locked (dev, pos, + clear, set); +default : +return pcie_capability_clear_and_set_word_unlocked (dev, pos, + clear, set); + } +} +static inline int pcie_capability_set_word (struct pci_dev *dev,int pos,u16 set) {
Signature · 1 line
2.30.2
PATCH v4 2/11 PCI: Make link retraining use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenDon't assume that the device is fully under the control of PCI core. Use RMW capability accessors in link retraining which do proper locking to avoid losing concurrent updates to the register values.
---
pci/enumeration branch moves the link retraining code into PCI core and also conflicts with a link retraining fix in pci/aspm. The changelog (and patch splitting) takes the move into account by not referring to ASPM while the change itself is not based on pci/enumeration (as per Bjorn's preference).
--- drivers/pci/pci.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
drivers/pci/pci.c +2 -6
@@ -4927,7 +4927,6 @@ static int pcie_wait_for_link_status(struct pci_dev *pdev,int pcie_retrain_link (struct pci_dev *pdev,bool use_lt) {int rc; -u16 lnkctl;/* * Ensure the updated LNKCTL parameters are used during link
@@ -4939,17 +4938,14 @@ int pcie_retrain_link(struct pci_dev *pdev, bool use_lt)if (rc)return rc; -pcie_capability_read_word (pdev,PCI_EXP_LNKCTL , &lnkctl); - lnkctl |=PCI_EXP_LNKCTL_RL ; -pcie_capability_write_word (pdev,PCI_EXP_LNKCTL , lnkctl); +pcie_capability_set_word (pdev,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_RL );if (pdev->clear_retrain_link) {/* * Due to an erratum in some devices the Retrain Link bit * needs to be cleared again manually to allow the link * training to succeed. */ - lnkctl &= ~PCI_EXP_LNKCTL_RL ; -pcie_capability_write_word (pdev,PCI_EXP_LNKCTL , lnkctl); +pcie_capability_clear_word (pdev,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_RL ); }return pcie_wait_for_link_status (pdev, use_lt, !use_lt);
Signature · 1 line
2.30.2
PATCH v4 3/11 PCI: pciehp: Use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenAs hotplug is not the only driver touching LNKCTL, use the RMW capability accessor which handles concurrent changes correctly.
--- drivers/pci/hotplug/pciehp_hpc.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-)
drivers/pci/hotplug/pciehp_hpc.c +3 -9
@@ -332,17 +332,11 @@ int pciehp_check_link_status(struct controller *ctrl)static int __pciehp_link_set (struct controller *ctrl,bool enable) {struct pci_dev *pdev =ctrl_dev (ctrl); -u16 lnk_ctrl; -pcie_capability_read_word (pdev,PCI_EXP_LNKCTL , &lnk_ctrl); +pcie_capability_clear_and_set_word (pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_LD , + !enable ?PCI_EXP_LNKCTL_LD :0 ); -if (enable) - lnk_ctrl &= ~PCI_EXP_LNKCTL_LD ; -else - lnk_ctrl |=PCI_EXP_LNKCTL_LD ; - -pcie_capability_write_word (pdev,PCI_EXP_LNKCTL , lnk_ctrl); -ctrl_dbg (ctrl,"%s: lnk_ctrl = %x\n" , __func__, lnk_ctrl);return 0 ; }
Signature · 1 line
2.30.2
Re: [PATCH v4 03/11] PCI: pciehp: Use RMW accessors for changing LNKCTL
Quoted 2 lines · Ilpo Järvinen
As hotplug is not the only driver touching LNKCTL, use the RMW capability accessor which handles concurrent changes correctly.
...
drivers/pci/hotplug/pciehp_hpc.c int pciehp_check_link_status(struct controller *ctrl) · line 338
Quoted 3 lines
drivers/pci/hotplug/pciehp_hpc.c +3
-0+pcie_capability_clear_and_set_word (pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_LD , + !enable ?PCI_EXP_LNKCTL_LD :0 );
enable ? 0 : PCI_EXP_LNKCTL_LD);
easier to read and get.
Signature · 2 lines
With Best Regards, Andy Shevchenko
PATCH v4 4/11 PCI/ASPM: Use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenDon't assume that the device is fully under the control of ASPM and use RMW capability accessors which do proper locking to avoid losing concurrent updates to the register values.
If configuration fails in pcie_aspm_configure_common_clock(), the function attempts to restore the old PCI_EXP_LNKCTL_CCC settings. Store only the old PCI_EXP_LNKCTL_CCC bit for the relevant devices rather than the content of the whole LNKCTL registers. It aligns better with how pcie_lnkctl_clear_and_set() expects its parameter and makes the code more obvious to understand.
--- drivers/pci/pcie/aspm.c | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-)
drivers/pci/pcie/aspm.c +14 -17
@@ -199,7 +199,7 @@ static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist)static void pcie_aspm_configure_common_clock (struct pcie_link_state *link) {int same_clock =1 ; -u16 reg16, parent_reg, child_reg[8 ]; +u16 reg16, parent_old_ccc, child_old_ccc[8 ];struct pci_dev *child, *parent = link->pdev;struct pci_bus *linkbus = parent->subordinate;/*
@@ -221,6 +221,7 @@ static void pcie_aspm_configure_common_clock(struct pcie_link_state *link)/* Port might be already in common clock mode */ pcie_capability_read_word (parent,PCI_EXP_LNKCTL , ®16); + parent_old_ccc = reg16 &PCI_EXP_LNKCTL_CCC ;if (same_clock && (reg16 &PCI_EXP_LNKCTL_CCC )) {bool consistent = true;
@@ -240,31 +241,27 @@ static void pcie_aspm_configure_common_clock(struct pcie_link_state *link)/* Configure downstream component, all functions */ list_for_each_entry (child, &linkbus->devices, bus_list) {pcie_capability_read_word (child,PCI_EXP_LNKCTL , ®16); - child_reg[PCI_FUNC (child->devfn)] = reg16; -if (same_clock) - reg16 |=PCI_EXP_LNKCTL_CCC ; -else - reg16 &= ~PCI_EXP_LNKCTL_CCC ; -pcie_capability_write_word (child,PCI_EXP_LNKCTL , reg16); + child_old_ccc[PCI_FUNC (child->devfn)] = reg16 &PCI_EXP_LNKCTL_CCC ; +pcie_capability_clear_and_set_word (child,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_CCC , + same_clock ?PCI_EXP_LNKCTL_CCC :0 ); }/* Configure upstream component */ -pcie_capability_read_word (parent,PCI_EXP_LNKCTL , ®16); - parent_reg = reg16; -if (same_clock) - reg16 |=PCI_EXP_LNKCTL_CCC ; -else - reg16 &= ~PCI_EXP_LNKCTL_CCC ; -pcie_capability_write_word (parent,PCI_EXP_LNKCTL , reg16); +pcie_capability_clear_and_set_word (parent,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_CCC , + same_clock ?PCI_EXP_LNKCTL_CCC :0 );if (pcie_retrain_link (link->pdev, true)) {/* Training failed. Restore common clock configurations */ pci_err (parent,"ASPM: Could not configure common clock\n" );list_for_each_entry (child, &linkbus->devices, bus_list) -pcie_capability_write_word (child,PCI_EXP_LNKCTL , - child_reg[PCI_FUNC (child->devfn)]); -pcie_capability_write_word (parent,PCI_EXP_LNKCTL , parent_reg); +pcie_capability_clear_and_set_word (child,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_CCC , + child_old_ccc[PCI_FUNC (child->devfn)]); +pcie_capability_clear_and_set_word (parent,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_CCC , parent_old_ccc); } }
Signature · 1 line
2.30.2
Re: [PATCH v4 04/11] PCI/ASPM: Use RMW accessors for changing LNKCTL
Quoted 10 lines · Ilpo Järvinen
Don't assume that the device is fully under the control of ASPM and use RMW capability accessors which do proper locking to avoid losing concurrent updates to the register values.
If configuration fails in
pcie_aspm_configure_common_clock(), the function attempts to restore the oldPCI_EXP_LNKCTL_CCCsettings. Store only the oldPCI_EXP_LNKCTL_CCCbit for the relevant devices rather than the content of the whole LNKCTL registers. It aligns better with howpcie_lnkctl_clear_and_set()expects its parameter and makes the code more obvious to understand.
...
drivers/pci/pcie/aspm.c static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist) · line 202
Quoted 3 lines
drivers/pci/pcie/aspm.c +1
-1int same_clock =1 ; -u16 reg16, parent_reg, child_reg[8 ]; +u16 reg16, parent_old_ccc, child_old_ccc[8 ];
u16 ccc;
Quoted 2 lines
drivers/pci/pcie/aspm.c +0
-0struct pci_dev *child, *parent = link->pdev;struct pci_bus *linkbus = parent->subordinate;
...
ccc = same_clock ? PCI_EXP_LNKCTL_CCC : 0);
drivers/pci/pcie/aspm.c static void pcie_aspm_configure_common_clock(struct pcie_link_state *link) · line 247
Quoted 13 lines
drivers/pci/pcie/aspm.c +4
-6/* Configure downstream component, all functions */ list_for_each_entry (child, &linkbus->devices, bus_list) {pcie_capability_read_word (child,PCI_EXP_LNKCTL , ®16); - child_reg[PCI_FUNC (child->devfn)] = reg16; -if (same_clock) - reg16 |=PCI_EXP_LNKCTL_CCC ; -else - reg16 &= ~PCI_EXP_LNKCTL_CCC ; -pcie_capability_write_word (child,PCI_EXP_LNKCTL , reg16); + child_old_ccc[PCI_FUNC (child->devfn)] = reg16 &PCI_EXP_LNKCTL_CCC ; +pcie_capability_clear_and_set_word (child,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_CCC , + same_clock ?PCI_EXP_LNKCTL_CCC :0 );
pcie_capability_clear_and_set_word(child, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_CCC, ccc);
Quoted 1 line
}
...
drivers/pci/pcie/aspm.c static void pcie_aspm_configure_common_clock(struct pcie_link_state *link) · line 253
Quoted 3 lines
drivers/pci/pcie/aspm.c +3
-0+pcie_capability_clear_and_set_word (parent,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_CCC , + same_clock ?PCI_EXP_LNKCTL_CCC :0 );
pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_CCC, ccc);
?
Signature · 2 lines
With Best Regards, Andy Shevchenko
PATCH v4 5/11 drm/amdgpu: Use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenDon't assume that only the driver would be accessing LNKCTL. ASPM policy changes can trigger write to LNKCTL outside of driver's control. And in the case of upstream bridge, the driver does not even own the device it's changing the registers for.
Use RMW capability accessors which do proper locking to avoid losing concurrent updates to the register value.
--- drivers/gpu/drm/amd/amdgpu/cik.c | 36 +++++++++----------------------- drivers/gpu/drm/amd/amdgpu/si.c | 36 +++++++++----------------------- 2 files changed, 20 insertions(+), 52 deletions(-)
drivers/gpu/drm/amd/amdgpu/cik.c +10 -26
@@ -1574,17 +1574,8 @@ static void cik_pcie_gen3_enable(struct amdgpu_device *adev)u16 bridge_cfg2, gpu_cfg2;u32 max_lw, current_lw, tmp; -pcie_capability_read_word (root,PCI_EXP_LNKCTL , - &bridge_cfg); -pcie_capability_read_word (adev->pdev,PCI_EXP_LNKCTL , - &gpu_cfg); - - tmp16 = bridge_cfg |PCI_EXP_LNKCTL_HAWD ; -pcie_capability_write_word (root,PCI_EXP_LNKCTL , tmp16); - - tmp16 = gpu_cfg |PCI_EXP_LNKCTL_HAWD ; -pcie_capability_write_word (adev->pdev,PCI_EXP_LNKCTL , - tmp16); +pcie_capability_set_word (root,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_HAWD ); +pcie_capability_set_word (adev->pdev,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_HAWD ); tmp =RREG32_PCIE (ixPCIE_LC_STATUS1); max_lw = (tmp & PCIE_LC_STATUS1__LC_DETECTED_LINK_WIDTH_MASK) >>
@@ -1637,21 +1628,14 @@ static void cik_pcie_gen3_enable(struct amdgpu_device *adev)msleep (100 );/* linkctl */ -pcie_capability_read_word (root,PCI_EXP_LNKCTL , - &tmp16); - tmp16 &= ~PCI_EXP_LNKCTL_HAWD ; - tmp16 |= (bridge_cfg &PCI_EXP_LNKCTL_HAWD ); -pcie_capability_write_word (root,PCI_EXP_LNKCTL , - tmp16); - -pcie_capability_read_word (adev->pdev, -PCI_EXP_LNKCTL , - &tmp16); - tmp16 &= ~PCI_EXP_LNKCTL_HAWD ; - tmp16 |= (gpu_cfg &PCI_EXP_LNKCTL_HAWD ); -pcie_capability_write_word (adev->pdev, -PCI_EXP_LNKCTL , - tmp16); +pcie_capability_clear_and_set_word (root,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_HAWD , + bridge_cfg & +PCI_EXP_LNKCTL_HAWD ); +pcie_capability_clear_and_set_word (adev->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_HAWD , + gpu_cfg & +PCI_EXP_LNKCTL_HAWD );/* linkctl2 */ pcie_capability_read_word(root, PCI_EXP_LNKCTL2,
drivers/gpu/drm/amd/amdgpu/si.c +10 -26
@@ -2276,17 +2276,8 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev)u16 bridge_cfg2, gpu_cfg2;u32 max_lw, current_lw, tmp; -pcie_capability_read_word (root,PCI_EXP_LNKCTL , - &bridge_cfg); -pcie_capability_read_word (adev->pdev,PCI_EXP_LNKCTL , - &gpu_cfg); - - tmp16 = bridge_cfg |PCI_EXP_LNKCTL_HAWD ; -pcie_capability_write_word (root,PCI_EXP_LNKCTL , tmp16); - - tmp16 = gpu_cfg |PCI_EXP_LNKCTL_HAWD ; -pcie_capability_write_word (adev->pdev,PCI_EXP_LNKCTL , - tmp16); +pcie_capability_set_word (root,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_HAWD ); +pcie_capability_set_word (adev->pdev,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_HAWD ); tmp =RREG32_PCIE (PCIE_LC_STATUS1 ); max_lw = (tmp &LC_DETECTED_LINK_WIDTH_MASK ) >>LC_DETECTED_LINK_WIDTH_SHIFT ;
@@ -2331,21 +2322,14 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev)mdelay (100 ); -pcie_capability_read_word (root,PCI_EXP_LNKCTL , - &tmp16); - tmp16 &= ~PCI_EXP_LNKCTL_HAWD ; - tmp16 |= (bridge_cfg &PCI_EXP_LNKCTL_HAWD ); -pcie_capability_write_word (root,PCI_EXP_LNKCTL , - tmp16); - -pcie_capability_read_word (adev->pdev, -PCI_EXP_LNKCTL , - &tmp16); - tmp16 &= ~PCI_EXP_LNKCTL_HAWD ; - tmp16 |= (gpu_cfg &PCI_EXP_LNKCTL_HAWD ); -pcie_capability_write_word (adev->pdev, -PCI_EXP_LNKCTL , - tmp16); +pcie_capability_clear_and_set_word (root,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_HAWD , + bridge_cfg & +PCI_EXP_LNKCTL_HAWD ); +pcie_capability_clear_and_set_word (adev->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_HAWD , + gpu_cfg & +PCI_EXP_LNKCTL_HAWD );pcie_capability_read_word (root,PCI_EXP_LNKCTL2 , &tmp16);
Signature · 1 line
2.30.2
PATCH v4 6/11 drm/radeon: Use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenDon't assume that only the driver would be accessing LNKCTL. ASPM policy changes can trigger write to LNKCTL outside of driver's control. And in the case of upstream bridge, the driver does not even own the device it's changing the registers for.
Use RMW capability accessors which do proper locking to avoid losing concurrent updates to the register value.
--- drivers/gpu/drm/radeon/cik.c | 36 ++++++++++------------------------- drivers/gpu/drm/radeon/si.c | 37 ++++++++++-------------------------- 2 files changed, 20 insertions(+), 53 deletions(-)
drivers/gpu/drm/radeon/cik.c +10 -26
@@ -9534,17 +9534,8 @@ static void cik_pcie_gen3_enable(struct radeon_device *rdev)u16 bridge_cfg2, gpu_cfg2;u32 max_lw, current_lw, tmp; -pcie_capability_read_word (root,PCI_EXP_LNKCTL , - &bridge_cfg); -pcie_capability_read_word (rdev->pdev,PCI_EXP_LNKCTL , - &gpu_cfg); - - tmp16 = bridge_cfg |PCI_EXP_LNKCTL_HAWD ; -pcie_capability_write_word (root,PCI_EXP_LNKCTL , tmp16); - - tmp16 = gpu_cfg |PCI_EXP_LNKCTL_HAWD ; -pcie_capability_write_word (rdev->pdev,PCI_EXP_LNKCTL , - tmp16); +pcie_capability_set_word (root,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_HAWD ); +pcie_capability_set_word (rdev->pdev,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_HAWD ); tmp =RREG32_PCIE_PORT (PCIE_LC_STATUS1 ); max_lw = (tmp &LC_DETECTED_LINK_WIDTH_MASK ) >>LC_DETECTED_LINK_WIDTH_SHIFT ;
@@ -9591,21 +9582,14 @@ static void cik_pcie_gen3_enable(struct radeon_device *rdev)msleep (100 );/* linkctl */ -pcie_capability_read_word (root,PCI_EXP_LNKCTL , - &tmp16); - tmp16 &= ~PCI_EXP_LNKCTL_HAWD ; - tmp16 |= (bridge_cfg &PCI_EXP_LNKCTL_HAWD ); -pcie_capability_write_word (root,PCI_EXP_LNKCTL , - tmp16); - -pcie_capability_read_word (rdev->pdev, -PCI_EXP_LNKCTL , - &tmp16); - tmp16 &= ~PCI_EXP_LNKCTL_HAWD ; - tmp16 |= (gpu_cfg &PCI_EXP_LNKCTL_HAWD ); -pcie_capability_write_word (rdev->pdev, -PCI_EXP_LNKCTL , - tmp16); +pcie_capability_clear_and_set_word (root,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_HAWD , + bridge_cfg & +PCI_EXP_LNKCTL_HAWD ); +pcie_capability_clear_and_set_word (rdev->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_HAWD , + gpu_cfg & +PCI_EXP_LNKCTL_HAWD );/* linkctl2 */ pcie_capability_read_word(root, PCI_EXP_LNKCTL2,
drivers/gpu/drm/radeon/si.c +10 -27
@@ -7131,17 +7131,8 @@ static void si_pcie_gen3_enable(struct radeon_device *rdev)u16 bridge_cfg2, gpu_cfg2;u32 max_lw, current_lw, tmp; -pcie_capability_read_word (root,PCI_EXP_LNKCTL , - &bridge_cfg); -pcie_capability_read_word (rdev->pdev,PCI_EXP_LNKCTL , - &gpu_cfg); - - tmp16 = bridge_cfg |PCI_EXP_LNKCTL_HAWD ; -pcie_capability_write_word (root,PCI_EXP_LNKCTL , tmp16); - - tmp16 = gpu_cfg |PCI_EXP_LNKCTL_HAWD ; -pcie_capability_write_word (rdev->pdev,PCI_EXP_LNKCTL , - tmp16); +pcie_capability_set_word (root,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_HAWD ); +pcie_capability_set_word (rdev->pdev,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_HAWD ); tmp =RREG32_PCIE (PCIE_LC_STATUS1 ); max_lw = (tmp &LC_DETECTED_LINK_WIDTH_MASK ) >>LC_DETECTED_LINK_WIDTH_SHIFT ;
@@ -7188,22 +7179,14 @@ static void si_pcie_gen3_enable(struct radeon_device *rdev)msleep (100 );/* linkctl */ -pcie_capability_read_word (root,PCI_EXP_LNKCTL , - &tmp16); - tmp16 &= ~PCI_EXP_LNKCTL_HAWD ; - tmp16 |= (bridge_cfg &PCI_EXP_LNKCTL_HAWD ); -pcie_capability_write_word (root, -PCI_EXP_LNKCTL , - tmp16); - -pcie_capability_read_word (rdev->pdev, -PCI_EXP_LNKCTL , - &tmp16); - tmp16 &= ~PCI_EXP_LNKCTL_HAWD ; - tmp16 |= (gpu_cfg &PCI_EXP_LNKCTL_HAWD ); -pcie_capability_write_word (rdev->pdev, -PCI_EXP_LNKCTL , - tmp16); +pcie_capability_clear_and_set_word (root,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_HAWD , + bridge_cfg & +PCI_EXP_LNKCTL_HAWD ); +pcie_capability_clear_and_set_word (rdev->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_HAWD , + gpu_cfg & +PCI_EXP_LNKCTL_HAWD );/* linkctl2 */ pcie_capability_read_word(root, PCI_EXP_LNKCTL2,
Signature · 1 line
2.30.2
PATCH v4 7/11 net/mlx5: Use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenDon't assume that only the driver would be accessing LNKCTL of the upstream bridge. ASPM policy changes can trigger write to LNKCTL outside of driver's control.
Use RMW capability accessors which do proper locking to avoid losing concurrent updates to the register value.
--- drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c +2 -7
@@ -384,16 +384,11 @@ static int mlx5_pci_link_toggle(struct mlx5_core_dev *dev)pci_cfg_access_lock (sdev); }/* PCI link toggle */ - err =pci_read_config_word (bridge, cap +PCI_EXP_LNKCTL , ®16); -if (err) -return err; - reg16 |=PCI_EXP_LNKCTL_LD ; - err =pci_write_config_word (bridge, cap +PCI_EXP_LNKCTL , reg16); + err =pcie_capability_set_word (bridge,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_LD );if (err)return err;msleep (500 ); - reg16 &= ~PCI_EXP_LNKCTL_LD ; - err =pci_write_config_word (bridge, cap +PCI_EXP_LNKCTL , reg16); + err =pcie_capability_clear_word (bridge,PCI_EXP_LNKCTL ,PCI_EXP_LNKCTL_LD );if (err)return err;
Signature · 1 line
2.30.2
PATCH v4 8/11 wifi: ath11k: Use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenDon't assume that only the driver would be accessing LNKCTL. ASPM policy changes can trigger write to LNKCTL outside of driver's control.
Use RMW capability accessors which do proper locking to avoid losing concurrent updates to the register value. On restore, clear the ASPMC field properly.
--- drivers/net/wireless/ath/ath11k/pci.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/net/wireless/ath/ath11k/pci.c +6 -4
@@ -581,8 +581,8 @@ static void ath11k_pci_aspm_disable(struct ath11k_pci *ab_pci) u16_get_bits(ab_pci->link_ctl, PCI_EXP_LNKCTL_ASPM_L1));/* disable L0s and L1 */ -pcie_capability_write_word (ab_pci->pdev,PCI_EXP_LNKCTL , - ab_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC ); +pcie_capability_clear_word (ab_pci->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_ASPMC );set_bit (ATH11K_PCI_ASPM_RESTORE , &ab_pci->flags); }
@@ -590,8 +590,10 @@ static void ath11k_pci_aspm_disable(struct ath11k_pci *ab_pci)static void ath11k_pci_aspm_restore (struct ath11k_pci *ab_pci) {if (test_and_clear_bit (ATH11K_PCI_ASPM_RESTORE , &ab_pci->flags)) -pcie_capability_write_word (ab_pci->pdev,PCI_EXP_LNKCTL , - ab_pci->link_ctl); +pcie_capability_clear_and_set_word (ab_pci->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_ASPMC , + ab_pci->link_ctl & +PCI_EXP_LNKCTL_ASPMC ); }static int ath11k_pci_power_up (struct ath11k_base *ab)
Signature · 7 lines
2.30.2 -- ath11k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath11k
PATCH v4 9/11 wifi: ath12k: Use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenDon't assume that only the driver would be accessing LNKCTL. ASPM policy changes can trigger write to LNKCTL outside of driver's control.
Use RMW capability accessors which do proper locking to avoid losing concurrent updates to the register value. On restore, clear the ASPMC field properly.
--- drivers/net/wireless/ath/ath12k/pci.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/net/wireless/ath/ath12k/pci.c +6 -4
@@ -794,8 +794,8 @@ static void ath12k_pci_aspm_disable(struct ath12k_pci *ab_pci) u16_get_bits(ab_pci->link_ctl, PCI_EXP_LNKCTL_ASPM_L1));/* disable L0s and L1 */ -pcie_capability_write_word (ab_pci->pdev,PCI_EXP_LNKCTL , - ab_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC ); +pcie_capability_clear_word (ab_pci->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_ASPMC );set_bit (ATH12K_PCI_ASPM_RESTORE , &ab_pci->flags); }
@@ -803,8 +803,10 @@ static void ath12k_pci_aspm_disable(struct ath12k_pci *ab_pci)static void ath12k_pci_aspm_restore (struct ath12k_pci *ab_pci) {if (test_and_clear_bit (ATH12K_PCI_ASPM_RESTORE , &ab_pci->flags)) -pcie_capability_write_word (ab_pci->pdev,PCI_EXP_LNKCTL , - ab_pci->link_ctl); +pcie_capability_clear_and_set_word (ab_pci->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_ASPMC , + ab_pci->link_ctl & +PCI_EXP_LNKCTL_ASPMC ); }static void ath12k_pci_kill_tasklets (struct ath12k_base *ab)
Signature · 7 lines
2.30.2 -- ath12k mailing list [email protected] https://lists.infradead.org/mailman/listinfo/ath12k
PATCH v4 10/11 wifi: ath10k: Use RMW accessors for changing LNKCTL
In reply to Ilpo JärvinenDon't assume that only the driver would be accessing LNKCTL. ASPM policy changes can trigger write to LNKCTL outside of driver's control.
Use RMW capability accessors which does proper locking to avoid losing concurrent updates to the register value. On restore, clear the ASPMC field properly.
--- drivers/net/wireless/ath/ath10k/pci.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/net/wireless/ath/ath10k/pci.c +5 -4
@@ -1963,8 +1963,9 @@ static int ath10k_pci_hif_start(struct ath10k *ar)ath10k_pci_irq_enable (ar);ath10k_pci_rx_post (ar); -pcie_capability_write_word (ar_pci->pdev,PCI_EXP_LNKCTL , - ar_pci->link_ctl); +pcie_capability_clear_and_set_word (ar_pci->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_ASPMC , + ar_pci->link_ctl &PCI_EXP_LNKCTL_ASPMC );return 0 ; }
@@ -2821,8 +2822,8 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar,pcie_capability_read_word (ar_pci->pdev,PCI_EXP_LNKCTL , &ar_pci->link_ctl); -pcie_capability_write_word (ar_pci->pdev,PCI_EXP_LNKCTL , - ar_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC ); +pcie_capability_clear_word (ar_pci->pdev,PCI_EXP_LNKCTL , +PCI_EXP_LNKCTL_ASPMC );/* * Bring the target up cleanly.
Signature · 7 lines
2.30.2 _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
PATCH v4 11/11 PCI: Document the Capability accessor RMW improvements
In reply to Ilpo JärvinenDocumentation claims port service drivers should play nice wrt. PCIe Capability changes but the concurrency control is now provided in the Capability accessors as long as the correct ones are used.
Update the documention to match the RMW accessor behavior.
--- Documentation/PCI/pciebus-howto.rst | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
Documentation/PCI/pciebus-howto.rst +9 -5
@@ -213,8 +213,12 @@ PCI Config Registers -------------------- Each service driver runs its PCI config operations on its own -capability structure except the PCI Express capability structure, in -which Root Control register and Device Control register are shared -between PME and AER. This patch assumes that all service drivers -will be well behaved and not overwrite other service driver's -configuration settings. +capability structure except the PCI Express capability structure, +that is shared between many drivers including the service drivers. +RMW Capability accessors (pcie_capability_clear_and_set_word(), +pcie_capability_set_word(), and pcie_capability_clear_word()) protect +a selected set of PCI Express Capability Registers (Link Control +Register and Root Control Register). Any change to those registers +should be performed using RMW accessors to avoid problems due to +concurrent updates. For the up-to-date list of protected registers, +see pcie_capability_clear_and_set_word().
Signature · 1 line
2.30.2