PCI/ASPM: Add sysfs attributes for controlling ASPM

PATCH Applied Heiner Kallweit 4 participants 17 messages Oct 5, 2019 – Nov 24, 2019 PCI

Share & tools

Heiner Kallweit

author

2019-10-05 12:02 UTC #0 lore raw

v7 cover PCI/ASPM: Add sysfs attributes for controlling ASPM

Background of this extension is a problem with the r8169 network driver. Several combinations of board chipsets and network chip versions have problems if ASPM is enabled, therefore we have to disable ASPM per default. However especially on notebooks ASPM can provide significant power-saving, therefore we want to give users the option to enable ASPM. With the new sysfs attributes users can control which ASPM link-states are disabled.

v2:
- use a dedicated sysfs attribute per link state
- allow separate control of ASPM and PCI PM L1 sub-states

v3:
- patch 3: statically allocate the attribute group
- patch 3: replace snprintf with printf
- add patch 4

v4:
- patch 3: add call to sysfs_update_group because is_visible callback returns false always at file creation time
- patch 3: simplify code a little

v5:
- rebased to latest pci/next

v6:
- patch 3: consider several review comments from Bjorn
- patch 4: add discussion link to commit message

v7:
- Move adding pcie_aspm_get_link() to separate patch 3
- patch 4: change group name from aspm to link_pm
- patch 4: control visibility of attributes individually

Heiner Kallweit (5):
PCI/ASPM: add L1 sub-state support to pci_disable_link_state PCI/ASPM: allow to re-enable Clock PM
PCI/ASPM: Add and use helper pcie_aspm_get_link PCI/ASPM: Add sysfs attributes for controlling ASPM link states PCI/ASPM: Remove Kconfig option PCIEASPM_DEBUG and related code

 Documentation/ABI/testing/sysfs-bus-pci |  14 ++
 drivers/pci/pci-sysfs.c                 |   6 +-
 drivers/pci/pci.h                       |  12 +-
 drivers/pci/pcie/Kconfig                |   7 -
 drivers/pci/pcie/aspm.c                 | 252 ++++++++++++++++--------
 include/linux/pci.h                     |  10 +-
 6 files changed, 199 insertions(+), 102 deletions(-)

Signature · 1 line

2.23.0

Heiner Kallweit

author

2019-10-05 12:09 UTC #1 lore raw

PATCH v7 1/5 PCI/ASPM: Add L1 sub-state support to pci_disable_link_state

Add support for disabling states L1.1 and L1.2 to pci_disable_link_state(). Allow separate control of ASPM and PCI PM L1 sub-states.

Signed-off-by Heiner Kallweit <[email protected]>

---
v2:
- allow separate control of ASPM and PCI PM L1 sub-states

---
 drivers/pci/pcie/aspm.c | 11 ++++++++++-
 include/linux/pci.h     | 10 +++++++---
 2 files changed, 17 insertions(+), 4 deletions(-)

drivers/pci/pcie/aspm.c +10 -1

index 652ef23bb..ed463339e 100644
@@ -1094,7 +1094,16 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (state & PCIE_LINK_STATE_L0S)
 		link->aspm_disable |= ASPM_STATE_L0S;
 	if (state & PCIE_LINK_STATE_L1)
-		link->aspm_disable |= ASPM_STATE_L1;
+		/* sub-states require L1 */
+		link->aspm_disable |= ASPM_STATE_L1 | ASPM_STATE_L1SS;
+	if (state & PCIE_LINK_STATE_L1_1)
+		link->aspm_disable |= ASPM_STATE_L1_1;
+	if (state & PCIE_LINK_STATE_L1_2)
+		link->aspm_disable |= ASPM_STATE_L1_2;
+	if (state & PCIE_LINK_STATE_L1_1_PCIPM)
+		link->aspm_disable |= ASPM_STATE_L1_1_PCIPM;
+	if (state & PCIE_LINK_STATE_L1_2_PCIPM)
+		link->aspm_disable |= ASPM_STATE_L1_2_PCIPM;
 	pcie_config_aspm_link(link, policy_to_aspm_state(link));
 
 	if (state & PCIE_LINK_STATE_CLKPM) {

include/linux/pci.h +7 -3

index f9088c89a..9dc5bee14 100644
@@ -1544,9 +1544,13 @@ extern bool pcie_ports_native;
 #define pcie_ports_native	false
 #endif
 
-#define PCIE_LINK_STATE_L0S	1
-#define PCIE_LINK_STATE_L1	2
-#define PCIE_LINK_STATE_CLKPM	4
+#define PCIE_LINK_STATE_L0S		BIT(0)
+#define PCIE_LINK_STATE_L1		BIT(1)
+#define PCIE_LINK_STATE_CLKPM		BIT(2)
+#define PCIE_LINK_STATE_L1_1		BIT(3)
+#define PCIE_LINK_STATE_L1_2		BIT(4)
+#define PCIE_LINK_STATE_L1_1_PCIPM	BIT(5)
+#define PCIE_LINK_STATE_L1_2_PCIPM	BIT(6)
 
 #ifdef CONFIG_PCIEASPM
 int pci_disable_link_state(struct pci_dev *pdev, int state);

Signature · 1 line

2.23.0

Heiner Kallweit

author

2019-10-05 12:09 UTC #2 lore raw

PATCH v7 2/5 PCI/ASPM: Allow to re-enable Clock PM

In reply to Heiner Kallweit

So far Clock PM can't be re-enabled once it has been disabled with a call to pci_disable_link_state(). Reason is that clkpm_capable is reset. Change this by adding a clkpm_disable field similar to aspm_disable.

Signed-off-by Heiner Kallweit <[email protected]>

---
 drivers/pci/pcie/aspm.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

drivers/pci/pcie/aspm.c +11 -7

index ed463339e..574f822bf 100644
@@ -64,6 +64,7 @@ struct pcie_link_state {
 	u32 clkpm_capable:1;		/* Clock PM capable? */
 	u32 clkpm_enabled:1;		/* Current Clock PM state */
 	u32 clkpm_default:1;		/* Default Clock PM state by BIOS */
+	u32 clkpm_disable:1;		/* Clock PM disabled */
 
 	/* Exit latencies */
 	struct aspm_latency latency_up;	/* Upstream direction exit latency */

@@ -161,8 +162,11 @@ static void pcie_set_clkpm_nocheck(struct pcie_link_state *link, int enable)
 
 static void pcie_set_clkpm(struct pcie_link_state *link, int enable)
 {
-	/* Don't enable Clock PM if the link is not Clock PM capable */
-	if (!link->clkpm_capable)
+	/*
+	 * Don't enable Clock PM if the link is not Clock PM capable
+	 * or Clock PM is disabled
+	 */
+	if (!link->clkpm_capable || link->clkpm_disable)
 		enable = 0;
 	/* Need nothing if the specified equals to current state */
 	if (link->clkpm_enabled == enable)

@@ -192,7 +196,8 @@ static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist)
 	}
 	link->clkpm_enabled = enabled;
 	link->clkpm_default = enabled;
-	link->clkpm_capable = (blacklist) ? 0 : capable;
+	link->clkpm_capable = capable;
+	link->clkpm_disable = blacklist ? 1 : 0;
 }
 
 static bool pcie_retrain_link(struct pcie_link_state *link)

@@ -1106,10 +1111,9 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 		link->aspm_disable |= ASPM_STATE_L1_2_PCIPM;
 	pcie_config_aspm_link(link, policy_to_aspm_state(link));
 
-	if (state & PCIE_LINK_STATE_CLKPM) {
-		link->clkpm_capable = 0;
-		pcie_set_clkpm(link, 0);
-	}
+	if (state & PCIE_LINK_STATE_CLKPM)
+		link->clkpm_disable = 1;
+	pcie_set_clkpm(link, policy_to_clkpm_state(link));
 	mutex_unlock(&aspm_lock);
 	if (sem)
 		up_read(&pci_bus_sem);

Signature · 1 line

2.23.0

Heiner Kallweit

author

2019-10-05 12:09 UTC #3 lore raw

PATCH v7 3/5 PCI/ASPM: Add and use helper pcie_aspm_get_link

In reply to Heiner Kallweit

Factor out getting the link associated with a pci_dev and use this helper where appropriate. In addition this helper will be used in a subsequent patch of this series.

Signed-off-by Heiner Kallweit <[email protected]>

---
v7:
- add as separate patch

---
 drivers/pci/pcie/aspm.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)

drivers/pci/pcie/aspm.c +17 -9

index 574f822bf..91cfea673 100644
@@ -1066,19 +1066,28 @@ void pcie_aspm_powersave_config_link(struct pci_dev *pdev)
 	up_read(&pci_bus_sem);
 }
 
+static struct pcie_link_state *pcie_aspm_get_link(struct pci_dev *pdev)
+{
+	struct pci_dev *upstream;
+
+	if (pcie_downstream_port(pdev))
+		upstream = pdev;
+	else
+		upstream = pci_upstream_bridge(pdev);
+
+	return upstream ? upstream->link_state : NULL;
+}
+
 static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 {
-	struct pci_dev *parent = pdev->bus->self;
 	struct pcie_link_state *link;
 
 	if (!pci_is_pcie(pdev))
 		return 0;
 
-	if (pcie_downstream_port(pdev))
-		parent = pdev;
-	if (!parent || !parent->link_state)
+	link = pcie_aspm_get_link(pdev);
+	if (!link)
 		return -EINVAL;
-
 	/*
 	 * A driver requested that ASPM be disabled on this device, but
 	 * if we don't have permission to manage ASPM (e.g., on ACPI

@@ -1095,7 +1104,6 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (sem)
 		down_read(&pci_bus_sem);
 	mutex_lock(&aspm_lock);
-	link = parent->link_state;
 	if (state & PCIE_LINK_STATE_L0S)
 		link->aspm_disable |= ASPM_STATE_L0S;
 	if (state & PCIE_LINK_STATE_L1)

@@ -1188,14 +1196,14 @@ module_param_call(policy, pcie_aspm_set_policy, pcie_aspm_get_policy,
  */
 bool pcie_aspm_enabled(struct pci_dev *pdev)
 {
-	struct pci_dev *bridge = pci_upstream_bridge(pdev);
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 	bool ret;
 
-	if (!bridge)
+	if (!link)
 		return false;
 
 	mutex_lock(&aspm_lock);
-	ret = bridge->link_state ? !!bridge->link_state->aspm_enabled : false;
+	ret = link->aspm_enabled;
 	mutex_unlock(&aspm_lock);
 
 	return ret;

Signature · 1 line

2.23.0

Bjorn Helgaas

2019-10-08 01:51 UTC #4 lore raw

Re: [PATCH v7 3/5] PCI/ASPM: Add and use helper pcie_aspm_get_link

Quoted 17 lines · Heiner Kallweit

Factor out getting the link associated with a pci_dev and use this helper where appropriate. In addition this helper will be used in a subsequent patch of this series.
Signed-off-by Heiner Kallweit <[email protected]>
---
v7:
- add as separate patch
---
 drivers/pci/pcie/aspm.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
drivers/pci/pcie/aspm.c +0 -0
index 574f822bf..91cfea673 100644

drivers/pci/pcie/aspm.c void pcie_aspm_powersave_config_link(struct pci_dev *pdev) · line 1074

Quoted 9 lines

drivers/pci/pcie/aspm.c +6 -0

@@ -1066,19 +1066,28 @@ void pcie_aspm_powersave_config_link(struct pci_dev *pdev)
 	up_read(&pci_bus_sem);
 }
 
+static struct pcie_link_state *pcie_aspm_get_link(struct pci_dev *pdev)
+{
+	struct pci_dev *upstream;
+
+	if (pcie_downstream_port(pdev))
+		upstream = pdev;

Is there a case where this is called for a downstream port? After this series is completely applied, the callers I see are:

__pci_disable_link_state()
pcie_aspm_enabled()
aspm_attr_show_common()
aspm_attr_store_common()
clkpm_show()
clkpm_store()
aspm_ctrl_attrs_are_visible()

I'm pretty sure all of these only care about upstream ports, i.e., we only call them for endpoints, switch upstream ports, etc.

What do you think about something like this?

  static struct pcie_link_state *pcie_aspm_get_link(struct pci_dev *pdev)
  {
    struct pci_dev *bridge;

    if (pci_is_pcie(pdev)) {
      bridge = pci_upstream_bridge(pdev);
      if (bridge && pci_is_pcie(bridge))
        return bridge->link_state;
    }

    return NULL;

}

Then we could remove the pci_is_pcie() checks from __pci_disable_link_state() and aspm_ctrl_attrs_are_visible().

Quoted 38 lines

drivers/pci/pcie/aspm.c +9 -7

+	else
+		upstream = pci_upstream_bridge(pdev);
+
+	return upstream ? upstream->link_state : NULL;
+}
+
 static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 {
-	struct pci_dev *parent = pdev->bus->self;
 	struct pcie_link_state *link;
 
 	if (!pci_is_pcie(pdev))
 		return 0;
 
-	if (pcie_downstream_port(pdev))
-		parent = pdev;
-	if (!parent || !parent->link_state)
+	link = pcie_aspm_get_link(pdev);
+	if (!link)
 		return -EINVAL;
-
 	/*
 	 * A driver requested that ASPM be disabled on this device, but
 	 * if we don't have permission to manage ASPM (e.g., on ACPI

@@ -1095,7 +1104,6 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (sem)
 		down_read(&pci_bus_sem);
 	mutex_lock(&aspm_lock);
-	link = parent->link_state;
 	if (state & PCIE_LINK_STATE_L0S)
 		link->aspm_disable |= ASPM_STATE_L0S;
 	if (state & PCIE_LINK_STATE_L1)

@@ -1188,14 +1196,14 @@ module_param_call(policy, pcie_aspm_set_policy, pcie_aspm_get_policy,
  */
 bool pcie_aspm_enabled(struct pci_dev *pdev)
 {
-	struct pci_dev *bridge = pci_upstream_bridge(pdev);
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);

drivers/pci/pcie/aspm.c module_param_call(policy, pcie_aspm_set_policy, pcie_aspm_get_policy, · line 1207

Quoted 10 lines

drivers/pci/pcie/aspm.c +2 -2

 	bool ret;
 
-	if (!bridge)
+	if (!link)
 		return false;
 
 	mutex_lock(&aspm_lock);
-	ret = bridge->link_state ? !!bridge->link_state->aspm_enabled : false;
+	ret = link->aspm_enabled;
 	mutex_unlock(&aspm_lock);

I'm not sure why this mutex is needed; I cc'd you on my query to Rafael about this. Unrelated to your patch, of course.

Quoted 5 lines

drivers/pci/pcie/aspm.c +0 -0
 	return ret;
Signature · 1 line
2.23.0

Heiner Kallweit

author

2019-10-05 12:09 UTC #5 lore raw

PATCH v7 4/5 PCI/ASPM: Add sysfs attributes for controlling ASPM link states

In reply to Heiner Kallweit

Signed-off-by Heiner Kallweit <[email protected]>

---
v2:
- use a dedicated sysfs attribute per link state
- allow separate control of ASPM and PCI PM L1 sub-states
v3:
- statically allocate the attribute group
- replace snprintf with printf
- base on top of "PCI: Make pcie_downstream_port() available outside of access.c" v4:
- add call to sysfs_update_group because is_visible callback returns false always at file creation time
- simplify code a little
v5:
- rebased to latest pci/next
v6:
- fix style of added documentation and extend it
- don't use term "parent" and rename function to pcie_aspm_get_link
- rename pcie_check_valid_aspm_endpoint to pcie_is_aspm_dev
- enable the sysfs files also selected other port types
- avoid usage of !!
v7:
- change group name from aspm to link_pm
- control visibility of attributes individually

---
 Documentation/ABI/testing/sysfs-bus-pci |  14 ++
 drivers/pci/pci-sysfs.c                 |   3 +
 drivers/pci/pci.h                       |   4 +
 drivers/pci/pcie/aspm.c                 | 174 ++++++++++++++++++++++++
 4 files changed, 195 insertions(+)

Documentation/ABI/testing/sysfs-bus-pci +14 -0

index 8bfee557e..c86c8ab00 100644
@@ -347,3 +347,17 @@ Description:
 		If the device has any Peer-to-Peer memory registered, this
 	        file contains a '1' if the memory has been published for
 		use outside the driver that owns the device.
+
+What:		/sys/bus/pci/devices/.../link_pm/clkpm
+		/sys/bus/pci/devices/.../link_pm/l0s_aspm
+		/sys/bus/pci/devices/.../link_pm/l1_aspm
+		/sys/bus/pci/devices/.../link_pm/l1_1_aspm
+		/sys/bus/pci/devices/.../link_pm/l1_2_aspm
+		/sys/bus/pci/devices/.../link_pm/l1_1_pcipm
+		/sys/bus/pci/devices/.../link_pm/l1_2_pcipm
+Date:		October 2019
+Contact:	Heiner Kallweit <[email protected]>
+Description:	If ASPM is supported for an endpoint, then these files
+		can be used to disable or enable the individual
+		power management states. Write y/1/on to enable,
+		n/0/off to disable.

drivers/pci/pci-sysfs.c +3 -0

index 793412954..0e76c02e0 100644
@@ -1587,6 +1587,9 @@ static const struct attribute_group *pci_dev_attr_groups[] = {
 	&pcie_dev_attr_group,
 #ifdef CONFIG_PCIEAER
 	&aer_stats_attr_group,
+#endif
+#ifdef CONFIG_PCIEASPM
+	&aspm_ctrl_attr_group,
 #endif
 	NULL,
 };

drivers/pci/pci.h +4 -0

index 3f6947ee3..b2cd21e8c 100644
@@ -667,4 +667,8 @@ static inline int pci_acpi_program_hp_params(struct pci_dev *dev)
 }
 #endif
 
+#ifdef CONFIG_PCIEASPM
+extern const struct attribute_group aspm_ctrl_attr_group;
+#endif
+
 #endif /* DRIVERS_PCI_H */

drivers/pci/pcie/aspm.c +174 -0

192 lines not on this page · raw message

Signature · 1 line

2.23.0

Bjorn Helgaas

2019-10-08 01:53 UTC #6 lore raw

Re: [PATCH v7 4/5] PCI/ASPM: Add sysfs attributes for controlling ASPM link states

Quoted 13 lines · Heiner Kallweit

Background of this extension is a problem with the r8169 network driver. Several combinations of board chipsets and network chip versions have problems if ASPM is enabled, therefore we have to disable ASPM per default. However especially on notebooks ASPM can provide significant power-saving, therefore we want to give users the option to enable ASPM. With the new sysfs attributes users can control which ASPM link-states are enabled/disabled.
Signed-off-by Heiner Kallweit <[email protected]>
drivers/pci/pcie/aspm.c +3 -0
+static ssize_t aspm_attr_show_common(struct device *dev,
+				     struct device_attribute *attr,
+				     char *buf, u8 state)

drivers/pci/pcie/aspm.c void pcie_aspm_remove_sysfs_dev_files(struct pci_dev *pdev) · line 1341

Quoted 10 lines

drivers/pci/pcie/aspm.c +10 -0

+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct pcie_link_state *link;
+	bool enabled;
+
+	link = pcie_aspm_get_link(pdev);
+
+	mutex_lock(&aspm_lock);
+	enabled = link->aspm_enabled & state;
+	mutex_unlock(&aspm_lock);

Not sure the mutex is needed; do you have a reason, or is it just copied from existing code? If the latter, we can just wait to see what Rafael says.

Quoted 2 lines

(quoted diff) +2 -0

+	return sprintf(buf, "%d\n", enabled ? 1 : 0);
+}

Bjorn Helgaas

2019-11-21 20:49 UTC #7 lore raw

Re: [PATCH v7 4/5] PCI/ASPM: Add sysfs attributes for controlling ASPM link states

In reply to Heiner Kallweit

[+cc Rafael, Mika, Wong, Hui, Rajat, Keith, LKML, original patch at [5]]

Documentation/ABI/testing/sysfs-bus-pci Description: · line 363

Quoted 13 lines · Heiner Kallweit

Documentation/ABI/testing/sysfs-bus-pci +13 -0

+What:		/sys/bus/pci/devices/.../link_pm/clkpm
+		/sys/bus/pci/devices/.../link_pm/l0s_aspm
+		/sys/bus/pci/devices/.../link_pm/l1_aspm
+		/sys/bus/pci/devices/.../link_pm/l1_1_aspm
+		/sys/bus/pci/devices/.../link_pm/l1_2_aspm
+		/sys/bus/pci/devices/.../link_pm/l1_1_pcipm
+		/sys/bus/pci/devices/.../link_pm/l1_2_pcipm
+Date:		October 2019
+Contact:	Heiner Kallweit <[email protected]>
+Description:	If ASPM is supported for an endpoint, then these files
+		can be used to disable or enable the individual
+		power management states. Write y/1/on to enable,
+		n/0/off to disable.

This is queued up for the v5.5 merge window, so if we want to tweak anything (path names or otherwise), now is the time.

I think I might be inclined to change the directory from "link_pm" to "link", e.g.,

- /sys/bus/pci/devices/0000:00:1c.0/link_pm/clkpm
+ /sys/bus/pci/devices/0000:00:1c.0/link/clkpm

because there are other things that haven't been merged yet that could go in link/ as well:

* Mika's "link disable" control [1]
* Dilip's link width/speed controls [2,3]

The max_link_speed, max_link_width, current_link_speed, current_link_width files could also logically be in link/, although they've already been merged at the top level.

Rajat's AER statistics change [4] is also coming. Those stats aren't link-related, so they wouldn't go in link/. The current strawman is an "aer_stats" directory, but I wonder if we should make a more generic directory like "errors" that could be used for both AER and DPC and potentially other error-related things.

For example, we could have these link-related things:

/sys/.../0000:00:1c.0/link/clkpm # RW ASPM stuff
/sys/.../0000:00:1c.0/link/l0s_aspm
/sys/.../0000:00:1c.0/link/...

  /sys/.../0000:00:1c.0/link/disable          # RW Mika
  /sys/.../0000:00:1c.0/link/speed            # RW Dilip's control
  /sys/.../0000:00:1c.0/link/width            # RW Dilip's control
  /sys/.../0000:00:1c.0/link/max_speed        # RO possible rework
  /sys/.../0000:00:1c.0/link/max_width        # RO possible rework

With these backwards compatibility symlinks:

/sys/.../0000:00:1c.0/max_link_speed -> link/max_speed
/sys/.../0000:00:1c.0/current_link_speed -> link/speed

Rajat's current patch puts the AER stats here at the top level:

/sys/.../0000:00:1c.0/aer_stats/fatal_bit4_DLP

But maybe we could push them down like this:

/sys/.../0000:00:1c.0/errors/aer/stats/unc_04_dlp /sys/.../0000:00:1c.0/errors/aer/stats/unc_26_poison_tlb_blocked /sys/.../0000:00:1c.0/errors/aer/stats/cor_00_rx_err /sys/.../0000:00:1c.0/errors/aer/stats/cor_15_hdr_log_overflow

There are some AER-related things we don't have at all today that could go here:

/sys/.../0000:00:1c.0/errors/aer/ecrc_gen
/sys/.../0000:00:1c.0/errors/aer/ecrc_check
/sys/.../0000:00:1c.0/errors/aer/unc_err_status
/sys/.../0000:00:1c.0/errors/aer/unc_err_mask
/sys/.../0000:00:1c.0/errors/aer/unc_err_sev

And we might someday want DPC knobs like this:

/sys/.../0000:00:1c.0/errors/dpc/status
/sys/.../0000:00:1c.0/errors/dpc/error_source

Any thoughts?

Bjorn

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/d8574605f8e70f41ce1e88ccfb56b63c8f85e4df.1571638827.git.eswara.kota@linux.intel.com
[3] https://lore.kernel.org/r/[email protected]/
[4] https://lore.kernel.org/r/[email protected]
[5] https://lore.kernel.org/r/[email protected]

Rajat Jain

2019-11-21 21:03 UTC #8 lore raw

Re: [PATCH v7 4/5] PCI/ASPM: Add sysfs attributes for controlling ASPM link states

Hi,

Quoted 38 lines · Bjorn Helgaas

[+cc Rafael, Mika, Wong, Hui, Rajat, Keith, LKML, original patch at [5]]
Documentation/ABI/testing/sysfs-bus-pci Description: · line 363
Quoted 13 lines · Heiner Kallweit
Documentation/ABI/testing/sysfs-bus-pci +13 -0
+What:                /sys/bus/pci/devices/.../link_pm/clkpm
+             /sys/bus/pci/devices/.../link_pm/l0s_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_1_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_2_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_1_pcipm
+             /sys/bus/pci/devices/.../link_pm/l1_2_pcipm
+Date:                October 2019
+Contact:     Heiner Kallweit <[email protected]>
+Description: If ASPM is supported for an endpoint, then these files
+             can be used to disable or enable the individual
+             power management states. Write y/1/on to enable,
+             n/0/off to disable.
This is queued up for the v5.5 merge window, so if we want to tweak anything (path names or otherwise), now is the time.
I think I might be inclined to change the directory from "link_pm" to "link", e.g.,
- /sys/bus/pci/devices/0000:00:1c.0/link_pm/clkpm
+ /sys/bus/pci/devices/0000:00:1c.0/link/clkpm
because there are other things that haven't been merged yet that could go in link/ as well:
* Mika's "link disable" control [1]
* Dilip's link width/speed controls [2,3]
The max_link_speed, max_link_width, current_link_speed, current_link_width files could also logically be in link/, although they've already been merged at the top level.

Quoted 5 lines

Rajat's AER statistics change [4] is also coming. Those stats aren't link-related, so they wouldn't go in link/. The current strawman is an "aer_stats" directory, but I wonder if we should make a more generic directory like "errors" that could be used for both AER and DPC and potentially other error-related things.

Sorry, I haven't been able to find time for it for some time. I doubt if I'll be able to make it to 5.6 timeframe. Nevertheless...

Quoted 19 lines

For example, we could have these link-related things:
/sys/.../0000:00:1c.0/link/clkpm # RW ASPM stuff
/sys/.../0000:00:1c.0/link/l0s_aspm
/sys/.../0000:00:1c.0/link/...
  /sys/.../0000:00:1c.0/link/disable          # RW Mika
  /sys/.../0000:00:1c.0/link/speed            # RW Dilip's control
  /sys/.../0000:00:1c.0/link/width            # RW Dilip's control
  /sys/.../0000:00:1c.0/link/max_speed        # RO possible rework
  /sys/.../0000:00:1c.0/link/max_width        # RO possible rework
With these backwards compatibility symlinks:
/sys/.../0000:00:1c.0/max_link_speed -> link/max_speed
/sys/.../0000:00:1c.0/current_link_speed -> link/speed
Rajat's current patch puts the AER stats here at the top level:

Quoted 8 lines

/sys/.../0000:00:1c.0/aer_stats/fatal_bit4_DLP
But maybe we could push them down like this:
/sys/.../0000:00:1c.0/errors/aer/stats/unc_04_dlp /sys/.../0000:00:1c.0/errors/aer/stats/unc_26_poison_tlb_blocked /sys/.../0000:00:1c.0/errors/aer/stats/cor_00_rx_err /sys/.../0000:00:1c.0/errors/aer/stats/cor_15_hdr_log_overflow

How do we create sub-sub-sub directories in sysfs (errors/aer/stats)? My understanding is that we can only create 1 subdirectory by using a "named" attribute group. If we want more hierarchy, the "errors" and the "aer" will need to be backed up by a kobject. Doable, but just mentioning.

Overall the proposal looks like a step in the right direction to me.

Thanks & Best Regards,

Rajat

Quoted 24 lines

There are some AER-related things we don't have at all today that could go here:
/sys/.../0000:00:1c.0/errors/aer/ecrc_gen
/sys/.../0000:00:1c.0/errors/aer/ecrc_check
/sys/.../0000:00:1c.0/errors/aer/unc_err_status
/sys/.../0000:00:1c.0/errors/aer/unc_err_mask
/sys/.../0000:00:1c.0/errors/aer/unc_err_sev
And we might someday want DPC knobs like this:
/sys/.../0000:00:1c.0/errors/dpc/status
/sys/.../0000:00:1c.0/errors/dpc/error_source
Any thoughts?
Bjorn
[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/d8574605f8e70f41ce1e88ccfb56b63c8f85e4df.1571638827.git.eswara.kota@linux.intel.com
[3] https://lore.kernel.org/r/[email protected]/
[4] https://lore.kernel.org/r/[email protected]
[5] https://lore.kernel.org/r/[email protected]

Greg KH

2019-11-21 21:10 UTC #9 lore raw

Re: [PATCH v7 4/5] PCI/ASPM: Add sysfs attributes for controlling ASPM link states

Quoted 71 lines · Rajat Jain

Hi,
Quoted 38 lines · Bjorn Helgaas
[+cc Rafael, Mika, Wong, Hui, Rajat, Keith, LKML, original patch at [5]]
Documentation/ABI/testing/sysfs-bus-pci Description: · line 363
Quoted 13 lines · Heiner Kallweit
Documentation/ABI/testing/sysfs-bus-pci +13 -0
+What:                /sys/bus/pci/devices/.../link_pm/clkpm
+             /sys/bus/pci/devices/.../link_pm/l0s_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_1_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_2_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_1_pcipm
+             /sys/bus/pci/devices/.../link_pm/l1_2_pcipm
+Date:                October 2019
+Contact:     Heiner Kallweit <[email protected]>
+Description: If ASPM is supported for an endpoint, then these files
+             can be used to disable or enable the individual
+             power management states. Write y/1/on to enable,
+             n/0/off to disable.
This is queued up for the v5.5 merge window, so if we want to tweak anything (path names or otherwise), now is the time.
I think I might be inclined to change the directory from "link_pm" to "link", e.g.,
- /sys/bus/pci/devices/0000:00:1c.0/link_pm/clkpm
+ /sys/bus/pci/devices/0000:00:1c.0/link/clkpm
because there are other things that haven't been merged yet that could go in link/ as well:
* Mika's "link disable" control [1]
* Dilip's link width/speed controls [2,3]
The max_link_speed, max_link_width, current_link_speed, current_link_width files could also logically be in link/, although they've already been merged at the top level.
Quoted 5 lines
Rajat's AER statistics change [4] is also coming. Those stats aren't link-related, so they wouldn't go in link/. The current strawman is an "aer_stats" directory, but I wonder if we should make a more generic directory like "errors" that could be used for both AER and DPC and potentially other error-related things.
Sorry, I haven't been able to find time for it for some time. I doubt if I'll be able to make it to 5.6 timeframe. Nevertheless...
Quoted 21 lines
For example, we could have these link-related things:
/sys/.../0000:00:1c.0/link/clkpm # RW ASPM stuff
/sys/.../0000:00:1c.0/link/l0s_aspm
/sys/.../0000:00:1c.0/link/...
  /sys/.../0000:00:1c.0/link/disable          # RW Mika
  /sys/.../0000:00:1c.0/link/speed            # RW Dilip's control
  /sys/.../0000:00:1c.0/link/width            # RW Dilip's control
  /sys/.../0000:00:1c.0/link/max_speed        # RO possible rework
  /sys/.../0000:00:1c.0/link/max_width        # RO possible rework
With these backwards compatibility symlinks:
/sys/.../0000:00:1c.0/max_link_speed -> link/max_speed
/sys/.../0000:00:1c.0/current_link_speed -> link/speed
Rajat's current patch puts the AER stats here at the top level:
/sys/.../0000:00:1c.0/aer_stats/fatal_bit4_DLP

Quoted 8 lines

Quoted 6 lines
But maybe we could push them down like this:
/sys/.../0000:00:1c.0/errors/aer/stats/unc_04_dlp /sys/.../0000:00:1c.0/errors/aer/stats/unc_26_poison_tlb_blocked /sys/.../0000:00:1c.0/errors/aer/stats/cor_00_rx_err /sys/.../0000:00:1c.0/errors/aer/stats/cor_15_hdr_log_overflow
How do we create sub-sub-sub directories in sysfs (errors/aer/stats)?

You should not.

Quoted 4 lines

My understanding is that we can only create 1 subdirectory by using a "named" attribute group. If we want more hierarchy, the "errors" and the "aer" will need to be backed up by a kobject. Doable, but just mentioning.

Not doable, you break userspace tools as they will not "see" those directories or attributes.

Keep it only 1 deep if at all possible please.

thanks,

greg k-h

Bjorn Helgaas

2019-11-21 23:04 UTC #10 lore raw

Re: [PATCH v7 4/5] PCI/ASPM: Add sysfs attributes for controlling ASPM link states

[-cc Wong, Hui]

Quoted 82 lines · Greg KH

Quoted 65 lines · Rajat Jain
Quoted 35 lines · Bjorn Helgaas
Documentation/ABI/testing/sysfs-bus-pci Description: · line 363
Quoted 13 lines · Heiner Kallweit
Documentation/ABI/testing/sysfs-bus-pci +13 -0
+What:                /sys/bus/pci/devices/.../link_pm/clkpm
+             /sys/bus/pci/devices/.../link_pm/l0s_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_1_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_2_aspm
+             /sys/bus/pci/devices/.../link_pm/l1_1_pcipm
+             /sys/bus/pci/devices/.../link_pm/l1_2_pcipm
+Date:                October 2019
+Contact:     Heiner Kallweit <[email protected]>
+Description: If ASPM is supported for an endpoint, then these files
+             can be used to disable or enable the individual
+             power management states. Write y/1/on to enable,
+             n/0/off to disable.
This is queued up for the v5.5 merge window, so if we want to tweak anything (path names or otherwise), now is the time.
I think I might be inclined to change the directory from "link_pm" to "link", e.g.,
- /sys/bus/pci/devices/0000:00:1c.0/link_pm/clkpm
+ /sys/bus/pci/devices/0000:00:1c.0/link/clkpm
because there are other things that haven't been merged yet that could go in link/ as well:
* Mika's "link disable" control [1]
* Dilip's link width/speed controls [2,3]
The max_link_speed, max_link_width, current_link_speed, current_link_width files could also logically be in link/, although they've already been merged at the top level.
Quoted 5 lines
Rajat's AER statistics change [4] is also coming. Those stats aren't link-related, so they wouldn't go in link/. The current strawman is an "aer_stats" directory, but I wonder if we should make a more generic directory like "errors" that could be used for both AER and DPC and potentially other error-related things.
Sorry, I haven't been able to find time for it for some time. I doubt if I'll be able to make it to 5.6 timeframe. Nevertheless...
Quoted 20 lines
For example, we could have these link-related things:
/sys/.../0000:00:1c.0/link/clkpm # RW ASPM stuff
/sys/.../0000:00:1c.0/link/l0s_aspm
/sys/.../0000:00:1c.0/link/...
  /sys/.../0000:00:1c.0/link/disable          # RW Mika
  /sys/.../0000:00:1c.0/link/speed            # RW Dilip's control
  /sys/.../0000:00:1c.0/link/width            # RW Dilip's control
  /sys/.../0000:00:1c.0/link/max_speed        # RO possible rework
  /sys/.../0000:00:1c.0/link/max_width        # RO possible rework
With these backwards compatibility symlinks:
/sys/.../0000:00:1c.0/max_link_speed -> link/max_speed
/sys/.../0000:00:1c.0/current_link_speed -> link/speed
Rajat's current patch puts the AER stats here at the top level:
/sys/.../0000:00:1c.0/aer_stats/fatal_bit4_DLP
Quoted 8 lines
Quoted 6 lines
But maybe we could push them down like this:
/sys/.../0000:00:1c.0/errors/aer/stats/unc_04_dlp /sys/.../0000:00:1c.0/errors/aer/stats/unc_26_poison_tlb_blocked /sys/.../0000:00:1c.0/errors/aer/stats/cor_00_rx_err /sys/.../0000:00:1c.0/errors/aer/stats/cor_15_hdr_log_overflow
How do we create sub-sub-sub directories in sysfs (errors/aer/stats)?
You should not.
Quoted 4 lines
My understanding is that we can only create 1 subdirectory by using a "named" attribute group. If we want more hierarchy, the "errors" and the "aer" will need to be backed up by a kobject. Doable, but just mentioning.

Quoted 4 lines

Not doable, you break userspace tools as they will not "see" those directories or attributes.
Keep it only 1 deep if at all possible please.

Oh, that's good to know, thanks! I guess we'll have to think more about the error stuff.

What sort of tools would this break? There are no AER tools since the AER stats sysfs files don't exist yet, so I assume there are some generic sysfs tools or libraries.

Incidentally,
https://www.kernel.org/doc/html/latest/admin-guide/sysfs-rules.html
suggests that maybe we should be documenting these files with /sys/devices paths instead of the symlinks in /sys/bus/pci/devices/, e.g.,

  diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci
  -What:		/sys/bus/pci/devices/.../msi_bus
  -What:		/sys/bus/pci/devices/.../msi_irqs/
  -What:		/sys/bus/pci/devices/.../msi_irqs/<N>
  +What:		/sys/devices/pci*/.../msi_bus
  +What:		/sys/devices/pci*/.../msi_irqs/
  +What:		/sys/devices/pci*/.../msi_irqs/<N>

Greg KH

2019-11-24 17:02 UTC #11 lore raw

Re: [PATCH v7 4/5] PCI/ASPM: Add sysfs attributes for controlling ASPM link states

Quoted 3 lines · Bjorn Helgaas

What sort of tools would this break? There are no AER tools since the AER stats sysfs files don't exist yet, so I assume there are some generic sysfs tools or libraries.

Any normal tool that looks at sysfs attributes, like udev and friends. They will "miss" the uevent for the subdirs and not know how to associate anything with the "parent" struct device.

Quoted 6 lines

Incidentally,
https://www.kernel.org/doc/html/latest/admin-guide/sysfs-rules.html
suggests that maybe we should be documenting these files with /sys/devices paths instead of the symlinks in /sys/bus/pci/devices/, e.g.,

Quoted 7 lines

  diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci
  -What:		/sys/bus/pci/devices/.../msi_bus
  -What:		/sys/bus/pci/devices/.../msi_irqs/
  -What:		/sys/bus/pci/devices/.../msi_irqs/<N>
  +What:		/sys/devices/pci*/.../msi_bus
  +What:		/sys/devices/pci*/.../msi_irqs/
  +What:		/sys/devices/pci*/.../msi_irqs/<N>

Either is fine, but yes, the second one is nicer.

thanks,

greg k-h

Heiner Kallweit

author

2019-10-05 12:09 UTC #12 lore raw

PATCH v7 5/5 PCI/ASPM: Remove Kconfig option PCIEASPM_DEBUG and related code

In reply to Heiner Kallweit

Now that we have sysfs attributes for enabling/disabling the individual ASPM link states, this debug code isn't needed any longer. Removing this debug code has been discussed for quite some time, see e.g. [0].

[0] https://lore.kernel.org/lkml/[email protected]/

Signed-off-by Heiner Kallweit <[email protected]>

---
v6:
- add discussion link to commit message

---
 drivers/pci/pci-sysfs.c  |   3 --
 drivers/pci/pci.h        |   8 ---
 drivers/pci/pcie/Kconfig |   7 ---
 drivers/pci/pcie/aspm.c  | 105 ---------------------------------------
 4 files changed, 123 deletions(-)

drivers/pci/pci-sysfs.c +0 -3

index 0e76c02e0..92bf1b1ce 100644
@@ -1330,7 +1330,6 @@ static int pci_create_capabilities_sysfs(struct pci_dev *dev)
 	int retval;
 
 	pcie_vpd_create_sysfs_dev_files(dev);
-	pcie_aspm_create_sysfs_dev_files(dev);
 
 	if (dev->reset_fn) {
 		retval = device_create_file(&dev->dev, &dev_attr_reset);

@@ -1340,7 +1339,6 @@ static int pci_create_capabilities_sysfs(struct pci_dev *dev)
 	return 0;
 
 error:
-	pcie_aspm_remove_sysfs_dev_files(dev);
 	pcie_vpd_remove_sysfs_dev_files(dev);
 	return retval;
 }

@@ -1416,7 +1414,6 @@ int __must_check pci_create_sysfs_dev_files(struct pci_dev *pdev)
 static void pci_remove_capabilities_sysfs(struct pci_dev *dev)
 {
 	pcie_vpd_remove_sysfs_dev_files(dev);
-	pcie_aspm_remove_sysfs_dev_files(dev);
 	if (dev->reset_fn) {
 		device_remove_file(&dev->dev, &dev_attr_reset);
 		dev->reset_fn = 0;

drivers/pci/pci.h +0 -8

index b2cd21e8c..ae231e3cd 100644
@@ -541,14 +541,6 @@ static inline void pcie_aspm_pm_state_change(struct pci_dev *pdev) { }
 static inline void pcie_aspm_powersave_config_link(struct pci_dev *pdev) { }
 #endif
 
-#ifdef CONFIG_PCIEASPM_DEBUG
-void pcie_aspm_create_sysfs_dev_files(struct pci_dev *pdev);
-void pcie_aspm_remove_sysfs_dev_files(struct pci_dev *pdev);
-#else
-static inline void pcie_aspm_create_sysfs_dev_files(struct pci_dev *pdev) { }
-static inline void pcie_aspm_remove_sysfs_dev_files(struct pci_dev *pdev) { }
-#endif
-
 #ifdef CONFIG_PCIE_ECRC
 void pcie_set_ecrc_checking(struct pci_dev *dev);
 void pcie_ecrc_get_policy(char *str);

drivers/pci/pcie/Kconfig +0 -7

index 362eb8cfa..a2e862d4e 100644
@@ -79,13 +79,6 @@ config PCIEASPM
 
 	  When in doubt, say Y.
 
-config PCIEASPM_DEBUG
-	bool "Debug PCI Express ASPM"
-	depends on PCIEASPM
-	help
-	  This enables PCI Express ASPM debug support. It will add per-device
-	  interface to control ASPM.
-
 choice
 	prompt "Default ASPM policy"
 	default PCIEASPM_DEFAULT

drivers/pci/pcie/aspm.c +0 ~~-105~~

111 lines not on this page · raw message

Signature · 1 line

2.23.0

Bjorn Helgaas

Applied

2019-10-08 22:10 UTC #13 lore raw

In reply to Heiner Kallweit

Quoted 41 lines · Heiner Kallweit

Background of this extension is a problem with the r8169 network driver. Several combinations of board chipsets and network chip versions have problems if ASPM is enabled, therefore we have to disable ASPM per default. However especially on notebooks ASPM can provide significant power-saving, therefore we want to give users the option to enable ASPM. With the new sysfs attributes users can control which ASPM link-states are disabled.
v2:
- use a dedicated sysfs attribute per link state
- allow separate control of ASPM and PCI PM L1 sub-states
v3:
- patch 3: statically allocate the attribute group
- patch 3: replace snprintf with printf
- add patch 4
v4:
- patch 3: add call to sysfs_update_group because is_visible callback returns false always at file creation time
- patch 3: simplify code a little
v5:
- rebased to latest pci/next
v6:
- patch 3: consider several review comments from Bjorn
- patch 4: add discussion link to commit message
v7:
- Move adding pcie_aspm_get_link() to separate patch 3
- patch 4: change group name from aspm to link_pm
- patch 4: control visibility of attributes individually
Heiner Kallweit (5):
PCI/ASPM: add L1 sub-state support to pci_disable_link_state PCI/ASPM: allow to re-enable Clock PM
PCI/ASPM: Add and use helper pcie_aspm_get_link PCI/ASPM: Add sysfs attributes for controlling ASPM link states PCI/ASPM: Remove Kconfig option PCIEASPM_DEBUG and related code

Quoted 7 lines

 Documentation/ABI/testing/sysfs-bus-pci |  14 ++
 drivers/pci/pci-sysfs.c                 |   6 +-
 drivers/pci/pci.h                       |  12 +-
 drivers/pci/pcie/Kconfig                |   7 -
 drivers/pci/pcie/aspm.c                 | 252 ++++++++++++++++--------
 include/linux/pci.h                     |  10 +-
 6 files changed, 199 insertions(+), 102 deletions(-)

I applied these to pci/aspm for v5.5. Thank you very much for all the work you put into this!

There are a couple questions that are still open, but I have no problem if we want to make minor tweaks before the merge window opens.

Bjorn

Bjorn Helgaas

2019-10-10 13:22 UTC #14 lore raw

Quoted 50 lines · Bjorn Helgaas

Quoted 41 lines · Heiner Kallweit
Background of this extension is a problem with the r8169 network driver. Several combinations of board chipsets and network chip versions have problems if ASPM is enabled, therefore we have to disable ASPM per default. However especially on notebooks ASPM can provide significant power-saving, therefore we want to give users the option to enable ASPM. With the new sysfs attributes users can control which ASPM link-states are disabled.
v2:
- use a dedicated sysfs attribute per link state
- allow separate control of ASPM and PCI PM L1 sub-states
v3:
- patch 3: statically allocate the attribute group
- patch 3: replace snprintf with printf
- add patch 4
v4:
- patch 3: add call to sysfs_update_group because is_visible callback returns false always at file creation time
- patch 3: simplify code a little
v5:
- rebased to latest pci/next
v6:
- patch 3: consider several review comments from Bjorn
- patch 4: add discussion link to commit message
v7:
- Move adding pcie_aspm_get_link() to separate patch 3
- patch 4: change group name from aspm to link_pm
- patch 4: control visibility of attributes individually
Heiner Kallweit (5):
PCI/ASPM: add L1 sub-state support to pci_disable_link_state PCI/ASPM: allow to re-enable Clock PM
PCI/ASPM: Add and use helper pcie_aspm_get_link PCI/ASPM: Add sysfs attributes for controlling ASPM link states PCI/ASPM: Remove Kconfig option PCIEASPM_DEBUG and related code
Quoted 7 lines
 Documentation/ABI/testing/sysfs-bus-pci |  14 ++
 drivers/pci/pci-sysfs.c                 |   6 +-
 drivers/pci/pci.h                       |  12 +-
 drivers/pci/pcie/Kconfig                |   7 -
 drivers/pci/pcie/aspm.c                 | 252 ++++++++++++++++--------
 include/linux/pci.h                     |  10 +-
 6 files changed, 199 insertions(+), 102 deletions(-)

Quoted 5 lines

I applied these to pci/aspm for v5.5. Thank you very much for all the work you put into this!
There are a couple questions that are still open, but I have no problem if we want to make minor tweaks before the merge window opens.

To resolve these open questions, I propose the diff below, which:

- Makes pcie_aspm_get_link() work only when called for an Upstream Port (Endpoint, Switch Upstream Port, or other component at the downstream end of a Link). I don't think there's any caller that needs to supply the upstream end.

- Makes pcie_aspm_get_link() check that both ends are PCIe devices. This might be overkill, but we can't rely on the PCI topology being "correct", e.g., we have to deal gracefully with a virtualization or similar scenario where a bridge is PCI and the child is PCIe. In that case, we shouldn't try to manage ASPM, so we don't need a link_state, but I couldn't quite convince myself that pcie_aspm_init_link_state() handles these cases.

- Removes the aspm_lock from the sysfs show functions. Per the discussion with Rafael, I don't think it's necessary there:

https://lore.kernel.org/r/[email protected]

I didn't remove it from the store functions because they do ASPM reconfiguration and I didn't try to figure out the locking there.

Let me know what you think about this. If it looks right, I'll just squash these changes into the relevant patches.

drivers/pci/pcie/aspm.c +20 ~~-46~~

index 200ec994299d..83a169a196f5 100644
@@ -1078,24 +1078,22 @@ void pcie_aspm_powersave_config_link(struct pci_dev *pdev)
 
 static struct pcie_link_state *pcie_aspm_get_link(struct pci_dev *pdev)
 {
-	struct pci_dev *upstream;
+	struct pci_dev *bridge;
 
-	if (pcie_downstream_port(pdev))
-		upstream = pdev;
-	else
-		upstream = pci_upstream_bridge(pdev);
+	if (!pci_is_pcie(pdev))
+		return NULL;
+
+	bridge = pci_upstream_bridge(pdev);
+	if (!bridge || !pci_is_pcie(bridge))
+		return NULL;
 
-	return upstream ? upstream->link_state : NULL;
+	return bridge->link_state;
 }
 
 static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 {
-	struct pcie_link_state *link;
-
-	if (!pci_is_pcie(pdev))
-		return 0;
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 
-	link = pcie_aspm_get_link(pdev);
 	if (!link)
 		return -EINVAL;
 	/*

@@ -1225,16 +1223,9 @@ static ssize_t aspm_attr_show_common(struct device *dev,
 				     char *buf, u8 state)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link;
-	bool enabled;
-
-	link = pcie_aspm_get_link(pdev);
-
-	mutex_lock(&aspm_lock);
-	enabled = link->aspm_enabled & state;
-	mutex_unlock(&aspm_lock);
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 
-	return sprintf(buf, "%d\n", enabled ? 1 : 0);
+	return sprintf(buf, "%d\n", (link->aspm_enabled & state) ? 1 : 0);
 }
 
 static ssize_t aspm_attr_store_common(struct device *dev,

@@ -1242,11 +1233,9 @@ static ssize_t aspm_attr_store_common(struct device *dev,
 				      const char *buf, size_t len, u8 state)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link;
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 	bool state_enable;
 
-	link = pcie_aspm_get_link(pdev);
-
 	if (strtobool(buf, &state_enable) < 0)
 		return -EINVAL;

@@ -1291,16 +1280,9 @@ static ssize_t clkpm_show(struct device *dev,
 			  struct device_attribute *attr, char *buf)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link;
-	int val;
-
-	link = pcie_aspm_get_link(pdev);
-
-	mutex_lock(&aspm_lock);
-	val = link->clkpm_enabled;
-	mutex_unlock(&aspm_lock);
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 
-	return sprintf(buf, "%d\n", val);
+	return sprintf(buf, "%d\n", link->clkpm_enabled);
 }
 
 static ssize_t clkpm_store(struct device *dev,

@@ -1308,11 +1290,9 @@ static ssize_t clkpm_store(struct device *dev,
 			   const char *buf, size_t len)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link;
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 	bool state_enable;
 
-	link = pcie_aspm_get_link(pdev);
-
 	if (strtobool(buf, &state_enable) < 0)
 		return -EINVAL;

@@ -1352,7 +1332,7 @@ static umode_t aspm_ctrl_attrs_are_visible(struct kobject *kobj,
 {
 	struct device *dev = kobj_to_dev(kobj);
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link = NULL;
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 	static const u8 aspm_state_map[] = {
 		ASPM_STATE_L0S,
 		ASPM_STATE_L1,

@@ -1362,19 +1342,13 @@ static umode_t aspm_ctrl_attrs_are_visible(struct kobject *kobj,
 		ASPM_STATE_L1_2_PCIPM,
 	};
 
-	if (aspm_disabled)
-		return 0;
-
-	if (pci_is_pcie(pdev))
-		link = pcie_aspm_get_link(pdev);
-
-	if (!link)
+	if (aspm_disabled || !link)
 		return 0;
 
-	if (n)
-		return link->aspm_capable & aspm_state_map[n - 1] ? a->mode : 0;
-	else
+	if (n == 0)
 		return link->clkpm_capable ? a->mode : 0;
+
+	return link->aspm_capable & aspm_state_map[n - 1] ? a->mode : 0;
 }
 
 const struct attribute_group aspm_ctrl_attr_group = {

Heiner Kallweit

author

2019-10-10 20:46 UTC #15 lore raw

Quoted 75 lines · Bjorn Helgaas

Quoted 50 lines · Bjorn Helgaas
Quoted 41 lines · Heiner Kallweit
Background of this extension is a problem with the r8169 network driver. Several combinations of board chipsets and network chip versions have problems if ASPM is enabled, therefore we have to disable ASPM per default. However especially on notebooks ASPM can provide significant power-saving, therefore we want to give users the option to enable ASPM. With the new sysfs attributes users can control which ASPM link-states are disabled.
v2:
- use a dedicated sysfs attribute per link state
- allow separate control of ASPM and PCI PM L1 sub-states
v3:
- patch 3: statically allocate the attribute group
- patch 3: replace snprintf with printf
- add patch 4
v4:
- patch 3: add call to sysfs_update_group because is_visible callback returns false always at file creation time
- patch 3: simplify code a little
v5:
- rebased to latest pci/next
v6:
- patch 3: consider several review comments from Bjorn
- patch 4: add discussion link to commit message
v7:
- Move adding pcie_aspm_get_link() to separate patch 3
- patch 4: change group name from aspm to link_pm
- patch 4: control visibility of attributes individually
Heiner Kallweit (5):
PCI/ASPM: add L1 sub-state support to pci_disable_link_state PCI/ASPM: allow to re-enable Clock PM
PCI/ASPM: Add and use helper pcie_aspm_get_link PCI/ASPM: Add sysfs attributes for controlling ASPM link states PCI/ASPM: Remove Kconfig option PCIEASPM_DEBUG and related code
Quoted 7 lines
 Documentation/ABI/testing/sysfs-bus-pci |  14 ++
 drivers/pci/pci-sysfs.c                 |   6 +-
 drivers/pci/pci.h                       |  12 +-
 drivers/pci/pcie/Kconfig                |   7 -
 drivers/pci/pcie/aspm.c                 | 252 ++++++++++++++++--------
 include/linux/pci.h                     |  10 +-
 6 files changed, 199 insertions(+), 102 deletions(-)
Quoted 5 lines
I applied these to pci/aspm for v5.5. Thank you very much for all the work you put into this!
There are a couple questions that are still open, but I have no problem if we want to make minor tweaks before the merge window opens.
To resolve these open questions, I propose the diff below, which:
- Makes pcie_aspm_get_link() work only when called for an Upstream Port (Endpoint, Switch Upstream Port, or other component at the downstream end of a Link). I don't think there's any caller that needs to supply the upstream end.
- Makes pcie_aspm_get_link() check that both ends are PCIe devices. This might be overkill, but we can't rely on the PCI topology being "correct", e.g., we have to deal gracefully with a virtualization or similar scenario where a bridge is PCI and the child is PCIe. In that case, we shouldn't try to manage ASPM, so we don't need a link_state, but I couldn't quite convince myself that pcie_aspm_init_link_state() handles these cases.
- Removes the aspm_lock from the sysfs show functions. Per the discussion with Rafael, I don't think it's necessary there:

Quoted 8 lines

https://lore.kernel.org/r/[email protected]
I didn't remove it from the store functions because they do ASPM reconfiguration and I didn't try to figure out the locking there.
Let me know what you think about this. If it looks right, I'll just squash these changes into the relevant patches.

Looks good to me. Thanks, Heiner

Quoted 136 lines

drivers/pci/pcie/aspm.c +20 ~~-46~~

index 200ec994299d..83a169a196f5 100644
@@ -1078,24 +1078,22 @@ void pcie_aspm_powersave_config_link(struct pci_dev *pdev)
 
 static struct pcie_link_state *pcie_aspm_get_link(struct pci_dev *pdev)
 {
-	struct pci_dev *upstream;
+	struct pci_dev *bridge;
 
-	if (pcie_downstream_port(pdev))
-		upstream = pdev;
-	else
-		upstream = pci_upstream_bridge(pdev);
+	if (!pci_is_pcie(pdev))
+		return NULL;
+
+	bridge = pci_upstream_bridge(pdev);
+	if (!bridge || !pci_is_pcie(bridge))
+		return NULL;
 
-	return upstream ? upstream->link_state : NULL;
+	return bridge->link_state;
 }
 
 static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 {
-	struct pcie_link_state *link;
-
-	if (!pci_is_pcie(pdev))
-		return 0;
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 
-	link = pcie_aspm_get_link(pdev);
 	if (!link)
 		return -EINVAL;
 	/*

@@ -1225,16 +1223,9 @@ static ssize_t aspm_attr_show_common(struct device *dev,
 				     char *buf, u8 state)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link;
-	bool enabled;
-
-	link = pcie_aspm_get_link(pdev);
-
-	mutex_lock(&aspm_lock);
-	enabled = link->aspm_enabled & state;
-	mutex_unlock(&aspm_lock);
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 
-	return sprintf(buf, "%d\n", enabled ? 1 : 0);
+	return sprintf(buf, "%d\n", (link->aspm_enabled & state) ? 1 : 0);
 }
 
 static ssize_t aspm_attr_store_common(struct device *dev,

@@ -1242,11 +1233,9 @@ static ssize_t aspm_attr_store_common(struct device *dev,
 				      const char *buf, size_t len, u8 state)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link;
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 	bool state_enable;
 
-	link = pcie_aspm_get_link(pdev);
-
 	if (strtobool(buf, &state_enable) < 0)
 		return -EINVAL;

@@ -1291,16 +1280,9 @@ static ssize_t clkpm_show(struct device *dev,
 			  struct device_attribute *attr, char *buf)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link;
-	int val;
-
-	link = pcie_aspm_get_link(pdev);
-
-	mutex_lock(&aspm_lock);
-	val = link->clkpm_enabled;
-	mutex_unlock(&aspm_lock);
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 
-	return sprintf(buf, "%d\n", val);
+	return sprintf(buf, "%d\n", link->clkpm_enabled);
 }
 
 static ssize_t clkpm_store(struct device *dev,

@@ -1308,11 +1290,9 @@ static ssize_t clkpm_store(struct device *dev,
 			   const char *buf, size_t len)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link;
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 	bool state_enable;
 
-	link = pcie_aspm_get_link(pdev);
-
 	if (strtobool(buf, &state_enable) < 0)
 		return -EINVAL;

@@ -1352,7 +1332,7 @@ static umode_t aspm_ctrl_attrs_are_visible(struct kobject *kobj,
 {
 	struct device *dev = kobj_to_dev(kobj);
 	struct pci_dev *pdev = to_pci_dev(dev);
-	struct pcie_link_state *link = NULL;
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 	static const u8 aspm_state_map[] = {
 		ASPM_STATE_L0S,
 		ASPM_STATE_L1,

@@ -1362,19 +1342,13 @@ static umode_t aspm_ctrl_attrs_are_visible(struct kobject *kobj,
 		ASPM_STATE_L1_2_PCIPM,
 	};
 
-	if (aspm_disabled)
-		return 0;
-
-	if (pci_is_pcie(pdev))
-		link = pcie_aspm_get_link(pdev);
-
-	if (!link)
+	if (aspm_disabled || !link)
 		return 0;
 
-	if (n)
-		return link->aspm_capable & aspm_state_map[n - 1] ? a->mode : 0;
-	else
+	if (n == 0)
 		return link->clkpm_capable ? a->mode : 0;
+
+	return link->aspm_capable & aspm_state_map[n - 1] ? a->mode : 0;
 }
 
 const struct attribute_group aspm_ctrl_attr_group = {

Bjorn Helgaas

2019-10-15 20:30 UTC #16 lore raw

Quoted 78 lines · Heiner Kallweit

Quoted 77 lines · Bjorn Helgaas
Quoted 50 lines · Bjorn Helgaas
Quoted 41 lines · Heiner Kallweit
Background of this extension is a problem with the r8169 network driver. Several combinations of board chipsets and network chip versions have problems if ASPM is enabled, therefore we have to disable ASPM per default. However especially on notebooks ASPM can provide significant power-saving, therefore we want to give users the option to enable ASPM. With the new sysfs attributes users can control which ASPM link-states are disabled.
v2:
- use a dedicated sysfs attribute per link state
- allow separate control of ASPM and PCI PM L1 sub-states
v3:
- patch 3: statically allocate the attribute group
- patch 3: replace snprintf with printf
- add patch 4
v4:
- patch 3: add call to sysfs_update_group because is_visible callback returns false always at file creation time
- patch 3: simplify code a little
v5:
- rebased to latest pci/next
v6:
- patch 3: consider several review comments from Bjorn
- patch 4: add discussion link to commit message
v7:
- Move adding pcie_aspm_get_link() to separate patch 3
- patch 4: change group name from aspm to link_pm
- patch 4: control visibility of attributes individually
Heiner Kallweit (5):
PCI/ASPM: add L1 sub-state support to pci_disable_link_state PCI/ASPM: allow to re-enable Clock PM
PCI/ASPM: Add and use helper pcie_aspm_get_link PCI/ASPM: Add sysfs attributes for controlling ASPM link states PCI/ASPM: Remove Kconfig option PCIEASPM_DEBUG and related code
Quoted 7 lines
 Documentation/ABI/testing/sysfs-bus-pci |  14 ++
 drivers/pci/pci-sysfs.c                 |   6 +-
 drivers/pci/pci.h                       |  12 +-
 drivers/pci/pcie/Kconfig                |   7 -
 drivers/pci/pcie/aspm.c                 | 252 ++++++++++++++++--------
 include/linux/pci.h                     |  10 +-
 6 files changed, 199 insertions(+), 102 deletions(-)
Quoted 5 lines
I applied these to pci/aspm for v5.5. Thank you very much for all the work you put into this!
There are a couple questions that are still open, but I have no problem if we want to make minor tweaks before the merge window opens.
To resolve these open questions, I propose the diff below, which:
- Makes pcie_aspm_get_link() work only when called for an Upstream Port (Endpoint, Switch Upstream Port, or other component at the downstream end of a Link). I don't think there's any caller that needs to supply the upstream end.
- Makes pcie_aspm_get_link() check that both ends are PCIe devices. This might be overkill, but we can't rely on the PCI topology being "correct", e.g., we have to deal gracefully with a virtualization or similar scenario where a bridge is PCI and the child is PCIe. In that case, we shouldn't try to manage ASPM, so we don't need a link_state, but I couldn't quite convince myself that pcie_aspm_init_link_state() handles these cases.
- Removes the aspm_lock from the sysfs show functions. Per the discussion with Rafael, I don't think it's necessary there:
https://lore.kernel.org/r/[email protected]

Quoted 7 lines

Quoted 6 lines
I didn't remove it from the store functions because they do ASPM reconfiguration and I didn't try to figure out the locking there.
Let me know what you think about this. If it looks right, I'll just squash these changes into the relevant patches.
Looks good to me. Thanks, Heiner

Thanks, I squashed these in and updated pci/aspm.