Skip to contentSkip to conversation

Updates to mlxbf-pmc

PATCH Superseded Shravan Kumar Ramani 2 participants 13 messages May 20, 2024 – Jun 14, 2024

Subsystems (2)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover Updates to mlxbf-pmc

This submission contains 4 patches relating to mlxbf-pmc.

Patch 1 adds documentation for the sysfs files created by the driver. Patches 2 and 3 add specific functionality to the driver for supporting 64-bit ocunters, cycle count and clock_measure performance block. Patch 4 adds documentation for the newly added sysfs entries.

v1 -> v2
Added patch 4 to document sysfs entries added in patches 2 and 3.

Shravan Kumar Ramani (4):
Documentation/ABI: Add document for Mellanox PMC driver platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and
cycle count
platform/mellanox: mlxbf-pmc: Add support for clock_measure
performance block
Documentation/ABI: Add new sysfs fields to sysfs-platform-mellanox-pmc

 .../ABI/testing/sysfs-platform-mellanox-pmc   |  65 +++++++
 drivers/platform/mellanox/mlxbf-pmc.c         | 180 +++++++++++++++++-
 2 files changed, 241 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-mellanox-pmc
Signature · 1 line
2.30.1
#1 lore raw

PATCH v2 1/4 Documentation/ABI: Add document for Mellanox PMC driver

The sysfs interface is created for programming and monitoring the performance counters in various HW blocks of Mellanox BlueField-1, BlueField-2 and BlueField-3.

---
 .../ABI/testing/sysfs-platform-mellanox-pmc   | 49 +++++++++++++++++++
 1 file changed, 49 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-mellanox-pmc
Documentation/ABI/testing/sysfs-platform-mellanox-pmc new file +49 -0
new file mode 100644
index 000000000000..47094024dbeb
@@ -0,0 +1,49 @@
+HID           Driver         Description
+MLNXBFD0      mlxbf-pmc      Performance counters (BlueField-1)
+MLNXBFD1      mlxbf-pmc      Performance counters (BlueField-2)
+MLNXBFD2      mlxbf-pmc      Performance counters (BlueField-3)
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/event_list
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		List of events supported by the counters in the specific block.
+		It is used to extract the event number or ID associated with
+		each event.
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/event<N>
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Event monitored by corresponding counter. This is used to
+		program or read back the event that should be or is currently
+		being monitored by counter<N>.
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/counter<N>
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Counter value of the event being monitored. This is used to
+		read the counter value of the event which was programmed using
+		event<N>. This is also used to clear or reset the counter value.
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/enable
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Start or stop counters. This is used to start the counters
+		for monitoring the programmed events and also to stop the
+		counters after the desired duration.
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/<reg>
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Value of register. This is used to read or reset the registers
+		where various performance statistics are counted for each block.
+
Signature · 1 line
2.30.1
maintainer
#2 lore raw

Re: [PATCH v2 1/4] Documentation/ABI: Add document for Mellanox PMC driver

Quoted 1 line · Shravan Kumar Ramani

The sysfs interface is created for programming and monitoring the

This patch is not creating interface. I suggest you change the wording to:

"Document the sysfs interface for ...

Quoted 37 lines

performance counters in various HW blocks of Mellanox BlueField-1, BlueField-2 and BlueField-3.

---
 .../ABI/testing/sysfs-platform-mellanox-pmc   | 49 +++++++++++++++++++
 1 file changed, 49 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-mellanox-pmc
Documentation/ABI/testing/sysfs-platform-mellanox-pmc new file +21 -0
new file mode 100644
index 000000000000..47094024dbeb
@@ -0,0 +1,49 @@
+HID           Driver         Description
+MLNXBFD0      mlxbf-pmc      Performance counters (BlueField-1)
+MLNXBFD1      mlxbf-pmc      Performance counters (BlueField-2)
+MLNXBFD2      mlxbf-pmc      Performance counters (BlueField-3)
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/event_list
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		List of events supported by the counters in the specific block.
+		It is used to extract the event number or ID associated with
+		each event.
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/event<N>
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Event monitored by corresponding counter. This is used to
+		program or read back the event that should be or is currently

Documentation/ABI/testing/sysfs-platform-mellanox-pmc line 31

Quoted 10 lines
Documentation/ABI/testing/sysfs-platform-mellanox-pmc new file +10 -0
+		being monitored by counter<N>.
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/counter<N>
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Counter value of the event being monitored. This is used to
+		read the counter value of the event which was programmed using
+		event<N>. This is also used to clear or reset the counter value.

Please document how to clear/reset it in more concrete terms (I didn't check the code but my guess would one writes zero there to reset the counter?).

Quoted 6 lines
Documentation/ABI/testing/sysfs-platform-mellanox-pmc new file +6 -0
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/enable
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Start or stop counters. This is used to start the counters

Documentation/ABI/testing/sysfs-platform-mellanox-pmc line 48

Quoted 10 lines
Documentation/ABI/testing/sysfs-platform-mellanox-pmc new file +10 -0
+		for monitoring the programmed events and also to stop the
+		counters after the desired duration.
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/<reg>
+Date:		Dec 2020
+KernelVersion:	5.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Value of register. This is used to read or reset the registers
+		where various performance statistics are counted for each block.

Ditto for the reset part.

Signature · 1 line
 i.
#3 lore raw

PATCH v2 2/4 platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and cycle count

In reply to Shravan Kumar Ramani

Add support for programming any counter to monitor the cycle count. Since counting of cycles using 32-bit ocunters would result in frequent wraparounds, add the ability to combine 2 adjacent 32-bit counters to form 1 64-bit counter.
Both these features are supported by BlueField-3 PMC hardware, hence the required bit-fields are exposed by the driver via sysfs to allow the user to configure as needed.

---
 drivers/platform/mellanox/mlxbf-pmc.c | 134 ++++++++++++++++++++++++++
 1 file changed, 134 insertions(+)
drivers/platform/mellanox/mlxbf-pmc.c +134 -0
index 4ed9c7fd2b62..635ecc3b3845 100644
@@ -88,6 +88,8 @@
 #define MLXBF_PMC_CRSPACE_PERFMON_CTL(n) (n * MLXBF_PMC_CRSPACE_PERFMON_REG0_SZ)
 #define MLXBF_PMC_CRSPACE_PERFMON_EN BIT(30)
 #define MLXBF_PMC_CRSPACE_PERFMON_CLR BIT(28)
+#define MLXBF_PMC_CRSPACE_PERFMON_UOC GENMASK(15, 0)
+#define MLXBF_PMC_CRSPACE_PERFMON_COUNT_CLOCK(n) (MLXBF_PMC_CRSPACE_PERFMON_CTL(n) + 0x4)
 #define MLXBF_PMC_CRSPACE_PERFMON_VAL0(n) (MLXBF_PMC_CRSPACE_PERFMON_CTL(n) + 0xc)
 
 /**
@@ -114,6 +116,8 @@ struct mlxbf_pmc_attribute {
  * @attr_event: Attributes for "event" sysfs files
  * @attr_event_list: Attributes for "event_list" sysfs files
  * @attr_enable: Attributes for "enable" sysfs files
+ * @attr_use_odd_counter: Attributes for "use_odd_counter" sysfs files
+ * @attr_count_clock: Attributes for "count_clock" sysfs files
  * @block_attr: All attributes needed for the block
  * @block_attr_grp: Attribute group for the block
  */
@@ -126,6 +130,8 @@ struct mlxbf_pmc_block_info {
 	struct mlxbf_pmc_attribute *attr_event;
 	struct mlxbf_pmc_attribute attr_event_list;
 	struct mlxbf_pmc_attribute attr_enable;
+	struct mlxbf_pmc_attribute attr_use_odd_counter;
+	struct mlxbf_pmc_attribute attr_count_clock;
 	struct attribute *block_attr[MLXBF_PMC_MAX_ATTRS];
 	struct attribute_group block_attr_grp;
 };
@@ -1763,6 +1769,103 @@ static ssize_t mlxbf_pmc_enable_store(struct device *dev,
 	return count;
 }
 
+/* Show function for "use_odd_counter" sysfs files - only for crspace */
+static ssize_t mlxbf_pmc_use_odd_counter_show(struct device *dev,
+					      struct device_attribute *attr, char *buf)
+{
+	struct mlxbf_pmc_attribute *attr_use_odd_counter = container_of(
+		attr, struct mlxbf_pmc_attribute, dev_attr);
+	unsigned int blk_num;
+	u32 value, reg;
+
+	blk_num = attr_use_odd_counter->nr;
+
+	if (mlxbf_pmc_readl(pmc->block[blk_num].mmio_base +
+			MLXBF_PMC_CRSPACE_PERFMON_CTL(pmc->block[blk_num].counters),
+			&reg))
+		return -EINVAL;
+
+	value = FIELD_GET(MLXBF_PMC_CRSPACE_PERFMON_UOC, reg);
+
+	return sysfs_emit(buf, "%u\n", value);
+}
+
+/* Store function for "use_odd_counter" sysfs files - only for crspace */
+static ssize_t mlxbf_pmc_use_odd_counter_store(struct device *dev,
+					       struct device_attribute *attr,
+					       const char *buf, size_t count)
+{
+	struct mlxbf_pmc_attribute *attr_use_odd_counter = container_of(
+		attr, struct mlxbf_pmc_attribute, dev_attr);
+	unsigned int blk_num;
+	u32 uoc, reg;
+	int err;
+
+	blk_num = attr_use_odd_counter->nr;
+
+	err = kstrtouint(buf, 0, &uoc);
+	if (err < 0)
+		return err;
+
+	err = mlxbf_pmc_readl(pmc->block[blk_num].mmio_base +
+		MLXBF_PMC_CRSPACE_PERFMON_CTL(pmc->block[blk_num].counters),
+		&reg);
+	if (err)
+		return -EINVAL;
+
+	reg &= ~MLXBF_PMC_CRSPACE_PERFMON_UOC;
+	reg |= FIELD_PREP(MLXBF_PMC_CRSPACE_PERFMON_UOC, uoc);
+
+	mlxbf_pmc_write(pmc->block[blk_num].mmio_base +
+		MLXBF_PMC_CRSPACE_PERFMON_CTL(pmc->block[blk_num].counters),
+		MLXBF_PMC_WRITE_REG_32, reg);
+
+	return count;
+}
+
+/* Show function for "count_clock" sysfs files - only for crspace */
+static ssize_t mlxbf_pmc_count_clock_show(struct device *dev,
+					  struct device_attribute *attr, char *buf)
+{
+	struct mlxbf_pmc_attribute *attr_count_clock = container_of(
+		attr, struct mlxbf_pmc_attribute, dev_attr);
+	unsigned int blk_num;
+	u32 reg;
+
+	blk_num = attr_count_clock->nr;
+
+	if (mlxbf_pmc_readl(pmc->block[blk_num].mmio_base +
+			MLXBF_PMC_CRSPACE_PERFMON_COUNT_CLOCK(pmc->block[blk_num].counters),
+			&reg))
+		return -EINVAL;
+
+	return sysfs_emit(buf, "%u\n", reg);
+}
+
+/* Store function for "count_clock" sysfs files - only for crspace */
+static ssize_t mlxbf_pmc_count_clock_store(struct device *dev,
+					   struct device_attribute *attr,
+					   const char *buf, size_t count)
+{
+	struct mlxbf_pmc_attribute *attr_count_clock = container_of(
+		attr, struct mlxbf_pmc_attribute, dev_attr);
+	unsigned int blk_num;
+	u32 reg;
+	int err;
+
+	blk_num = attr_count_clock->nr;
+
+	err = kstrtouint(buf, 0, &reg);
+	if (err < 0)
+		return err;
+
+	mlxbf_pmc_write(pmc->block[blk_num].mmio_base +
+		MLXBF_PMC_CRSPACE_PERFMON_COUNT_CLOCK(pmc->block[blk_num].counters),
+		MLXBF_PMC_WRITE_REG_32, reg);
+
+	return count;
+}
+
 /* Populate attributes for blocks with counters to monitor performance */
 static int mlxbf_pmc_init_perftype_counter(struct device *dev, unsigned int blk_num)
 {
@@ -1799,6 +1902,37 @@ static int mlxbf_pmc_init_perftype_counter(struct device *dev, unsigned int blk_
 		attr = NULL;
 	}
 
+	if (pmc->block[blk_num].type == MLXBF_PMC_TYPE_CRSPACE) {
+		/*
+		 * Couple adjacent odd and even 32-bit counters to form 64-bit counters
+		 * using "use_odd_counter" sysfs which has one bit per even counter.
+		 */
+		attr = &pmc->block[blk_num].attr_use_odd_counter;
+		attr->dev_attr.attr.mode = 0644;
+		attr->dev_attr.show = mlxbf_pmc_use_odd_counter_show;
+		attr->dev_attr.store = mlxbf_pmc_use_odd_counter_store;
+		attr->nr = blk_num;
+		attr->dev_attr.attr.name = devm_kasprintf(dev, GFP_KERNEL,
+							  "use_odd_counter");
+		if (!attr->dev_attr.attr.name)
+			return -ENOMEM;
+		pmc->block[blk_num].block_attr[++i] = &attr->dev_attr.attr;
+		attr = NULL;
+
+		/* Program crspace counters to count clock cycles using "count_clock" sysfs */
+		attr = &pmc->block[blk_num].attr_count_clock;
+		attr->dev_attr.attr.mode = 0644;
+		attr->dev_attr.show = mlxbf_pmc_count_clock_show;
+		attr->dev_attr.store = mlxbf_pmc_count_clock_store;
+		attr->nr = blk_num;
+		attr->dev_attr.attr.name = devm_kasprintf(dev, GFP_KERNEL,
+							  "count_clock");
+		if (!attr->dev_attr.attr.name)
+			return -ENOMEM;
+		pmc->block[blk_num].block_attr[++i] = &attr->dev_attr.attr;
+		attr = NULL;
+	}
+
 	pmc->block[blk_num].attr_counter = devm_kcalloc(
 		dev, pmc->block[blk_num].counters,
 		sizeof(struct mlxbf_pmc_attribute), GFP_KERNEL);
Signature · 1 line
2.30.1
maintainer
#4 lore raw

Re: [PATCH v2 2/4] platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and cycle count

Quoted 2 lines · Shravan Kumar Ramani

Add support for programming any counter to monitor the cycle count. Since counting of cycles using 32-bit ocunters would result in frequent

counters

Quoted 5 lines

wraparounds, add the ability to combine 2 adjacent 32-bit counters to form 1 64-bit counter.
Both these features are supported by BlueField-3 PMC hardware, hence the required bit-fields are exposed by the driver via sysfs to allow the user to configure as needed.

I'm trying to understand what happens for the other counter, when the use_odd_counter is enabled? This change also doesn't add code that would make the other counter -EBUSY, should that be done?

For 64-bit counter, I suppose the userspace is expected to read the full counter from two sysfs files and combine the value (your documentation doesn't explain this)? That seems non-optimal, why cannot kernel just return the full combined 64-value directly in kernel?

Similarly, are these cycle counters occupying the same space as non-cycle counters (so both can/cannot be used that the same time)? I'm asking this because you're adding a parallel interface to read the value and if it's either-or, I don't understand why the value needs to be read from different file depending on the counter counting in cycles or not.

Signature · 1 line
 i.
Quoted 176 lines
---
 drivers/platform/mellanox/mlxbf-pmc.c | 134 ++++++++++++++++++++++++++
 1 file changed, 134 insertions(+)
drivers/platform/mellanox/mlxbf-pmc.c +133 -0
index 4ed9c7fd2b62..635ecc3b3845 100644
@@ -88,6 +88,8 @@
 #define MLXBF_PMC_CRSPACE_PERFMON_CTL(n) (n * MLXBF_PMC_CRSPACE_PERFMON_REG0_SZ)
 #define MLXBF_PMC_CRSPACE_PERFMON_EN BIT(30)
 #define MLXBF_PMC_CRSPACE_PERFMON_CLR BIT(28)
+#define MLXBF_PMC_CRSPACE_PERFMON_UOC GENMASK(15, 0)
+#define MLXBF_PMC_CRSPACE_PERFMON_COUNT_CLOCK(n) (MLXBF_PMC_CRSPACE_PERFMON_CTL(n) + 0x4)
 #define MLXBF_PMC_CRSPACE_PERFMON_VAL0(n) (MLXBF_PMC_CRSPACE_PERFMON_CTL(n) + 0xc)
 
 /**
@@ -114,6 +116,8 @@ struct mlxbf_pmc_attribute {
  * @attr_event: Attributes for "event" sysfs files
  * @attr_event_list: Attributes for "event_list" sysfs files
  * @attr_enable: Attributes for "enable" sysfs files
+ * @attr_use_odd_counter: Attributes for "use_odd_counter" sysfs files
+ * @attr_count_clock: Attributes for "count_clock" sysfs files
  * @block_attr: All attributes needed for the block
  * @block_attr_grp: Attribute group for the block
  */
@@ -126,6 +130,8 @@ struct mlxbf_pmc_block_info {
 	struct mlxbf_pmc_attribute *attr_event;
 	struct mlxbf_pmc_attribute attr_event_list;
 	struct mlxbf_pmc_attribute attr_enable;
+	struct mlxbf_pmc_attribute attr_use_odd_counter;
+	struct mlxbf_pmc_attribute attr_count_clock;
 	struct attribute *block_attr[MLXBF_PMC_MAX_ATTRS];
 	struct attribute_group block_attr_grp;
 };
@@ -1763,6 +1769,103 @@ static ssize_t mlxbf_pmc_enable_store(struct device *dev,
 	return count;
 }
 
+/* Show function for "use_odd_counter" sysfs files - only for crspace */
+static ssize_t mlxbf_pmc_use_odd_counter_show(struct device *dev,
+					      struct device_attribute *attr, char *buf)
+{
+	struct mlxbf_pmc_attribute *attr_use_odd_counter = container_of(
+		attr, struct mlxbf_pmc_attribute, dev_attr);
+	unsigned int blk_num;
+	u32 value, reg;
+
+	blk_num = attr_use_odd_counter->nr;
+
+	if (mlxbf_pmc_readl(pmc->block[blk_num].mmio_base +
+			MLXBF_PMC_CRSPACE_PERFMON_CTL(pmc->block[blk_num].counters),
+			&reg))
+		return -EINVAL;
+
+	value = FIELD_GET(MLXBF_PMC_CRSPACE_PERFMON_UOC, reg);
+
+	return sysfs_emit(buf, "%u\n", value);
+}
+
+/* Store function for "use_odd_counter" sysfs files - only for crspace */
+static ssize_t mlxbf_pmc_use_odd_counter_store(struct device *dev,
+					       struct device_attribute *attr,
+					       const char *buf, size_t count)
+{
+	struct mlxbf_pmc_attribute *attr_use_odd_counter = container_of(
+		attr, struct mlxbf_pmc_attribute, dev_attr);
+	unsigned int blk_num;
+	u32 uoc, reg;
+	int err;
+
+	blk_num = attr_use_odd_counter->nr;
+
+	err = kstrtouint(buf, 0, &uoc);
+	if (err < 0)
+		return err;
+
+	err = mlxbf_pmc_readl(pmc->block[blk_num].mmio_base +
+		MLXBF_PMC_CRSPACE_PERFMON_CTL(pmc->block[blk_num].counters),
+		&reg);
+	if (err)
+		return -EINVAL;
+
+	reg &= ~MLXBF_PMC_CRSPACE_PERFMON_UOC;
+	reg |= FIELD_PREP(MLXBF_PMC_CRSPACE_PERFMON_UOC, uoc);
+
+	mlxbf_pmc_write(pmc->block[blk_num].mmio_base +
+		MLXBF_PMC_CRSPACE_PERFMON_CTL(pmc->block[blk_num].counters),
+		MLXBF_PMC_WRITE_REG_32, reg);
+
+	return count;
+}
+
+/* Show function for "count_clock" sysfs files - only for crspace */
+static ssize_t mlxbf_pmc_count_clock_show(struct device *dev,
+					  struct device_attribute *attr, char *buf)
+{
+	struct mlxbf_pmc_attribute *attr_count_clock = container_of(
+		attr, struct mlxbf_pmc_attribute, dev_attr);
+	unsigned int blk_num;
+	u32 reg;
+
+	blk_num = attr_count_clock->nr;
+
+	if (mlxbf_pmc_readl(pmc->block[blk_num].mmio_base +
+			MLXBF_PMC_CRSPACE_PERFMON_COUNT_CLOCK(pmc->block[blk_num].counters),
+			&reg))
+		return -EINVAL;
+
+	return sysfs_emit(buf, "%u\n", reg);
+}
+
+/* Store function for "count_clock" sysfs files - only for crspace */
+static ssize_t mlxbf_pmc_count_clock_store(struct device *dev,
+					   struct device_attribute *attr,
+					   const char *buf, size_t count)
+{
+	struct mlxbf_pmc_attribute *attr_count_clock = container_of(
+		attr, struct mlxbf_pmc_attribute, dev_attr);
+	unsigned int blk_num;
+	u32 reg;
+	int err;
+
+	blk_num = attr_count_clock->nr;
+
+	err = kstrtouint(buf, 0, &reg);
+	if (err < 0)
+		return err;
+
+	mlxbf_pmc_write(pmc->block[blk_num].mmio_base +
+		MLXBF_PMC_CRSPACE_PERFMON_COUNT_CLOCK(pmc->block[blk_num].counters),
+		MLXBF_PMC_WRITE_REG_32, reg);
+
+	return count;
+}
+
 /* Populate attributes for blocks with counters to monitor performance */
 static int mlxbf_pmc_init_perftype_counter(struct device *dev, unsigned int blk_num)
 {
@@ -1799,6 +1902,37 @@ static int mlxbf_pmc_init_perftype_counter(struct device *dev, unsigned int blk_
 		attr = NULL;
 	}
 
+	if (pmc->block[blk_num].type == MLXBF_PMC_TYPE_CRSPACE) {
+		/*
+		 * Couple adjacent odd and even 32-bit counters to form 64-bit counters
+		 * using "use_odd_counter" sysfs which has one bit per even counter.
+		 */
+		attr = &pmc->block[blk_num].attr_use_odd_counter;
+		attr->dev_attr.attr.mode = 0644;
+		attr->dev_attr.show = mlxbf_pmc_use_odd_counter_show;
+		attr->dev_attr.store = mlxbf_pmc_use_odd_counter_store;
+		attr->nr = blk_num;
+		attr->dev_attr.attr.name = devm_kasprintf(dev, GFP_KERNEL,
+							  "use_odd_counter");
+		if (!attr->dev_attr.attr.name)
+			return -ENOMEM;
+		pmc->block[blk_num].block_attr[++i] = &attr->dev_attr.attr;
+		attr = NULL;
+
+		/* Program crspace counters to count clock cycles using "count_clock" sysfs */
+		attr = &pmc->block[blk_num].attr_count_clock;
+		attr->dev_attr.attr.mode = 0644;
+		attr->dev_attr.show = mlxbf_pmc_count_clock_show;
+		attr->dev_attr.store = mlxbf_pmc_count_clock_store;
+		attr->nr = blk_num;
+		attr->dev_attr.attr.name = devm_kasprintf(dev, GFP_KERNEL,
+							  "count_clock");
+		if (!attr->dev_attr.attr.name)
+			return -ENOMEM;
+		pmc->block[blk_num].block_attr[++i] = &attr->dev_attr.attr;
+		attr = NULL;
+	}
#5 lore raw

Re: [PATCH v2 2/4] platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and cycle count

Quoted 8 lines
Quoted 4 lines

Both these features are supported by BlueField-3 PMC hardware, hence the required bit-fields are exposed by the driver via sysfs to allow the user to configure as needed.

I'm trying to understand what happens for the other counter, when the use_odd_counter is enabled? This change also doesn't add code that would make the other counter -EBUSY, should that be done?

When 2 32-bit counters are coupled to form a 64-bit counter using this setting, one counter will hold the lower 32 bits while the other will hold the upper 32. So the other counter (or syses corresponding to it) also needs to be accessed.

Quoted 4 lines

For 64-bit counter, I suppose the userspace is expected to read the full counter from two sysfs files and combine the value (your documentation doesn't explain this)? That seems non-optimal, why cannot kernel just return the full combined 64-value directly in kernel?

I will add more clear comments for this.
While it is true that the driver could combine the 2 fields and present a 64-bit value via one of the sysfs, the reason for the current approach is that there are other interfaces which expose the same counters for our platform and there are tools that are expected to work on top of both interfaces for the purpose of collecting performance stats. The other interfaces follow this approach of having lower and upper 32-bits separately in each counter, and the tools expect the same. Hence the driver follows this approach to keep things consistent across the BlueField platform.

Quoted 6 lines

Similarly, are these cycle counters occupying the same space as non-cycle counters (so both can/cannot be used that the same time)? I'm asking this because you're adding a parallel interface to read the value and if it's either-or, I don't understand why the value needs to be read from different file depending on the counter counting in cycles or not.

It is the same file. The count_clock sysfs exposes 16 bits, one for each counter, to allow the user to dedicate any of the 16 counters to counting cycles. Once set, the corresponding counter can no longer monitor other events, and the same sysfs can be accessed to read the cycle count. Again, I will try capture this better and more elaborately in the documentation.

Thanks,
Shravan

maintainer
#6 lore raw

Re: [PATCH v2 2/4] platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and cycle count

Quoted 24 lines · Shravan Ramani
Quoted 7 lines
Quoted 3 lines

Both these features are supported by BlueField-3 PMC hardware, hence the required bit-fields are exposed by the driver via sysfs to allow the user to configure as needed.

I'm trying to understand what happens for the other counter, when the use_odd_counter is enabled? This change also doesn't add code that would make the other counter -EBUSY, should that be done?

When 2 32-bit counters are coupled to form a 64-bit counter using this setting, one counter will hold the lower 32 bits while the other will hold the upper 32. So the other counter (or syses corresponding to it) also needs to be accessed.

Quoted 4 lines

For 64-bit counter, I suppose the userspace is expected to read the full counter from two sysfs files and combine the value (your documentation doesn't explain this)? That seems non-optimal, why cannot kernel just return the full combined 64-value directly in kernel?

I will add more clear comments for this.
While it is true that the driver could combine the 2 fields and present a 64-bit value via one of the sysfs, the reason for the current approach is that there are other interfaces which expose the same counters for our platform and there are tools that are expected to work on top of both interfaces for the purpose of collecting performance stats.

Quoted 4 lines

The other interfaces follow this
approach of having lower and upper 32-bits separately in each counter, and the tools expect the same. Hence the driver follows this approach to keep things consistent across the BlueField platform.

Hi,

I went to look through the existing arrays in mlxbf-pmc.c but did not find any entries that would have clearly indicated the counters being hi/lo parts of the same counter. There were a few 0/1 ones which could be the same counter although I suspect even they are not parts of the same counter but two separate entities called 0 and 1 having the same counter.

Could you please elaborate further what you meant with the note about other interfaces above so I can better assess the claim?

Signature · 1 line
 i.
#7 lore raw

Re: [PATCH v2 2/4] platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and cycle count

Quoted 23 lines
Quoted 16 lines

When 2 32-bit counters are coupled to form a 64-bit counter using this setting, one counter will hold the lower 32 bits while the other will hold the upper 32. So the other counter (or syses corresponding to it) also needs to be accessed.

Quoted 4 lines

For 64-bit counter, I suppose the userspace is expected to read the full counter from two sysfs files and combine the value (your documentation doesn't explain this)? That seems non-optimal, why cannot kernel just return the full combined 64-value directly in kernel?

I will add more clear comments for this.
While it is true that the driver could combine the 2 fields and present a 64-bit value via one of the sysfs, the reason for the current approach is that there are other interfaces which expose the same counters for our platform and there are tools that are expected to work on top of both interfaces for the purpose of collecting performance stats.

Quoted 4 lines

The other interfaces follow this
approach of having lower and upper 32-bits separately in each counter, and the tools expect the same. Hence the driver follows this approach to keep things consistent across the BlueField platform.

Hi,

Quoted 8 lines

I went to look through the existing arrays in mlxbf-pmc.c but did not find any entries that would have clearly indicated the counters being hi/lo parts of the same counter. There were a few 0/1 ones which could be the same counter although I suspect even they are not parts of the same counter but two separate entities called 0 and 1 having the same counter.

Could you please elaborate further what you meant with the note about other interfaces above so I can better assess the claim?

When combining 2 counters using the "use_odd_counter" setting, the mechanism of joining them or assigning upper or lower 32 bits to a counter is handled in HW and not by the driver. For example, if bit0 of "use_odd_counter" is set, counter0 and counter1 (which were originally separate counters) automatically become the lower and upper bits of one 64-bit value. The user needs to read both these sysfs separately to get the full 64-bit value. The driver does not do any special handling for such cases, merely provides access to both counter0 and counter1.

Since the events supported by the blocks are quite HW centric and low-level in nature, the driver is generally used alongside various tools which work on top of this driver to collect telemetry info and provide more readable statistics to the end-user. Similar to this driver, there are other FW interfaces providing access to these counters (same and other additional ones as well that belong to other HW blocks). For the sake of consistency and to allow the tools to be compatible with all interfaces, the counter data needs to be accessible in the same way, ie, as 32-bit upper and lower values in counter0 and counter1 sysfs as in the above case.

Thanks,
Shravan

maintainer
#8 lore raw

Re: [PATCH v2 2/4] platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and cycle count

Quoted 32 lines · Shravan Ramani
Quoted 23 lines
Quoted 16 lines

When 2 32-bit counters are coupled to form a 64-bit counter using this setting, one counter will hold the lower 32 bits while the other will hold the upper 32. So the other counter (or syses corresponding to it) also needs to be accessed.

Quoted 4 lines

For 64-bit counter, I suppose the userspace is expected to read the full counter from two sysfs files and combine the value (your documentation doesn't explain this)? That seems non-optimal, why cannot kernel just return the full combined 64-value directly in kernel?

I will add more clear comments for this.
While it is true that the driver could combine the 2 fields and present a 64-bit value via one of the sysfs, the reason for the current approach is that there are other interfaces which expose the same counters for our platform and there are tools that are expected to work on top of both interfaces for the purpose of collecting performance stats.

Quoted 4 lines

The other interfaces follow this
approach of having lower and upper 32-bits separately in each counter, and the tools expect the same. Hence the driver follows this approach to keep things consistent across the BlueField platform.

Hi,

Quoted 8 lines

I went to look through the existing arrays in mlxbf-pmc.c but did not find any entries that would have clearly indicated the counters being hi/lo parts of the same counter. There were a few 0/1 ones which could be the same counter although I suspect even they are not parts of the same counter but two separate entities called 0 and 1 having the same counter.

Could you please elaborate further what you meant with the note about other interfaces above so I can better assess the claim?

Quoted 7 lines

When combining 2 counters using the "use_odd_counter" setting, the mechanism of joining them or assigning upper or lower 32 bits to a counter is handled in HW and not by the driver. For example, if bit0 of "use_odd_counter" is set, counter0 and counter1 (which were originally separate counters) automatically become the lower and upper bits of one 64-bit value. The user needs to read both these sysfs separately to get the full 64-bit value. The driver does not do any special handling for such cases, merely provides access to both counter0 and counter1.

I know all this by now, but we're discussion here is whether kernel should do "special handling". Although, it's not really correct to depict representing 64-bit counter in its entirety as "special handling".

I think the kernel should combine the 64-bit halved and you argumented it shouldn't. When I went to confirm the claim your argument was based on, I couldn't find on what basis the claim was made.

Quoted 8 lines

Since the events supported by the blocks are quite HW centric and low-level in nature, the driver is generally used alongside various tools which work on top of this driver to collect telemetry info and provide more readable statistics to the end-user. Similar to this driver, there are other FW interfaces providing access to these counters (same and other additional ones as well that belong to other HW blocks). For the sake of consistency and to allow the tools to be compatible with all interfaces, the counter data needs to be accessible in the same way, ie, as 32-bit upper and lower values in counter0 and counter1 sysfs as in the above case.

This does nothing to answer my question. Where in the kernel, there's an example where a 64-bit counter for BlueField platform is presented as 2 32-bit counters? If there isn't any examples in the kernel, your statement about consistency within the platform doesn't hold water, quoted (again) here for clarity what I'm refering to:

"The other interfaces follow this approach of having lower and upper 32-bits separately in each counter, and the tools expect the same. Hence the driver follows this approach to keep things consistent across the BlueField platform."

Where I can find those "other interfaces" that already follow this convention?

Signature · 1 line
 i.
#9 lore raw

Re: [PATCH v2 2/4] platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and cycle count

Quoted 7 lines

This does nothing to answer my question. Where in the kernel, there's an example where a 64-bit counter for BlueField platform is presented as 2 32-bit counters? If there isn't any examples in the kernel, your statement about consistency within the platform doesn't hold water, quoted (again) here for clarity what I'm refering to:

Quoted 7 lines

"The other interfaces follow this approach of having lower and upper 32-bits separately in each counter, and the tools expect the same. Hence the driver follows this approach to keep things consistent across the BlueField platform."

Where I can find those "other interfaces" that already follow this convention?

Ah, I think I misunderstood the question and went on elaborating the same thing, apologies. The other interfaces are not part of the kernel. They are part of the BlueField Software Package, which also contains the tools that put together the performance metrics.
My thinking was that since this is a platform driver and is used along with the BlueField Software Package, consistency with the tools which were developed following the same convention could be considered, as long as the driver is not doing something non-standard, of course. I can change the driver handling to present 64-bit data if you insist.

Thanks,
Shravan

maintainer
#10 lore raw

Re: [PATCH v2 2/4] platform/mellanox: mlxbf-pmc: Add support for 64-bit counters and cycle count

Quoted 14 lines · Shravan Ramani
Quoted 6 lines

This does nothing to answer my question. Where in the kernel, there's an example where a 64-bit counter for BlueField platform is presented as 2 32-bit counters? If there isn't any examples in the kernel, your statement about consistency within the platform doesn't hold water, quoted (again) here for clarity what I'm refering to:

Quoted 7 lines

"The other interfaces follow this approach of having lower and upper 32-bits separately in each counter, and the tools expect the same. Hence the driver follows this approach to keep things consistent across the BlueField platform."

Where I can find those "other interfaces" that already follow this convention?

Quoted 9 lines

Ah, I think I misunderstood the question and went on elaborating the same thing, apologies. The other interfaces are not part of the kernel. They are part of the BlueField Software Package, which also contains the tools that put together the performance metrics.
My thinking was that since this is a platform driver and is used along with the BlueField Software Package, consistency with the tools which were developed following the same convention could be considered, as long as the driver is not doing something non-standard, of course. I can change the driver handling to present 64-bit data if you insist.

I'd certainly prefer 64-bit data be presented as such by the kernel. While you make that change, please make sure the driver correctly handles the lower dword wraps without returning an inconsistent reading (assuming the counter parts are read non-atomically, it is a common pitfall).

Signature · 1 line
 i.
#11 lore raw

PATCH v2 3/4 platform/mellanox: mlxbf-pmc: Add support for clock_measure performance block

In reply to Shravan Kumar Ramani

The HW clock_measure counter info is passed to the driver from ACPI. Create a new sub-directory for clock_measure events and provide read access to the user. Writes are blocked since the fields are RO.

---
 drivers/platform/mellanox/mlxbf-pmc.c | 46 ++++++++++++++++++++++++---
 1 file changed, 42 insertions(+), 4 deletions(-)
drivers/platform/mellanox/mlxbf-pmc.c +42 -4
index 635ecc3b3845..1212a96fb3eb 100644
@@ -865,6 +865,37 @@ static const struct mlxbf_pmc_events mlxbf_pmc_llt_miss_events[] = {
 	{75, "HISTOGRAM_HISTOGRAM_BIN9"},
 };
 
+static const struct mlxbf_pmc_events mlxbf_pmc_clock_events[] = {
+	{ 0x0, "FMON_CLK_LAST_COUNT_PLL_D1_INST0" },
+	{ 0x4, "REFERENCE_WINDOW_WIDTH_PLL_D1_INST0" },
+	{ 0x8, "FMON_CLK_LAST_COUNT_PLL_D1_INST1" },
+	{ 0xc, "REFERENCE_WINDOW_WIDTH_PLL_D1_INST1" },
+	{ 0x10, "FMON_CLK_LAST_COUNT_PLL_G1" },
+	{ 0x14, "REFERENCE_WINDOW_WIDTH_PLL_G1" },
+	{ 0x18, "FMON_CLK_LAST_COUNT_PLL_W1" },
+	{ 0x1c, "REFERENCE_WINDOW_WIDTH_PLL_W1" },
+	{ 0x20, "FMON_CLK_LAST_COUNT_PLL_T1" },
+	{ 0x24, "REFERENCE_WINDOW_WIDTH_PLL_T1" },
+	{ 0x28, "FMON_CLK_LAST_COUNT_PLL_A0" },
+	{ 0x2c, "REFERENCE_WINDOW_WIDTH_PLL_A0" },
+	{ 0x30, "FMON_CLK_LAST_COUNT_PLL_C0" },
+	{ 0x34, "REFERENCE_WINDOW_WIDTH_PLL_C0" },
+	{ 0x38, "FMON_CLK_LAST_COUNT_PLL_N1" },
+	{ 0x3c, "REFERENCE_WINDOW_WIDTH_PLL_N1" },
+	{ 0x40, "FMON_CLK_LAST_COUNT_PLL_I1" },
+	{ 0x44, "REFERENCE_WINDOW_WIDTH_PLL_I1" },
+	{ 0x48, "FMON_CLK_LAST_COUNT_PLL_R1" },
+	{ 0x4c, "REFERENCE_WINDOW_WIDTH_PLL_R1" },
+	{ 0x50, "FMON_CLK_LAST_COUNT_PLL_P1" },
+	{ 0x54, "REFERENCE_WINDOW_WIDTH_PLL_P1" },
+	{ 0x58, "FMON_CLK_LAST_COUNT_REF_100_INST0" },
+	{ 0x5c, "REFERENCE_WINDOW_WIDTH_REF_100_INST0" },
+	{ 0x60, "FMON_CLK_LAST_COUNT_REF_100_INST1" },
+	{ 0x64, "REFERENCE_WINDOW_WIDTH_REF_100_INST1" },
+	{ 0x68, "FMON_CLK_LAST_COUNT_REF_156" },
+	{ 0x6c, "REFERENCE_WINDOW_WIDTH_REF_156" },
+};
+
 static struct mlxbf_pmc_context *pmc;
 
 /* UUID used to probe ATF service. */
@@ -1038,6 +1069,9 @@ static const struct mlxbf_pmc_events *mlxbf_pmc_event_list(const char *blk, size
 	} else if (strstr(blk, "llt")) {
 		events = mlxbf_pmc_llt_events;
 		size = ARRAY_SIZE(mlxbf_pmc_llt_events);
+	} else if (strstr(blk, "clock_measure")) {
+		events = mlxbf_pmc_clock_events;
+		size = ARRAY_SIZE(mlxbf_pmc_clock_events);
 	} else {
 		events = NULL;
 		size = 0;
@@ -1472,14 +1506,15 @@ static int mlxbf_pmc_read_event(unsigned int blk_num, u32 cnt_num, bool is_l3, u
 /* Method to read a register */
 static int mlxbf_pmc_read_reg(unsigned int blk_num, u32 offset, u64 *result)
 {
-	u32 ecc_out;
+	u32 reg;
 
-	if (strstr(pmc->block_name[blk_num], "ecc")) {
+	if ((strstr(pmc->block_name[blk_num], "ecc")) ||
+	    (strstr(pmc->block_name[blk_num], "clock_measure"))) {
 		if (mlxbf_pmc_readl(pmc->block[blk_num].mmio_base + offset,
-				    &ecc_out))
+				    &reg))
 			return -EFAULT;
 
-		*result = ecc_out;
+		*result = reg;
 		return 0;
 	}
 
@@ -1493,6 +1528,9 @@ static int mlxbf_pmc_read_reg(unsigned int blk_num, u32 offset, u64 *result)
 /* Method to write to a register */
 static int mlxbf_pmc_write_reg(unsigned int blk_num, u32 offset, u64 data)
 {
+	if (strstr(pmc->block_name[blk_num], "clock_measure"))
+		return -EINVAL;
+
 	if (strstr(pmc->block_name[blk_num], "ecc")) {
 		return mlxbf_pmc_write(pmc->block[blk_num].mmio_base + offset,
 				       MLXBF_PMC_WRITE_REG_32, data);
Signature · 1 line
2.30.1
#12 lore raw

PATCH v2 4/4 Documentation/ABI: Add new sysfs fields to sysfs-platform-mellanox-pmc

In reply to Shravan Kumar Ramani

Document newly added "count_clock" and "use_odd_counter" sysfs entries for the Mellanox BlueField PMC driver.

---
 .../ABI/testing/sysfs-platform-mellanox-pmc      | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
Documentation/ABI/testing/sysfs-platform-mellanox-pmc +16 -0
index 47094024dbeb..b9973ebec2fd 100644
@@ -47,3 +47,19 @@ Description:
 		Value of register. This is used to read or reset the registers
 		where various performance statistics are counted for each block.
 
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/count_clock
+Date:		May 2024
+KernelVersion:	6.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Use counter for counting cycles. This is used to program any of
+		the counters in the block to count cycles.
+
+What:		/sys/bus/platform/devices/<HID>/hwmon/hwmonX/<block>/use_odd_counter
+Date:		May 2024
+KernelVersion:	6.10
+Contact:	"Shravan Kumar Ramani <[email protected]>"
+Description:
+		Form 64-bit counter using 2 32-bit counters. This is used to combine
+		2 adjacent counters to form a single 64-bit counter.
+
Signature · 1 line
2.30.1