Skip to contentSkip to conversation

OpenSBI PMU improvements

PATCH New Anup Patel 3 participants 11 messages Aug 25, 2022 – Sep 1, 2022 1 message not in the archive

lore
Share & tools
lore

Anup Patel

author
#0 lore raw

cover OpenSBI PMU improvements

This series aims to improve SBI PMU implementation in following ways:
1) Optimize memory usage by reducing global data to track per-HART firmware counters (almost 80KB saved)
2) Allow platform to implement custom firmware counters and quirks related to hardware counters (This will be useful for T-Head)
3) Few other fixes for better SBI spec compliance

These patches can also be found in sbi_pmu_imp_v1 branch at:
https://github.com/avpatel/opensbi.git

Anup Patel (7):
lib: sbi_pmu: Remove "event_idx" member from struct sbi_pmu_fw_event
lib: sbi_pmu: Replace sbi_pmu_ctr_read() with sbi_pmu_ctr_fw_read()
lib: sbi_pmu: Firmware counters are always 64 bits wide
lib: sbi_pmu: Simplify FW counters to reduce memory usage
lib: sbi_pmu: Add custom PMU device operations
lib: sbi: Print platform PMU device at boot-time
include: sbi: Reduce includes in sbi_pmu.h

 include/sbi/sbi_pmu.h   |  67 ++++++++++--
 lib/sbi/sbi_ecall_pmu.c |   3 +-
 lib/sbi/sbi_init.c      |   4 +
 lib/sbi/sbi_pmu.c       | 218 +++++++++++++++++++++++-----------------
 lib/sbi/sbi_trap.c      |   1 +
 5 files changed, 190 insertions(+), 103 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/7 lib: sbi_pmu: Remove "event_idx" member from struct sbi_pmu_fw_event

The "event_idx" member of struct sbi_pmu_fw_event is not used anywhere so let us remove it.

---
 lib/sbi/sbi_pmu.c | 3 ---
 1 file changed, 3 deletions(-)
lib/sbi/sbi_pmu.c +0 -3
index 5a294a2..535e5cc 100644
@@ -35,9 +35,6 @@ struct sbi_pmu_hw_event {
 
 /** Representation of a firmware event */
 struct sbi_pmu_fw_event {
-	/* Event associated with the particular counter */
-	uint32_t event_idx;
-
 	/* Current value of the counter */
 	unsigned long curr_count;
 
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/7 lib: sbi_pmu: Replace sbi_pmu_ctr_read() with sbi_pmu_ctr_fw_read()

In reply to Anup Patel

The "read a firmware counter" SBI call should only work for firmware counters so let us replace sbi_pmu_ctr_read() with sbi_pmu_ctr_fw_read() which works only on firmware counters.

---
 include/sbi/sbi_pmu.h   |  2 +-
 lib/sbi/sbi_ecall_pmu.c |  3 ++-
 lib/sbi/sbi_pmu.c       | 45 +++++++----------------------------------
 3 files changed, 10 insertions(+), 40 deletions(-)
include/sbi/sbi_pmu.h +1 -1
index f5e3dbd..34336f7 100644
@@ -53,7 +53,7 @@ int sbi_pmu_add_hw_event_counter_map(u32 eidx_start, u32 eidx_end, u32 cmap);
 
 int sbi_pmu_add_raw_event_counter_map(uint64_t select, uint64_t select_mask, u32 cmap);
 
-int sbi_pmu_ctr_read(uint32_t cidx, unsigned long *cval);
+int sbi_pmu_ctr_fw_read(uint32_t cidx, uint64_t *cval);
 
 int sbi_pmu_ctr_stop(unsigned long cidx_base, unsigned long cidx_mask,
 		     unsigned long flag);
lib/sbi/sbi_ecall_pmu.c +2 -1
index 9ee9e81..826c8a8 100644
@@ -51,7 +51,8 @@ static int sbi_ecall_pmu_handler(unsigned long extid, unsigned long funcid,
 
 		break;
 	case SBI_EXT_PMU_COUNTER_FW_READ:
-		ret = sbi_pmu_ctr_read(regs->a0, out_val);
+		ret = sbi_pmu_ctr_fw_read(regs->a0, &temp);
+		*out_val = temp;
 		break;
 	case SBI_EXT_PMU_COUNTER_START:
 
lib/sbi/sbi_pmu.c +7 -38

57 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#3 lore raw

PATCH 3/7 lib: sbi_pmu: Firmware counters are always 64 bits wide

In reply to Anup Patel

As-per SBI specification, all firmware counters are always 64 bits wide so let us update the SBI PMU implementation to reflect this fact.

---
 lib/sbi/sbi_pmu.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
lib/sbi/sbi_pmu.c +3 -3
index 250808e..1c01d34 100644
@@ -36,7 +36,7 @@ struct sbi_pmu_hw_event {
 /** Representation of a firmware event */
 struct sbi_pmu_fw_event {
 	/* Current value of the counter */
-	unsigned long curr_count;
+	uint64_t curr_count;
 
 	/* A flag indicating pmu event monitoring is started */
 	bool bStarted;
@@ -719,8 +719,8 @@ int sbi_pmu_ctr_get_info(uint32_t cidx, unsigned long *ctr_info)
 	} else {
 		/* it's a firmware counter */
 		cinfo.type = SBI_PMU_CTR_TYPE_FW;
-		/* Firmware counters are XLEN bits wide */
-		cinfo.width = BITS_PER_LONG - 1;
+		/* Firmware counters are always 64 bits wide */
+		cinfo.width = 63;
 	}
 
 	*ctr_info = cinfo.value;
Signature · 1 line
2.34.1
author
#4 lore raw

PATCH 4/7 lib: sbi_pmu: Simplify FW counters to reduce memory usage

In reply to Anup Patel

Currently, we have 32 elements (i.e. SBI_PMU_FW_EVENT_MAX) array of "struct sbi_pmu_fw_event" for each of 128 possible HARTs (i.e. SBI_HARTMASK_MAX_BITS).

To reduce memory usage of OpenSBI, we update FW counter implementation as follows:
1) Remove SBI_PMU_FW_EVENT_MAX
2) Remove "struct sbi_pmu_fw_event"
3) Create per-HART bitmap of XLEN bits to track FW counters which are started on each HART
4) Create per-HART uint64_t array to track values of FW counters on each HART.

---
 include/sbi/sbi_pmu.h |  3 --
 lib/sbi/sbi_pmu.c     | 78 ++++++++++++++++++++-----------------------
 2 files changed, 36 insertions(+), 45 deletions(-)
include/sbi/sbi_pmu.h +0 -3
index 34336f7..39cf007 100644
@@ -19,9 +19,6 @@
 /* Maximum number of hardware events that can mapped by OpenSBI */
 #define SBI_PMU_HW_EVENT_MAX 256
 
-/* Maximum number of firmware events that can mapped by OpenSBI */
-#define SBI_PMU_FW_EVENT_MAX 32
-
 /* Counter related macros */
 #define SBI_PMU_FW_CTR_MAX 16
 #define SBI_PMU_HW_CTR_MAX 32
lib/sbi/sbi_pmu.c +36 -42

161 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#5 lore raw

PATCH 5/7 lib: sbi_pmu: Add custom PMU device operations

In reply to Anup Patel

We extend SBI PMU implementation to allow custom PMU device operations which a platform can use for platform specific quirks.

The custom PMU device operations added by this patch include:
1) Operations to allow a platform implement custom firmware events. These custom firmware events can be SBI vendor extension related events or platform specific per-HART events are not possible to count through HPM CSRs.
2) Operations to allow a platform implement custom way for enabling (or disabling) an overflow interrupt (e.g. T-Head C9xx).

---
 include/sbi/sbi_pmu.h |  57 ++++++++++++++++++++++
 lib/sbi/sbi_pmu.c     | 108 ++++++++++++++++++++++++++++++++++--------
 2 files changed, 144 insertions(+), 21 deletions(-)
include/sbi/sbi_pmu.h +57 -0
index 39cf007..d787575 100644
@@ -25,6 +25,63 @@
 #define SBI_PMU_CTR_MAX	   (SBI_PMU_HW_CTR_MAX + SBI_PMU_FW_CTR_MAX)
 #define SBI_PMU_FIXED_CTR_MASK 0x07
 
+struct sbi_pmu_device {
+	/** Name of the PMU platform device */
+	char name[32];
+
+	/**
+	 * Validate event code of custom firmware event
+	 * Note: SBI_PMU_FW_MAX <= event_idx_code
+	 */
+	int (*fw_event_validate_code)(uint32_t event_idx_code);
+
+	/**
+	 * Match custom firmware counter with custom firmware event
+	 * Note: 0 <= counter_index < SBI_PMU_FW_CTR_MAX
+	 */
+	bool (*fw_counter_match_code)(uint32_t counter_index,
+				      uint32_t event_idx_code);
+
+	/**
+	 * Read value of custom firmware counter
+	 * Note: 0 <= counter_index < SBI_PMU_FW_CTR_MAX
+	 */
+	uint64_t (*fw_counter_read_value)(uint32_t counter_index);
+
+	/**
+	 * Start custom firmware counter
+	 * Note: SBI_PMU_FW_MAX <= event_idx_code
+	 * Note: 0 <= counter_index < SBI_PMU_FW_CTR_MAX
+	 */
+	int (*fw_counter_start)(uint32_t counter_index,
+				uint32_t event_idx_code,
+				uint64_t init_val, bool init_val_update);
+
+	/**
+	 * Stop custom firmware counter
+	 * Note: 0 <= counter_index < SBI_PMU_FW_CTR_MAX
+	 */
+	int (*fw_counter_stop)(uint32_t counter_index);
+
+	/**
+	 * Custom enable irq for hardware counter
+	 * Note: 0 <= counter_index < SBI_PMU_HW_CTR_MAX
+	 */
+	void (*hw_counter_enable_irq)(uint32_t counter_index);
+
+	/**
+	 * Custom disable irq for hardware counter
+	 * Note: 0 <= counter_index < SBI_PMU_HW_CTR_MAX
+	 */
+	void (*hw_counter_disable_irq)(uint32_t counter_index);
+};
+
+/** Get the PMU platform device */
+const struct sbi_pmu_device *sbi_pmu_get_device(void);
+
+/** Set the PMU platform device */
+void sbi_pmu_set_device(const struct sbi_pmu_device *dev);
+
 /** Initialize PMU */
 int sbi_pmu_init(struct sbi_scratch *scratch, bool cold_boot);
 
lib/sbi/sbi_pmu.c +87 -21

206 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#7 lore raw

PATCH 6/7 lib: sbi: Print platform PMU device at boot-time

In reply to Anup Patel

Let us print the platform PMU device name at the boot-time so that users know whether the underlying platform has custom per-HART PMU operations.

---
 lib/sbi/sbi_init.c | 4 ++++
 1 file changed, 4 insertions(+)
lib/sbi/sbi_init.c +4 -0
index d57efa7..a8500e5 100644
@@ -63,6 +63,7 @@ static void sbi_boot_print_banner(struct sbi_scratch *scratch)
 static void sbi_boot_print_general(struct sbi_scratch *scratch)
 {
 	char str[128];
+	const struct sbi_pmu_device *pdev;
 	const struct sbi_hsm_device *hdev;
 	const struct sbi_ipi_device *idev;
 	const struct sbi_timer_device *tdev;
@@ -93,6 +94,9 @@ static void sbi_boot_print_general(struct sbi_scratch *scratch)
 	hdev = sbi_hsm_get_device();
 	sbi_printf("Platform HSM Device       : %s\n",
 		   (hdev) ? hdev->name : "---");
+	pdev = sbi_pmu_get_device();
+	sbi_printf("Platform PMU Device       : %s\n",
+		   (pdev) ? pdev->name : "---");
 	srdev = sbi_system_reset_get_device(SBI_SRST_RESET_TYPE_COLD_REBOOT, 0);
 	sbi_printf("Platform Reboot Device    : %s\n",
 		   (srdev) ? srdev->name : "---");
Signature · 1 line
2.34.1
author
#10 lore raw

PATCH 7/7 include: sbi: Reduce includes in sbi_pmu.h

In reply to Anup Patel

The sbi_pmu.h should only include minimal required headers whereas sbi_pmu.c should include all required headers.

---
 include/sbi/sbi_pmu.h | 5 ++---
 lib/sbi/sbi_pmu.c     | 2 ++
 lib/sbi/sbi_trap.c    | 1 +
 3 files changed, 5 insertions(+), 3 deletions(-)
include/sbi/sbi_pmu.h +2 -3
index d787575..a2ce42d 100644
@@ -11,9 +11,8 @@
 #define __SBI_PMU_H__
 
 #include <sbi/sbi_types.h>
-#include <sbi/sbi_hartmask.h>
-#include <sbi/sbi_scratch.h>
-#include <sbi/sbi_ecall_interface.h>
+
+struct sbi_scratch;
 
 /* Event related macros */
 /* Maximum number of hardware events that can mapped by OpenSBI */
lib/sbi/sbi_pmu.c +2 -0
index 87e1ff8..214d5e8 100644
@@ -10,7 +10,9 @@
 #include <sbi/riscv_asm.h>
 #include <sbi/sbi_bitops.h>
 #include <sbi/sbi_console.h>
+#include <sbi/sbi_ecall_interface.h>
 #include <sbi/sbi_hart.h>
+#include <sbi/sbi_hartmask.h>
 #include <sbi/sbi_platform.h>
 #include <sbi/sbi_pmu.h>
 #include <sbi/sbi_scratch.h>
lib/sbi/sbi_trap.c +1 -0
index 0e30946..4c1339e 100644
@@ -9,6 +9,7 @@
 
 #include <sbi/riscv_asm.h>
 #include <sbi/riscv_encoding.h>
+#include <sbi/sbi_bitops.h>
 #include <sbi/sbi_console.h>
 #include <sbi/sbi_ecall.h>
 #include <sbi/sbi_error.h>
Signature · 1 line
2.34.1
#6 lore raw

PATCH 5/7 lib: sbi_pmu: Add custom PMU device operations

In reply to Anup Patel
Quoted 11 lines · Anup Patel

We extend SBI PMU implementation to allow custom PMU device operations which a platform can use for platform specific quirks.

The custom PMU device operations added by this patch include:
1) Operations to allow a platform implement custom firmware events. These custom firmware events can be SBI vendor extension related events or platform specific per-HART events are not possible to count through HPM CSRs.
2) Operations to allow a platform implement custom way for enabling (or disabling) an overflow interrupt (e.g. T-Head C9xx).

Quoted 6 lines
---
 include/sbi/sbi_pmu.h |  57 ++++++++++++++++++++++
 lib/sbi/sbi_pmu.c     | 108 ++++++++++++++++++++++++++++++++++--------
 2 files changed, 144 insertions(+), 21 deletions(-)
#9 lore raw

PATCH 6/7 lib: sbi: Print platform PMU device at boot-time

In reply to a message not in the archive
Quoted 29 lines · Atish Patra
Quoted 25 lines · Anup Patel

Let us print the platform PMU device name at the boot-time so that users know whether the underlying platform has custom per-HART PMU operations.

---
 lib/sbi/sbi_init.c | 4 ++++
 1 file changed, 4 insertions(+)
lib/sbi/sbi_init.c +1 -0
index d57efa7..a8500e5 100644
@@ -63,6 +63,7 @@ static void sbi_boot_print_banner(struct sbi_scratch *scratch)
 static void sbi_boot_print_general(struct sbi_scratch *scratch)
 {
        char str[128];
+       const struct sbi_pmu_device *pdev;
        const struct sbi_hsm_device *hdev;
        const struct sbi_ipi_device *idev;
        const struct sbi_timer_device *tdev;
@@ -93,6 +94,9 @@ static void sbi_boot_print_general(struct sbi_scratch *scratch)
        hdev = sbi_hsm_get_device();
        sbi_printf("Platform HSM Device       : %s\n",
Quoted 8 lines

lib/sbi/sbi_init.c static void sbi_boot_print_general(struct sbi_scratch *scratch) · line 99

Quoted 4 lines
lib/sbi/sbi_init.c +3 -0
                   (hdev) ? hdev->name : "---");
+       pdev = sbi_pmu_get_device();
+       sbi_printf("Platform PMU Device       : %s\n",
+                  (pdev) ? pdev->name : "---");

Just a nit pick thought ;)
Should we only print the entire line if pdev is present ?

In general, I agree with your suggestion but for consistency let us keep it as-is. You can send a separate patch print device for various things only if xdev != NULL.

Quoted 9 lines
Quoted 8 lines
lib/sbi/sbi_init.c +0 -1
        srdev = sbi_system_reset_get_device(SBI_SRST_RESET_TYPE_COLD_REBOOT, 0);
        sbi_printf("Platform Reboot Device    : %s\n",
                   (srdev) ? srdev->name : "---");
--

2.34.1

Quoted 7 lines

Otherwise, LGTM.

--
Regards,
Atish

Regards,
Anup

Applied
#11 lore raw

cover OpenSBI PMU improvements

In reply to Anup Patel
Quoted 11 lines · Anup Patel

This series aims to improve SBI PMU implementation in following ways:
1) Optimize memory usage by reducing global data to track per-HART firmware counters (almost 80KB saved)
2) Allow platform to implement custom firmware counters and quirks related to hardware counters (This will be useful for T-Head)
3) Few other fixes for better SBI spec compliance

These patches can also be found in sbi_pmu_imp_v1 branch at:
https://github.com/avpatel/opensbi.git

Quoted 8 lines

Anup Patel (7):
lib: sbi_pmu: Remove "event_idx" member from struct sbi_pmu_fw_event
lib: sbi_pmu: Replace sbi_pmu_ctr_read() with sbi_pmu_ctr_fw_read()
lib: sbi_pmu: Firmware counters are always 64 bits wide
lib: sbi_pmu: Simplify FW counters to reduce memory usage
lib: sbi_pmu: Add custom PMU device operations
lib: sbi: Print platform PMU device at boot-time
include: sbi: Reduce includes in sbi_pmu.h

Applied this series to the riscv/opensbi repo.

Andrew (Drew) had already reviewed most of the patches before it landed on OpenSBI mailing list so I have added his Reviewed-by.

Thanks,
Anup

Quoted 11 lines
 include/sbi/sbi_pmu.h   |  67 ++++++++++--
 lib/sbi/sbi_ecall_pmu.c |   3 +-
 lib/sbi/sbi_init.c      |   4 +
 lib/sbi/sbi_pmu.c       | 218 +++++++++++++++++++++++-----------------
 lib/sbi/sbi_trap.c      |   1 +
 5 files changed, 190 insertions(+), 103 deletions(-)

--
2.34.1