Skip to contentSkip to conversation

Miscellaneous PMU fixes

PATCH Superseded Atish Patra 4 participants 17 messages Jan 7, 2022 – Feb 4, 2022 RISC-V

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover Miscellaneous PMU fixes

This series contain several small fixes in PMU related code. The first patches is a sanity check for counter mappings in DT while rest of the patches updates mhpmevent correctly during init and counter config matching.

Atish Patra (3):
lib: sbi: Map only the counters enabled in hardware
lib: sbi: Disable interrupt during config matching
lib: sbi: Disable interrupt and inhibit counting in M-mode during init

include/sbi/riscv_encoding.h |  3 ++-
lib/sbi/sbi_hart.c           | 18 ++++++++++++++++++
lib/sbi/sbi_pmu.c            | 16 ++++++++++++----
3 files changed, 32 insertions(+), 5 deletions(-)

--
2.30.2

#1 lore raw

PATCH 1/3 lib: sbi: Map only the counters enabled in hardware

The counter mapping in DT may be incorrect if all the counters specified in the mapping are actually not physically present in the hardware. OpenSBI should only keep a mapping of counters enabled in hardware and defined in DT. This assume that all the programmable hpmcounters are consecutive as it doesn't make sense to build a system with sparse hpmcounters.

---
 lib/sbi/sbi_pmu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
lib/sbi/sbi_pmu.c +5 -1
index 5b845f835e79..376afec13491 100644
@@ -181,6 +181,9 @@ static int pmu_add_hw_event_map(u32 eidx_start, u32 eidx_end, u32 cmap,
 	int i = 0;
 	bool is_overlap;
 	struct sbi_pmu_hw_event *event = &hw_event_map[num_hw_events];
+	struct sbi_scratch *scratch = sbi_scratch_thishart_ptr();
+	int hw_ctr_avail = sbi_hart_mhpm_count(scratch);
+	uint32_t ctr_avail_mask = ((uint32_t)(~0) >> (32 - (hw_ctr_avail + 3)));
 
 	/* The first two counters are reserved by priv spec */
 	if (eidx_start > SBI_PMU_HW_INSTRUCTIONS && (cmap & SBI_PMU_FIXED_CTR_MASK))
@@ -208,7 +211,8 @@ static int pmu_add_hw_event_map(u32 eidx_start, u32 eidx_end, u32 cmap,
 	}
 
 	event->select_mask = select_mask;
-	event->counters = cmap;
+	/* Map the only the counters that are available in the hardware */
+	event->counters = cmap & ctr_avail_mask;
 	event->select = select;
 	num_hw_events++;
 
Signature · 1 line
2.30.2
#2 lore raw

PATCH 1/3 lib: sbi: Map only the counters enabled in hardware

Quoted 9 lines · Atish Patra

The counter mapping in DT may be incorrect if all the counters specified in the mapping are actually not physically present in the hardware. OpenSBI should only keep a mapping of counters enabled in hardware and defined in DT. This assume that all the programmable hpmcounters are consecutive as it doesn't make sense to build a system with sparse hpmcounters.

Looks good to me.

Regards,
Anup

Quoted 31 lines
---
 lib/sbi/sbi_pmu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
lib/sbi/sbi_pmu.c +5 -2
index 5b845f835e79..376afec13491 100644
@@ -181,6 +181,9 @@ static int pmu_add_hw_event_map(u32 eidx_start, u32 eidx_end, u32 cmap,
        int i = 0;
        bool is_overlap;
        struct sbi_pmu_hw_event *event = &hw_event_map[num_hw_events];
+       struct sbi_scratch *scratch = sbi_scratch_thishart_ptr();
+       int hw_ctr_avail = sbi_hart_mhpm_count(scratch);
+       uint32_t ctr_avail_mask = ((uint32_t)(~0) >> (32 - (hw_ctr_avail + 3)));
 
        /* The first two counters are reserved by priv spec */
        if (eidx_start > SBI_PMU_HW_INSTRUCTIONS && (cmap & SBI_PMU_FIXED_CTR_MASK))
@@ -208,7 +211,8 @@ static int pmu_add_hw_event_map(u32 eidx_start, u32 eidx_end, u32 cmap,
        }
 
        event->select_mask = select_mask;
-       event->counters = cmap;
+       /* Map the only the counters that are available in the hardware */
+       event->counters = cmap & ctr_avail_mask;
        event->select = select;
        num_hw_events++;
 
--

2.30.2

Applied
#3 lore raw

PATCH 1/3 lib: sbi: Map only the counters enabled in hardware

Quoted 10 lines · Anup Patel
Quoted 8 lines · Atish Patra

The counter mapping in DT may be incorrect if all the counters specified in the mapping are actually not physically present in the hardware. OpenSBI should only keep a mapping of counters enabled in hardware and defined in DT. This assume that all the programmable hpmcounters are consecutive as it doesn't make sense to build a system with sparse hpmcounters.

Quoted 5 lines
Quoted 1 line

Looks good to me.

Applied this patch to the riscv/openbis repo.

Regards,
Anup

Quoted 35 lines

Regards,
Anup

Quoted 31 lines
---
 lib/sbi/sbi_pmu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
lib/sbi/sbi_pmu.c +5 -2
index 5b845f835e79..376afec13491 100644
@@ -181,6 +181,9 @@ static int pmu_add_hw_event_map(u32 eidx_start, u32 eidx_end, u32 cmap,
        int i = 0;
        bool is_overlap;
        struct sbi_pmu_hw_event *event = &hw_event_map[num_hw_events];
+       struct sbi_scratch *scratch = sbi_scratch_thishart_ptr();
+       int hw_ctr_avail = sbi_hart_mhpm_count(scratch);
+       uint32_t ctr_avail_mask = ((uint32_t)(~0) >> (32 - (hw_ctr_avail + 3)));
 
        /* The first two counters are reserved by priv spec */
        if (eidx_start > SBI_PMU_HW_INSTRUCTIONS && (cmap & SBI_PMU_FIXED_CTR_MASK))
@@ -208,7 +211,8 @@ static int pmu_add_hw_event_map(u32 eidx_start, u32 eidx_end, u32 cmap,
        }
 
        event->select_mask = select_mask;
-       event->counters = cmap;
+       /* Map the only the counters that are available in the hardware */
+       event->counters = cmap & ctr_avail_mask;
        event->select = select;
        num_hw_events++;
 
--

2.30.2

#4 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

In reply to Atish Patra

PMU overflow interrupt should be disabled durinig initial configuration of counters. They should be enabled while starting counters.

Fixes 730f01bb41a6 ("lib: sbi: Support sscofpmf extension in OpenSBI")
---
 include/sbi/riscv_encoding.h |  3 ++-
 lib/sbi/sbi_pmu.c            | 10 +++++++---
 2 files changed, 9 insertions(+), 4 deletions(-)
include/sbi/riscv_encoding.h +2 -1
index 8a29f8e05c9a..ac8590d72e4f 100644
@@ -181,13 +181,14 @@
 #define MHPMEVENT_VSINH			(_UL(1) << 59)
 #define MHPMEVENT_VUINH			(_UL(1) << 58)
 #else
-#define MHPMEVENTH_OF			(_UL(1) << 31)
+#define MHPMEVENTH_OF			(_ULL(1) << 31)
 #define MHPMEVENTH_MINH			(_ULL(1) << 30)
 #define MHPMEVENTH_SINH			(_ULL(1) << 29)
 #define MHPMEVENTH_UINH			(_ULL(1) << 28)
 #define MHPMEVENTH_VSINH		(_ULL(1) << 27)
 #define MHPMEVENTH_VUINH		(_ULL(1) << 26)
 
+#define MHPMEVENT_OF			(MHPMEVENTH_OF << 32)
 #define MHPMEVENT_MINH			(MHPMEVENTH_MINH << 32)
 #define MHPMEVENT_SINH			(MHPMEVENTH_SINH << 32)
 #define MHPMEVENT_UINH			(MHPMEVENTH_UINH << 32)
lib/sbi/sbi_pmu.c +7 -3
index 376afec13491..a209c8e266d6 100644
@@ -256,7 +256,7 @@ static int pmu_ctr_enable_irq_hw(int ctr_idx)
 
 #if __riscv_xlen == 32
 	mhpmevent_csr = CSR_MHPMEVENT3H  + ctr_idx - 3;
-	of_mask = ~MHPMEVENTH_OF;
+	of_mask = (uint32_t)~MHPMEVENTH_OF;
 #else
 	mhpmevent_csr = CSR_MHPMEVENT3 + ctr_idx - 3;
 	of_mask = ~MHPMEVENT_OF;
@@ -470,9 +470,13 @@ static int pmu_update_hw_mhpmevent(struct sbi_pmu_hw_event *hw_evt, int ctr_idx,
 	if (!mhpmevent_val || ctr_idx < 3 || ctr_idx >= SBI_PMU_HW_CTR_MAX)
 		return SBI_EFAIL;
 
-	/* Always clear the OVF bit and inhibit countin of events in M-mode */
+	/**
+	 * Always set the OVF bit(disable interrupts) and inhibit counting of
+	 * events in M-mode. The OVF bit should be enabled during the start call.
+	 */
 	if (sbi_hart_has_feature(scratch, SBI_HART_HAS_SSCOFPMF))
-		mhpmevent_val = (mhpmevent_val & ~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+		mhpmevent_val = (mhpmevent_val & ~MHPMEVENT_SSCOF_MASK) |
+				 MHPMEVENT_MINH | MHPMEVENT_OF;
 
 	/* Update the inhibit flags based on inhibit flags received from supervisor */
 	pmu_update_inhibit_flags(flags, &mhpmevent_val);
Signature · 1 line
2.30.2
#5 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

Hello Atish!

Quoted 63 lines · On Fri, 7 Jan 2022 10:54:08 -0800 Atish Patra <[email protected]> wrote:

PMU overflow interrupt should be disabled durinig initial configuration of counters. They should be enabled while starting counters.

Fixes 730f01bb41a6 ("lib: sbi: Support sscofpmf extension in

OpenSBI")

---
 include/sbi/riscv_encoding.h |  3 ++-
 lib/sbi/sbi_pmu.c            | 10 +++++++---
 2 files changed, 9 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/include/sbi/riscv_encoding.h index 8a29f8e05c9a..ac8590d72e4f 100644

include/sbi/riscv_encoding.h +2 -1
@@ -181,13 +181,14 @@
 #define MHPMEVENT_VSINH			(_UL(1) << 59)
 #define MHPMEVENT_VUINH			(_UL(1) << 58)
 #else
-#define MHPMEVENTH_OF			(_UL(1) << 31)
+#define MHPMEVENTH_OF			(_ULL(1) << 31)
 #define MHPMEVENTH_MINH			(_ULL(1) << 30)
 #define MHPMEVENTH_SINH			(_ULL(1) << 29)
 #define MHPMEVENTH_UINH			(_ULL(1) << 28)
 #define MHPMEVENTH_VSINH		(_ULL(1) << 27)
 #define MHPMEVENTH_VUINH		(_ULL(1) << 26)
 
+#define MHPMEVENT_OF			(MHPMEVENTH_OF << 32)
 #define MHPMEVENT_MINH			(MHPMEVENTH_MINH << 32)
 #define MHPMEVENT_SINH			(MHPMEVENTH_SINH << 32)
 #define MHPMEVENT_UINH			(MHPMEVENTH_UINH << 32)
lib/sbi/sbi_pmu.c +1 -1
index 376afec13491..a209c8e266d6 100644
@@ -256,7 +256,7 @@ static int pmu_ctr_enable_irq_hw(int ctr_idx)
 
 #if __riscv_xlen == 32
 	mhpmevent_csr = CSR_MHPMEVENT3H  + ctr_idx - 3;
-	of_mask = ~MHPMEVENTH_OF;
+	of_mask = (uint32_t)~MHPMEVENTH_OF;
 #else
 	mhpmevent_csr = CSR_MHPMEVENT3 + ctr_idx - 3;
 	of_mask = ~MHPMEVENT_OF;
@@ -470,9 +470,13 @@ static int pmu_update_hw_mhpmevent(struct

sbi_pmu_hw_event *hw_evt, int ctr_idx, if (!mhpmevent_val || ctr_idx < 3 || ctr_idx >= SBI_PMU_HW_CTR_MAX) return SBI_EFAIL;

lib/sbi/sbi_pmu.c +0 -1
-	/* Always clear the OVF bit and inhibit countin of events in

M-mode */

lib/sbi/sbi_pmu.c +2 -0
+	/**
+	 * Always set the OVF bit(disable interrupts) and inhibit

counting of

lib/sbi/sbi_pmu.c +1 -0
+	 * events in M-mode. The OVF bit should be enabled during

the start call.

lib/sbi/sbi_pmu.c +1 -1
+	 */
 	if (sbi_hart_has_feature(scratch, SBI_HART_HAS_SSCOFPMF))
-		mhpmevent_val = (mhpmevent_val &

~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;

lib/sbi/sbi_pmu.c +1 -0
+		mhpmevent_val = (mhpmevent_val &

~MHPMEVENT_SSCOF_MASK) |

lib/sbi/sbi_pmu.c static int pmu_update_hw_mhpmevent(struct sbi_pmu_hw_event *hw_evt, int ctr_idx, · line 479

Quoted 1 line
lib/sbi/sbi_pmu.c +1 -0
+				 MHPMEVENT_MINH | MHPMEVENT_OF;

Is there some strong reason to inhibit M-Mode counting here or it is just a workaround until mode filtering support will come up ?

Quoted 3 lines

/* Update the inhibit flags based on inhibit flags received
from supervisor */ pmu_update_inhibit_flags(flags, &mhpmevent_val);

#6 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

Quoted 65 lines · Nikita Shubin

Hello Atish!

Quoted 59 lines · On Fri, 7 Jan 2022 10:54:08 -0800 Atish Patra <[email protected]> wrote:

PMU overflow interrupt should be disabled durinig initial configuration of counters. They should be enabled while starting counters.

Fixes 730f01bb41a6 ("lib: sbi: Support sscofpmf extension in

OpenSBI")

---
 include/sbi/riscv_encoding.h |  3 ++-
 lib/sbi/sbi_pmu.c            | 10 +++++++---
 2 files changed, 9 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/include/sbi/riscv_encoding.h index 8a29f8e05c9a..ac8590d72e4f 100644

include/sbi/riscv_encoding.h +2 -1
@@ -181,13 +181,14 @@
 #define MHPMEVENT_VSINH                      (_UL(1) << 59)
 #define MHPMEVENT_VUINH                      (_UL(1) << 58)
 #else
-#define MHPMEVENTH_OF                        (_UL(1) << 31)
+#define MHPMEVENTH_OF                        (_ULL(1) << 31)
 #define MHPMEVENTH_MINH                      (_ULL(1) << 30)
 #define MHPMEVENTH_SINH                      (_ULL(1) << 29)
 #define MHPMEVENTH_UINH                      (_ULL(1) << 28)
 #define MHPMEVENTH_VSINH             (_ULL(1) << 27)
 #define MHPMEVENTH_VUINH             (_ULL(1) << 26)
 
+#define MHPMEVENT_OF                 (MHPMEVENTH_OF << 32)
 #define MHPMEVENT_MINH                       (MHPMEVENTH_MINH << 32)
 #define MHPMEVENT_SINH                       (MHPMEVENTH_SINH << 32)
 #define MHPMEVENT_UINH                       (MHPMEVENTH_UINH << 32)
lib/sbi/sbi_pmu.c +1 -1
index 376afec13491..a209c8e266d6 100644
@@ -256,7 +256,7 @@ static int pmu_ctr_enable_irq_hw(int ctr_idx)
 
 #if __riscv_xlen == 32
      mhpmevent_csr = CSR_MHPMEVENT3H  + ctr_idx - 3;
-     of_mask = ~MHPMEVENTH_OF;
+     of_mask = (uint32_t)~MHPMEVENTH_OF;
 #else
      mhpmevent_csr = CSR_MHPMEVENT3 + ctr_idx - 3;
      of_mask = ~MHPMEVENT_OF;
@@ -470,9 +470,13 @@ static int pmu_update_hw_mhpmevent(struct

sbi_pmu_hw_event *hw_evt, int ctr_idx, if (!mhpmevent_val || ctr_idx < 3 || ctr_idx >= SBI_PMU_HW_CTR_MAX) return SBI_EFAIL;

- /* Always clear the OVF bit and inhibit countin of events in
M-mode */

lib/sbi/sbi_pmu.c +2 -0
+     /**
+      * Always set the OVF bit(disable interrupts) and inhibit

counting of
+ * events in M-mode. The OVF bit should be enabled during
the start call.
+ */
if (sbi_hart_has_feature(scratch, SBI_HART_HAS_SSCOFPMF))

Quoted 9 lines
Quoted 5 lines

- mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+ mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) |
+ MHPMEVENT_MINH | MHPMEVENT_OF;

Is there some strong reason to inhibit M-Mode counting here or it is just a workaround until mode filtering support will come up ?

As a security reason, any perf analysis shouldn't allow inspecting the M-mode for hardware counters.

Privilege mode filtering is already supported in Qemu. You can set exclude user/kernel
from perf command line for Qemu specific events (dTLB-load-misses, dTLB-store-misses, iTLB-load-misses)

Quoted 9 lines
Quoted 3 lines
      /* Update the inhibit flags based on inhibit flags received

from supervisor */ pmu_update_inhibit_flags(flags, &mhpmevent_val);

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
#7 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

Hello Atish!

Quoted 8 lines · On Sun, 9 Jan 2022 17:19:10 -0800 Atish Patra <[email protected]> wrote:
Quoted 8 lines
Quoted 7 lines
lib/sbi/sbi_pmu.c +1 -1
+      */
      if (sbi_hart_has_feature(scratch, SBI_HART_HAS_SSCOFPMF))
-             mhpmevent_val = (mhpmevent_val &

~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+ mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) |
+ MHPMEVENT_MINH | MHPMEVENT_OF;

Quoted 7 lines
Quoted 3 lines

Is there some strong reason to inhibit M-Mode counting here or it is just a workaround until mode filtering support will come up ?

As a security reason, any perf analysis shouldn't allow inspecting the M-mode for hardware counters.

It looks a bit strange to me just to disable something that we have and it is working.

May be it's worth considering putting control of such a behavior in FW_OPTIONS or a compile option ?

I see strong benefits in being able to profile M-Mode also.

Quoted 5 lines

Privilege mode filtering is already supported in Qemu. You can set exclude user/kernel
from perf command line for Qemu specific events (dTLB-load-misses, dTLB-store-misses, iTLB-load-misses)

I see now - didn't study your v5 series carefully enough.

Quoted 13 lines
Quoted 10 lines
Quoted 4 lines
      /* Update the inhibit flags based on inhibit flags received

from supervisor */ pmu_update_inhibit_flags(flags, &mhpmevent_val);

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

#8 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

Quoted 18 lines · Nikita Shubin

Hello Atish!

Quoted 12 lines · On Sun, 9 Jan 2022 17:19:10 -0800 Atish Patra <[email protected]> wrote:
Quoted 11 lines
Quoted 7 lines
lib/sbi/sbi_pmu.c +1 -1
+      */
      if (sbi_hart_has_feature(scratch, SBI_HART_HAS_SSCOFPMF))
-             mhpmevent_val = (mhpmevent_val &

~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+ mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) |
+ MHPMEVENT_MINH | MHPMEVENT_OF;

Is there some strong reason to inhibit M-Mode counting here or it is just a workaround until mode filtering support will come up ?

Quoted 7 lines
Quoted 3 lines

As a security reason, any perf analysis shouldn't allow inspecting the M-mode for hardware counters.

It looks a bit strange to me just to disable something that we have and it is working.

We have privilege mode filtering working at S/U level because current perf has options for that. However, profiling the highest privilege mode(M-mode in RISC-V) is not an option for current
perf tool.

Quoted 5 lines

May be it's worth considering putting control of such a behavior in FW_OPTIONS or a compile option ?

I see strong benefits in being able to profile M-Mode also.

Yes but with an user aware of doing that. That's why firmware events are defined in the first place.
If we just enable the M-mode profiling by default without user explicitly requesting for it, the profiling data
may be inaccurate as well apart from the security concerns.

I was thinking about enabling M-mode profiling via the following route:

1. Add a firmware option to perf tool. At first iteration, it will be a RISC-V specific option.
2. If the user explicitly requests for firmware (M-mode in this case), MINH bit should be cleared.
Otherwise, it should be enabled.

We should add this feature only after the kernel feature is accepted. I am open to any other suggestions as well.

Quoted 22 lines
Quoted 5 lines

Privilege mode filtering is already supported in Qemu. You can set exclude user/kernel
from perf command line for Qemu specific events (dTLB-load-misses, dTLB-store-misses, iTLB-load-misses)

I see now - didn't study your v5 series carefully enough.

Quoted 13 lines
Quoted 10 lines
Quoted 4 lines
      /* Update the inhibit flags based on inhibit flags received

from supervisor */ pmu_update_inhibit_flags(flags, &mhpmevent_val);

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
#9 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

Quoted 26 lines · On Mon, 10 Jan 2022 12:25:40 -0800 Atish Patra <[email protected]> wrote:
Quoted 24 lines · Nikita Shubin

Hello Atish!

Quoted 17 lines · On Sun, 9 Jan 2022 17:19:10 -0800 Atish Patra <[email protected]> wrote:
Quoted 10 lines
Quoted 9 lines

+ */
if (sbi_hart_has_feature(scratch,
SBI_HART_HAS_SSCOFPMF))
- mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+ mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) |
+ MHPMEVENT_MINH | MHPMEVENT_OF;

Quoted 3 lines

Is there some strong reason to inhibit M-Mode counting here or it is just a workaround until mode filtering support will come up ?

As a security reason, any perf analysis shouldn't allow inspecting the M-mode for hardware counters.

Quoted 8 lines
Quoted 3 lines

It looks a bit strange to me just to disable something that we have and it is working.

We have privilege mode filtering working at S/U level because current perf has options for that. However, profiling the highest privilege mode(M-mode in RISC-V) is not an option for current
perf tool.

I've already done a patch for this with new exclude_machine option, didn't sent it yet cause, it is totally useless without your pmu series merged.

Quoted 4 lines
Quoted 4 lines

May be it's worth considering putting control of such a behavior in FW_OPTIONS or a compile option ?

Quoted 8 lines
Quoted 2 lines

I see strong benefits in being able to profile M-Mode also.

Yes but with an user aware of doing that. That's why firmware events are defined in the first place.
If we just enable the M-mode profiling by default without user explicitly requesting for it, the profiling data
may be inaccurate as well apart from the security concerns.

So inhibiting by default is really some kind of workaround.

Quoted 4 lines

I was thinking about enabling M-mode profiling via the following route:

Quoted 8 lines

1. Add a firmware option to perf tool. At first iteration, it will be a RISC-V specific option.
2. If the user explicitly requests for firmware (M-mode in this case), MINH bit should be cleared.
Otherwise, it should be enabled.

We should add this feature only after the kernel feature is accepted. I am open to any other suggestions as well.

I thought an opposite way first OpenSBI and PMU v5 accepted then perf tools :).

I have something like this already, through i didn't find a good way to make it architecture specific.

Okay - i'll fire an RFC then to collect comments, option is not RISC-V specific in my version, but harmless for machines without M-Mode.

Than we can work a way to make it secure and available.

Quoted 25 lines
Quoted 23 lines
Quoted 6 lines

Privilege mode filtering is already supported in Qemu. You can set exclude user/kernel
from perf command line for Qemu specific events (dTLB-load-misses, dTLB-store-misses, iTLB-load-misses)

I see now - didn't study your v5 series carefully enough.

Quoted 13 lines
Quoted 10 lines
Quoted 4 lines
      /* Update the inhibit flags based on inhibit flags

received from supervisor */ pmu_update_inhibit_flags(flags, &mhpmevent_val);

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

#10 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

Quoted 68 lines · Nikita Shubin
Quoted 26 lines · On Mon, 10 Jan 2022 12:25:40 -0800 Atish Patra <[email protected]> wrote:
Quoted 24 lines · Nikita Shubin

Hello Atish!

Quoted 17 lines · On Sun, 9 Jan 2022 17:19:10 -0800 Atish Patra <[email protected]> wrote:
Quoted 10 lines
Quoted 9 lines

+ */
if (sbi_hart_has_feature(scratch,
SBI_HART_HAS_SSCOFPMF))
- mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+ mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) |
+ MHPMEVENT_MINH | MHPMEVENT_OF;

Quoted 3 lines

Is there some strong reason to inhibit M-Mode counting here or it is just a workaround until mode filtering support will come up ?

As a security reason, any perf analysis shouldn't allow inspecting the M-mode for hardware counters.

Quoted 8 lines
Quoted 3 lines

It looks a bit strange to me just to disable something that we have and it is working.

We have privilege mode filtering working at S/U level because current perf has options for that. However, profiling the highest privilege mode(M-mode in RISC-V) is not an option for current
perf tool.

I've already done a patch for this with new exclude_machine option, didn't sent it yet cause, it is totally useless without your pmu series merged.

Quoted 4 lines
Quoted 4 lines

May be it's worth considering putting control of such a behavior in FW_OPTIONS or a compile option ?

Quoted 8 lines
Quoted 2 lines

I see strong benefits in being able to profile M-Mode also.

Yes but with an user aware of doing that. That's why firmware events are defined in the first place.
If we just enable the M-mode profiling by default without user explicitly requesting for it, the profiling data
may be inaccurate as well apart from the security concerns.

So inhibiting by default is really some kind of workaround.

Quoted 10 lines

I was thinking about enabling M-mode profiling via the following route:

1. Add a firmware option to perf tool. At first iteration, it will be a RISC-V specific option.
2. If the user explicitly requests for firmware (M-mode in this case), MINH bit should be cleared.
Otherwise, it should be enabled.

Quoted 6 lines
Quoted 2 lines

We should add this feature only after the kernel feature is accepted. I am open to any other suggestions as well.

I thought an opposite way first OpenSBI and PMU v5 accepted then perf tools :).

PMU series must be merged before any changes to perf tools can be merged. I am hoping the PMU series can make it in 5.18.

Regarding the OpenSBI changes, they are very small changes (few lines at best). That's why, I would at least like to see a broader agreement on the firmware option in the community.
We can patch OpenSBI after that.

Quoted 3 lines

I have something like this already, through i didn't find a good way to make it architecture specific.

Documenting is enough :).

Quoted 3 lines

Okay - i'll fire an RFC then to collect comments, option is not RISC-V specific in my version, but harmless for machines without M-Mode.

Thanks for the quick patch. I have left some comments.

Quoted 28 lines

Than we can work a way to make it secure and available.

Quoted 25 lines
Quoted 23 lines
Quoted 6 lines

Privilege mode filtering is already supported in Qemu. You can set exclude user/kernel
from perf command line for Qemu specific events (dTLB-load-misses, dTLB-store-misses, iTLB-load-misses)

I see now - didn't study your v5 series carefully enough.

Quoted 13 lines
Quoted 10 lines
Quoted 4 lines
      /* Update the inhibit flags based on inhibit flags

received from supervisor */ pmu_update_inhibit_flags(flags, &mhpmevent_val);

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Signature · 2 lines
Regards,
Atish
#11 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

Quoted 77 lines · On Tue, 11 Jan 2022 11:09:49 -0800 Atish Patra <[email protected]> wrote:
Quoted 68 lines · Nikita Shubin
Quoted 26 lines · On Mon, 10 Jan 2022 12:25:40 -0800 Atish Patra <[email protected]> wrote:
Quoted 24 lines · Nikita Shubin

Hello Atish!

Quoted 17 lines · On Sun, 9 Jan 2022 17:19:10 -0800 Atish Patra <[email protected]> wrote:
Quoted 10 lines
Quoted 9 lines

+ */
if (sbi_hart_has_feature(scratch,
SBI_HART_HAS_SSCOFPMF))
- mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+ mhpmevent_val = (mhpmevent_val &
~MHPMEVENT_SSCOF_MASK) |
+ MHPMEVENT_MINH |
MHPMEVENT_OF;

Quoted 3 lines

Is there some strong reason to inhibit M-Mode counting here or it is just a workaround until mode filtering support will come up ?

As a security reason, any perf analysis shouldn't allow inspecting the M-mode for hardware counters.

Quoted 8 lines
Quoted 3 lines

It looks a bit strange to me just to disable something that we have and it is working.

We have privilege mode filtering working at S/U level because current perf has options for that. However, profiling the highest privilege mode(M-mode in RISC-V) is not an option for current perf tool.

I've already done a patch for this with new exclude_machine option, didn't sent it yet cause, it is totally useless without your pmu series merged.

Quoted 4 lines
Quoted 4 lines

May be it's worth considering putting control of such a behavior in FW_OPTIONS or a compile option ?

Quoted 8 lines
Quoted 2 lines

I see strong benefits in being able to profile M-Mode also.

Yes but with an user aware of doing that. That's why firmware events are defined in the first place.
If we just enable the M-mode profiling by default without user explicitly requesting for it, the profiling data
may be inaccurate as well apart from the security concerns.

So inhibiting by default is really some kind of workaround.

Quoted 10 lines

I was thinking about enabling M-mode profiling via the following route:

1. Add a firmware option to perf tool. At first iteration, it will be a RISC-V specific option.
2. If the user explicitly requests for firmware (M-mode in this case), MINH bit should be cleared.
Otherwise, it should be enabled.

Quoted 6 lines
Quoted 2 lines

We should add this feature only after the kernel feature is accepted. I am open to any other suggestions as well.

I thought an opposite way first OpenSBI and PMU v5 accepted then perf tools :).

Quoted 8 lines

PMU series must be merged before any changes to perf tools can be merged. I am hoping the PMU series can make it in 5.18.

Regarding the OpenSBI changes, they are very small changes (few lines at best). That's why, I would at least like to see a broader agreement on the firmware option in the community.
We can patch OpenSBI after that.

That might be the best way to go, as it seems RISC-V mail-lists are overcrowded and patches are not recieving enough attention.

Quoted 4 lines
Quoted 3 lines

I have something like this already, through i didn't find a good way to make it architecture specific.

Quoted 8 lines

Documenting is enough :).

Quoted 3 lines

Okay - i'll fire an RFC then to collect comments, option is not RISC-V specific in my version, but harmless for machines without M-Mode.

Thanks for the quick patch. I have left some comments.

Thank you for your comments and your HPM patches, i am already using then on our piece of RTL.

Quoted 30 lines
Quoted 28 lines

Than we can work a way to make it secure and available.

Quoted 25 lines
Quoted 23 lines
Quoted 6 lines

Privilege mode filtering is already supported in Qemu. You can set exclude user/kernel
from perf command line for Qemu specific events (dTLB-load-misses, dTLB-store-misses, iTLB-load-misses)

I see now - didn't study your v5 series carefully enough.

Quoted 13 lines
Quoted 10 lines
Quoted 4 lines
      /* Update the inhibit flags based on inhibit flags

received from supervisor */
pmu_update_inhibit_flags(flags, &mhpmevent_val);

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

#12 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

In reply to Atish Patra
Quoted 7 lines · Atish Patra

PMU overflow interrupt should be disabled durinig initial configuration of counters. They should be enabled while starting counters.

Fixes 730f01bb41a6 ("lib: sbi: Support sscofpmf extension in OpenSBI")

Overall, this patch looks good but I agree that for debugging purposes we need an option in FW_OPTIONS which can allow monitoring events in M-mode (i.e. MINH=0). The actual behaviour of the option is a separate topic/patch.

Regards,
Anup

Quoted 57 lines
---
 include/sbi/riscv_encoding.h |  3 ++-
 lib/sbi/sbi_pmu.c            | 10 +++++++---
 2 files changed, 9 insertions(+), 4 deletions(-)
include/sbi/riscv_encoding.h +2 -1
index 8a29f8e05c9a..ac8590d72e4f 100644
@@ -181,13 +181,14 @@
 #define MHPMEVENT_VSINH                        (_UL(1) << 59)
 #define MHPMEVENT_VUINH                        (_UL(1) << 58)
 #else
-#define MHPMEVENTH_OF                  (_UL(1) << 31)
+#define MHPMEVENTH_OF                  (_ULL(1) << 31)
 #define MHPMEVENTH_MINH                        (_ULL(1) << 30)
 #define MHPMEVENTH_SINH                        (_ULL(1) << 29)
 #define MHPMEVENTH_UINH                        (_ULL(1) << 28)
 #define MHPMEVENTH_VSINH               (_ULL(1) << 27)
 #define MHPMEVENTH_VUINH               (_ULL(1) << 26)
 
+#define MHPMEVENT_OF                   (MHPMEVENTH_OF << 32)
 #define MHPMEVENT_MINH                 (MHPMEVENTH_MINH << 32)
 #define MHPMEVENT_SINH                 (MHPMEVENTH_SINH << 32)
 #define MHPMEVENT_UINH                 (MHPMEVENTH_UINH << 32)
lib/sbi/sbi_pmu.c +7 -4
index 376afec13491..a209c8e266d6 100644
@@ -256,7 +256,7 @@ static int pmu_ctr_enable_irq_hw(int ctr_idx)
 
 #if __riscv_xlen == 32
        mhpmevent_csr = CSR_MHPMEVENT3H  + ctr_idx - 3;
-       of_mask = ~MHPMEVENTH_OF;
+       of_mask = (uint32_t)~MHPMEVENTH_OF;
 #else
        mhpmevent_csr = CSR_MHPMEVENT3 + ctr_idx - 3;
        of_mask = ~MHPMEVENT_OF;
@@ -470,9 +470,13 @@ static int pmu_update_hw_mhpmevent(struct sbi_pmu_hw_event *hw_evt, int ctr_idx,
        if (!mhpmevent_val || ctr_idx < 3 || ctr_idx >= SBI_PMU_HW_CTR_MAX)
                return SBI_EFAIL;
 
-       /* Always clear the OVF bit and inhibit countin of events in M-mode */
+       /**
+        * Always set the OVF bit(disable interrupts) and inhibit counting of
+        * events in M-mode. The OVF bit should be enabled during the start call.
+        */
        if (sbi_hart_has_feature(scratch, SBI_HART_HAS_SSCOFPMF))
-               mhpmevent_val = (mhpmevent_val & ~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+               mhpmevent_val = (mhpmevent_val & ~MHPMEVENT_SSCOF_MASK) |
+                                MHPMEVENT_MINH | MHPMEVENT_OF;
 
        /* Update the inhibit flags based on inhibit flags received from supervisor */
        pmu_update_inhibit_flags(flags, &mhpmevent_val);
--

2.30.2

Applied
#13 lore raw

PATCH 2/3 lib: sbi: Disable interrupt during config matching

Quoted 8 lines · Anup Patel
Quoted 6 lines · Atish Patra

PMU overflow interrupt should be disabled durinig initial configuration of counters. They should be enabled while starting counters.

Fixes 730f01bb41a6 ("lib: sbi: Support sscofpmf extension in OpenSBI")
Quoted 8 lines
Quoted 1 line

Overall, this patch looks good but I agree that for debugging purposes we need an option in FW_OPTIONS which can allow monitoring events in M-mode (i.e. MINH=0). The actual behaviour of the option is a separate topic/patch.

Applied this patch to the riscv/openbis repo.

Regards,
Anup

Quoted 61 lines

Regards,
Anup

Quoted 57 lines
---
 include/sbi/riscv_encoding.h |  3 ++-
 lib/sbi/sbi_pmu.c            | 10 +++++++---
 2 files changed, 9 insertions(+), 4 deletions(-)
include/sbi/riscv_encoding.h +2 -1
index 8a29f8e05c9a..ac8590d72e4f 100644
@@ -181,13 +181,14 @@
 #define MHPMEVENT_VSINH                        (_UL(1) << 59)
 #define MHPMEVENT_VUINH                        (_UL(1) << 58)
 #else
-#define MHPMEVENTH_OF                  (_UL(1) << 31)
+#define MHPMEVENTH_OF                  (_ULL(1) << 31)
 #define MHPMEVENTH_MINH                        (_ULL(1) << 30)
 #define MHPMEVENTH_SINH                        (_ULL(1) << 29)
 #define MHPMEVENTH_UINH                        (_ULL(1) << 28)
 #define MHPMEVENTH_VSINH               (_ULL(1) << 27)
 #define MHPMEVENTH_VUINH               (_ULL(1) << 26)
 
+#define MHPMEVENT_OF                   (MHPMEVENTH_OF << 32)
 #define MHPMEVENT_MINH                 (MHPMEVENTH_MINH << 32)
 #define MHPMEVENT_SINH                 (MHPMEVENTH_SINH << 32)
 #define MHPMEVENT_UINH                 (MHPMEVENTH_UINH << 32)
lib/sbi/sbi_pmu.c +7 -4
index 376afec13491..a209c8e266d6 100644
@@ -256,7 +256,7 @@ static int pmu_ctr_enable_irq_hw(int ctr_idx)
 
 #if __riscv_xlen == 32
        mhpmevent_csr = CSR_MHPMEVENT3H  + ctr_idx - 3;
-       of_mask = ~MHPMEVENTH_OF;
+       of_mask = (uint32_t)~MHPMEVENTH_OF;
 #else
        mhpmevent_csr = CSR_MHPMEVENT3 + ctr_idx - 3;
        of_mask = ~MHPMEVENT_OF;
@@ -470,9 +470,13 @@ static int pmu_update_hw_mhpmevent(struct sbi_pmu_hw_event *hw_evt, int ctr_idx,
        if (!mhpmevent_val || ctr_idx < 3 || ctr_idx >= SBI_PMU_HW_CTR_MAX)
                return SBI_EFAIL;
 
-       /* Always clear the OVF bit and inhibit countin of events in M-mode */
+       /**
+        * Always set the OVF bit(disable interrupts) and inhibit counting of
+        * events in M-mode. The OVF bit should be enabled during the start call.
+        */
        if (sbi_hart_has_feature(scratch, SBI_HART_HAS_SSCOFPMF))
-               mhpmevent_val = (mhpmevent_val & ~MHPMEVENT_SSCOF_MASK) | MHPMEVENT_MINH;
+               mhpmevent_val = (mhpmevent_val & ~MHPMEVENT_SSCOF_MASK) |
+                                MHPMEVENT_MINH | MHPMEVENT_OF;
 
        /* Update the inhibit flags based on inhibit flags received from supervisor */
        pmu_update_inhibit_flags(flags, &mhpmevent_val);
--

2.30.2

#14 lore raw

PATCH 3/3 lib: sbi: Disable interrupt and inhibit counting in M-mode during init

In reply to Atish Patra

Currently, the mhpmevent CSRs are untouched during hart init during cold/warm boot. Ideally, we should clear out all the bits except overflow and MINH bit. That is required to disable overflow interrupt and inhbit counting in M-mode to avoid any spurious interrupts before perf start.

---
 lib/sbi/sbi_hart.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
lib/sbi/sbi_hart.c +18 -0
index d9a15d994a6b..0a651752cdda 100644
@@ -40,6 +40,9 @@ static unsigned long hart_features_offset;
 static void mstatus_init(struct sbi_scratch *scratch)
 {
 	unsigned long mstatus_val = 0;
+	int cidx;
+	unsigned int num_mhpm = sbi_hart_mhpm_count(scratch);
+	uint64_t mhpmevent_init_val = 0;
 
 	/* Enable FPU */
 	if (misa_extension('D') || misa_extension('F'))
@@ -68,6 +71,21 @@ static void mstatus_init(struct sbi_scratch *scratch)
 	if (sbi_hart_has_feature(scratch, SBI_HART_HAS_MCOUNTINHIBIT))
 		csr_write(CSR_MCOUNTINHIBIT, 0xFFFFFFF8);
 
+	/**
+	 * The mhpmeventn[h] CSR should be initialized with interrupt disabled
+	 * and inhibited running in M-mode during init.
+	 * To keep it simple, only contiguous mhpmcounters are supported as a
+	 * platform with discontiguous mhpmcounters may not make much sense.
+	 */
+	mhpmevent_init_val |= (MHPMEVENT_OF | MHPMEVENT_MINH);
+	for (cidx = 0; cidx < num_mhpm; cidx++) {
+#if __riscv_xlen == 32
+		csr_write_num(CSR_MHPMEVENT3 + cidx, mhpmevent_init_val & 0xFFFFFFFF);
+		csr_write_num(CSR_MHPMEVENT3H + cidx, mhpmevent_init_val >> BITS_PER_LONG);
+#else
+		csr_write_num(CSR_MHPMEVENT3 + cidx, mhpmevent_init_val);
+#endif
+	}
 	/* Disable all interrupts */
 	csr_write(CSR_MIE, 0);
 
Signature · 1 line
2.30.2
#15 lore raw

PATCH 3/3 lib: sbi: Disable interrupt and inhibit counting in M-mode during init

Quoted 5 lines · Atish Patra

Currently, the mhpmevent CSRs are untouched during hart init during cold/warm boot. Ideally, we should clear out all the bits except overflow and MINH bit. That is required to disable overflow interrupt and inhbit counting in M-mode to avoid any spurious

s/inhbit/inhibit/

Quoted 3 lines

interrupts before perf start.

Apart from a minor comment above, it looks good to me.

Regards,
Anup

Quoted 43 lines
---
 lib/sbi/sbi_hart.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
lib/sbi/sbi_hart.c +18 -1
index d9a15d994a6b..0a651752cdda 100644
@@ -40,6 +40,9 @@ static unsigned long hart_features_offset;
 static void mstatus_init(struct sbi_scratch *scratch)
 {
        unsigned long mstatus_val = 0;
+       int cidx;
+       unsigned int num_mhpm = sbi_hart_mhpm_count(scratch);
+       uint64_t mhpmevent_init_val = 0;
 
        /* Enable FPU */
        if (misa_extension('D') || misa_extension('F'))
@@ -68,6 +71,21 @@ static void mstatus_init(struct sbi_scratch *scratch)
        if (sbi_hart_has_feature(scratch, SBI_HART_HAS_MCOUNTINHIBIT))
                csr_write(CSR_MCOUNTINHIBIT, 0xFFFFFFF8);
 
+       /**
+        * The mhpmeventn[h] CSR should be initialized with interrupt disabled
+        * and inhibited running in M-mode during init.
+        * To keep it simple, only contiguous mhpmcounters are supported as a
+        * platform with discontiguous mhpmcounters may not make much sense.
+        */
+       mhpmevent_init_val |= (MHPMEVENT_OF | MHPMEVENT_MINH);
+       for (cidx = 0; cidx < num_mhpm; cidx++) {
+#if __riscv_xlen == 32
+               csr_write_num(CSR_MHPMEVENT3 + cidx, mhpmevent_init_val & 0xFFFFFFFF);
+               csr_write_num(CSR_MHPMEVENT3H + cidx, mhpmevent_init_val >> BITS_PER_LONG);
+#else
+               csr_write_num(CSR_MHPMEVENT3 + cidx, mhpmevent_init_val);
+#endif
+       }
        /* Disable all interrupts */
        csr_write(CSR_MIE, 0);
 
--

2.30.2

Applied
#16 lore raw

PATCH 3/3 lib: sbi: Disable interrupt and inhibit counting in M-mode during init

Quoted 10 lines · Anup Patel
Quoted 5 lines · Atish Patra

Currently, the mhpmevent CSRs are untouched during hart init during cold/warm boot. Ideally, we should clear out all the bits except overflow and MINH bit. That is required to disable overflow interrupt and inhbit counting in M-mode to avoid any spurious

s/inhbit/inhibit/

Quoted 7 lines
Quoted 3 lines

interrupts before perf start.

Apart from a minor comment above, it looks good to me.

I have taken care of the minor comment while merging this patch.

Applied this patch to the riscv/openbis repo.

Regards,
Anup

Quoted 47 lines

Regards,
Anup

Quoted 43 lines
---
 lib/sbi/sbi_hart.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
lib/sbi/sbi_hart.c +18 -1
index d9a15d994a6b..0a651752cdda 100644
@@ -40,6 +40,9 @@ static unsigned long hart_features_offset;
 static void mstatus_init(struct sbi_scratch *scratch)
 {
        unsigned long mstatus_val = 0;
+       int cidx;
+       unsigned int num_mhpm = sbi_hart_mhpm_count(scratch);
+       uint64_t mhpmevent_init_val = 0;
 
        /* Enable FPU */
        if (misa_extension('D') || misa_extension('F'))
@@ -68,6 +71,21 @@ static void mstatus_init(struct sbi_scratch *scratch)
        if (sbi_hart_has_feature(scratch, SBI_HART_HAS_MCOUNTINHIBIT))
                csr_write(CSR_MCOUNTINHIBIT, 0xFFFFFFF8);
 
+       /**
+        * The mhpmeventn[h] CSR should be initialized with interrupt disabled
+        * and inhibited running in M-mode during init.
+        * To keep it simple, only contiguous mhpmcounters are supported as a
+        * platform with discontiguous mhpmcounters may not make much sense.
+        */
+       mhpmevent_init_val |= (MHPMEVENT_OF | MHPMEVENT_MINH);
+       for (cidx = 0; cidx < num_mhpm; cidx++) {
+#if __riscv_xlen == 32
+               csr_write_num(CSR_MHPMEVENT3 + cidx, mhpmevent_init_val & 0xFFFFFFFF);
+               csr_write_num(CSR_MHPMEVENT3H + cidx, mhpmevent_init_val >> BITS_PER_LONG);
+#else
+               csr_write_num(CSR_MHPMEVENT3 + cidx, mhpmevent_init_val);
+#endif
+       }
        /* Disable all interrupts */
        csr_write(CSR_MIE, 0);
 
--

2.30.2