Skip to contentSkip to conversation

add support for self monitoring

PATCH Superseded Tomasz Duszynski 14 participants 222 messages Nov 11, 2022 – Jun 10, 2025 Page 2 of 4

Subsystems (2)
lore
Share & tools
lore
#77 lore raw

PATCH v8 1/4 lib: add generic support for reading PMU events

In reply to Tomasz Duszynski

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   1 +
 app/test/test_pmu.c                    |  55 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   9 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 464 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 205 +++++++++++
 lib/pmu/version.map                    |  20 ++
 13 files changed, 813 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
index 9a0f416d2e..9f13eafd95 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build +1 -0
index f34d19e3c3..7b6b69dcf1 100644
@@ -111,6 +111,7 @@ test_sources = files(
         'test_reciprocal_division_perf.c',
         'test_red.c',
         'test_pie.c',
+        'test_pmu.c',
         'test_reorder.c',
         'test_rib.c',
         'test_rib6.c',
app/test/test_pmu.c new file +55 -0

55 lines not on this page · raw message

doc/api/doxy-api-index.md +2 -1
index de488c7abf..7f1938f92f 100644
@@ -222,7 +222,8 @@ The public API headers are grouped by topics:
   [log](@ref rte_log.h),
   [errno](@ref rte_errno.h),
   [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),
+  [pmu](@ref rte_pmu.h)
 
 - **misc**:
   [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
index f0886c3bd1..920e615996 100644
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
                           @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +8 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
doc/guides/rel_notes/release_23_03.rst +9 -0
index 73f5d94e14..733541d56c 100644
@@ -55,10 +55,19 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added PMU library.**
+
+  Added a new PMU (performance measurement unit) library which allows applications
+  to perform self monitoring activities without depending on external utilities like perf.
+  After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters
+  can be stored in CTF format for further analysis.
+
 * **Updated AMD axgbe driver.**
 
   * Added multi-process support.
 
+* **Added multi-process support for axgbe PMD.**
+
 * **Updated Corigine nfp driver.**
 
   * Added support for meter options.
lib/meson.build +1 -0
index a90fee31b7..7132131b5c 100644
@@ -11,6 +11,7 @@
 libraries = [
         'kvargs', # eal depends on kvargs
         'telemetry', # basic info querying
+        'pmu',
         'eal', # everything depends on eal
         'ring',
         'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
new file mode 100644
index 0000000000..a4160b494e
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2023 Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +29 -0
new file mode 100644
index 0000000000..849549b125
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +464 -0

464 lines not on this page · raw message

lib/pmu/rte_pmu.h new file +205 -0

205 lines not on this page · raw message

lib/pmu/version.map new file +20 -0
new file mode 100644
index 0000000000..50fb0f354e
@@ -0,0 +1,20 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
+
+INTERNAL {
+	global:
+
+	rte_pmu_enable_group;
+};
Signature · 1 line
2.34.1
#78 lore raw

RE: [PATCH v8 1/4] lib: add generic support for reading PMU events

Quoted 37 lines

-----Original Message-----
From: Tomasz Duszynski <[email protected]>
Sent: Thursday, February 2, 2023 5:44 PM
To: [email protected]; [email protected]; Tomasz Duszynski <[email protected]>
Cc: [email protected]; Ruifeng Wang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v8 1/4] lib: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   1 +
 app/test/test_pmu.c                    |  55 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   9 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 464 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 205 +++++++++++
 lib/pmu/version.map                    |  20 ++
 13 files changed, 813 insertions(+), 1 deletion(-)  create mode 100644

app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.map

<snip>

doc/guides/rel_notes/release_23_03.rst New Features · line 60

Quoted 12 lines
doc/guides/rel_notes/release_23_03.rst +3 -0
index 73f5d94e14..733541d56c 100644
@@ -55,10 +55,19 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added PMU library.**
+
+  Added a new PMU (performance measurement unit) library which allows

Overall looks good to me. Just a minor comment.
Should it be 'performance *monitoring* unit'?
I see the same terminology is used across architectures. It will be better if we align with that.

Thanks.

<snip>

#79 lore raw

PATCH v8 2/4 pmu: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c         |  4 ++
 lib/pmu/meson.build         |  7 +++
 lib/pmu/pmu_arm64.c         | 94 +++++++++++++++++++++++++++++++++++++
 lib/pmu/rte_pmu.h           |  4 ++
 lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++
 5 files changed, 139 insertions(+)
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
app/test/test_pmu.c +4 -0
index a9bfb1a427..623e04b691 100644
@@ -26,6 +26,10 @@ test_pmu_read(void)
 	if (rte_pmu_init() < 0)
 		return TEST_FAILED;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/pmu/meson.build +7 -0
index a4160b494e..e857681137 100644
@@ -11,3 +11,10 @@ includes = [global_inc]
 
 sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
+indirect_headers += files(
+        'rte_pmu_pmc_arm64.h',
+)
+
+if dpdk_conf.has('RTE_ARCH_ARM64')
+    sources += files('pmu_arm64.c')
+endif
lib/pmu/pmu_arm64.c new file +94 -0

94 lines not on this page · raw message

lib/pmu/rte_pmu.h +4 -0
index e360375a0c..b18938dab1 100644
@@ -26,6 +26,10 @@ extern "C" {
 #include <rte_compat.h>
 #include <rte_spinlock.h>
 
+#if defined(RTE_ARCH_ARM64)
+#include "rte_pmu_pmc_arm64.h"
+#endif
+
 /** Maximum number of events in a group */
 #define MAX_NUM_GROUP_EVENTS 8
 
lib/pmu/rte_pmu_pmc_arm64.h new file +30 -0
new file mode 100644
index 0000000000..10648f0c5f
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_ARM64_H_
+#define _RTE_PMU_PMC_ARM64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_ARM64_H_ */
Signature · 1 line
2.34.1
#80 lore raw

PATCH v8 3/4 pmu: support reading Intel x86_64 PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel x86_64 PMU events in runtime.

---
 app/test/test_pmu.c          |  2 ++
 lib/pmu/meson.build          |  1 +
 lib/pmu/rte_pmu.h            |  2 ++
 lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++++++++++++++++++++++++
 4 files changed, 29 insertions(+)
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
app/test/test_pmu.c +2 -0
index 623e04b691..614395482f 100644
@@ -28,6 +28,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/pmu/meson.build +1 -0
index e857681137..5b92e5c4e3 100644
@@ -13,6 +13,7 @@ sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
 indirect_headers += files(
         'rte_pmu_pmc_arm64.h',
+        'rte_pmu_pmc_x86_64.h',
 )
 
 if dpdk_conf.has('RTE_ARCH_ARM64')
lib/pmu/rte_pmu.h +2 -0
index b18938dab1..0f7004c31c 100644
@@ -28,6 +28,8 @@ extern "C" {
 
 #if defined(RTE_ARCH_ARM64)
 #include "rte_pmu_pmc_arm64.h"
+#elif defined(RTE_ARCH_X86_64)
+#include "rte_pmu_pmc_x86_64.h"
 #endif
 
 /** Maximum number of events in a group */
lib/pmu/rte_pmu_pmc_x86_64.h new file +24 -0
new file mode 100644
index 0000000000..7b67466960
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_X86_64_H_
+#define _RTE_PMU_PMC_X86_64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t low, high;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return low | (high << 32);
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_X86_64_H_ */
Signature · 1 line
2.34.1
#81 lore raw

PATCH v8 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               | 10 ++++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 +++++++++++++
 lib/eal/common/eal_common_trace.c        | 13 ++++-
 lib/eal/common/eal_common_trace_points.c |  5 ++
 lib/eal/include/rte_eal_trace.h          | 13 +++++
 lib/eal/meson.build                      |  3 ++
 lib/eal/version.map                      |  1 +
 lib/pmu/rte_pmu.c                        | 61 ++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 14 ++++++
 lib/pmu/version.map                      |  1 +
 11 files changed, 157 insertions(+), 1 deletion(-)
app/test/test_trace_perf.c +10 -0
index 46ae7d8074..f1929f2734 100644
@@ -114,6 +114,10 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+#ifdef RTE_EXEC_ENV_LINUX
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
+#endif
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +126,9 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+#ifdef RTE_EXEC_ENV_LINUX
+WORKER_DEFINE(READ_PMU)
+#endif
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +181,9 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+#ifdef RTE_EXEC_ENV_LINUX
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
+#endif
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index a8b501fe0c..6a53341c6b 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..a8e97ee1ec 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86-64 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='.*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace.c +12 -1
index 75162b722d..8796052d0c 100644
@@ -11,6 +11,9 @@
 #include <rte_errno.h>
 #include <rte_lcore.h>
 #include <rte_per_lcore.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_string_fns.h>
 
 #include "eal_trace.h"
@@ -71,8 +74,13 @@ eal_trace_init(void)
 		goto free_meta;
 
 	/* Apply global configurations */
-	STAILQ_FOREACH(arg, &trace.args, next)
+	STAILQ_FOREACH(arg, &trace.args, next) {
 		trace_args_apply(arg->val);
+#ifdef RTE_EXEC_ENV_LINUX
+		if (rte_pmu_init() == 0)
+			rte_pmu_add_events_by_pattern(arg->val);
+#endif
+	}
 
 	rte_trace_mode_set(trace.mode);
 
@@ -88,6 +96,9 @@ eal_trace_init(void)
 void
 eal_trace_fini(void)
 {
+#ifdef RTE_EXEC_ENV_LINUX
+	rte_pmu_fini();
+#endif
 	trace_mem_free();
 	trace_metadata_destroy();
 	eal_trace_args_free();
lib/eal/common/eal_common_trace_points.c +5 -0
index 0b0b254615..1e46ce549a 100644
@@ -75,3 +75,8 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
+#endif
lib/eal/include/rte_eal_trace.h +13 -0
index 5ef4398230..afb459b198 100644
@@ -17,6 +17,9 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +282,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(unsigned int index),
+	uint64_t val;
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+#endif
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/meson.build +3 -0
index 056beb9461..f5865dbcd9 100644
@@ -26,6 +26,9 @@ deps += ['kvargs']
 if not is_windows
     deps += ['telemetry']
 endif
+if is_linux
+    deps += ['pmu']
+endif
 if dpdk_conf.has('RTE_USE_LIBBSD')
     ext_deps += libbsd
 endif
lib/eal/version.map +1 -0
index 6523102157..2f8f66874b 100644
@@ -441,6 +441,7 @@ EXPERIMENTAL {
 	rte_thread_join;
 
 	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 	rte_thread_set_name;
 };
 
lib/pmu/rte_pmu.c +61 -0

67 lines not on this page · raw message

lib/pmu/rte_pmu.h +14 -0
index 0f7004c31c..0f6250e81f 100644
@@ -169,6 +169,20 @@ __rte_experimental
 int
 rte_pmu_add_event(const char *name);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add events matching pattern to the group of enabled events.
+ *
+ * @param pattern
+ *   Pattern e=ev1[,ev2,...] matching events, where evX is a placeholder for an event listed under
+ *   /sys/bus/event_source/devices/pmu/events.
+ */
+__rte_experimental
+int
+rte_pmu_add_events_by_pattern(const char *pattern);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice
lib/pmu/version.map +1 -0
index 50fb0f354e..20a27d085c 100644
@@ -8,6 +8,7 @@ EXPERIMENTAL {
 	per_lcore__event_group;
 	rte_pmu;
 	rte_pmu_add_event;
+	rte_pmu_add_events_by_pattern;
 	rte_pmu_fini;
 	rte_pmu_init;
 	rte_pmu_read;
Signature · 1 line
2.34.1
#82 lore raw

v9 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v9:
- fix 'maybe-uninitialized' warning reported by CI
v8:
- just rebase series
v7:
- use per-lcore event group instead of global table index by lcore-id
- don't add pmu_autotest to fast tests because due to lack of suported on every arch
v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there
v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library

 MAINTAINERS                              |   5 +
 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  61 +++
 app/test/test_trace_perf.c               |  10 +
 doc/api/doxy-api-index.md                |   3 +-
 doc/api/doxy-api.conf.in                 |   1 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 doc/guides/rel_notes/release_23_03.rst   |   9 +
 lib/eal/common/eal_common_trace.c        |  13 +-
 lib/eal/common/eal_common_trace_points.c |   5 +
 lib/eal/include/rte_eal_trace.h          |  13 +
 lib/eal/meson.build                      |   3 +
 lib/eal/version.map                      |   1 +
 lib/meson.build                          |   1 +
 lib/pmu/meson.build                      |  21 +
 lib/pmu/pmu_arm64.c                      |  94 ++++
 lib/pmu/pmu_private.h                    |  29 ++
 lib/pmu/rte_pmu.c                        | 525 +++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 225 ++++++++++
 lib/pmu/rte_pmu_pmc_arm64.h              |  30 ++
 lib/pmu/rte_pmu_pmc_x86_64.h             |  24 ++
 lib/pmu/version.map                      |  21 +
 23 files changed, 1138 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
 create mode 100644 lib/pmu/version.map

--
2.34.1

#83 lore raw

PATCH v9 1/4 lib: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   1 +
 app/test/test_pmu.c                    |  55 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   9 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 464 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 205 +++++++++++
 lib/pmu/version.map                    |  20 ++
 13 files changed, 813 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
index 9a0f416d2e..9f13eafd95 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build +1 -0
index f34d19e3c3..7b6b69dcf1 100644
@@ -111,6 +111,7 @@ test_sources = files(
         'test_reciprocal_division_perf.c',
         'test_red.c',
         'test_pie.c',
+        'test_pmu.c',
         'test_reorder.c',
         'test_rib.c',
         'test_rib6.c',
app/test/test_pmu.c new file +55 -0

55 lines not on this page · raw message

doc/api/doxy-api-index.md +2 -1
index de488c7abf..7f1938f92f 100644
@@ -222,7 +222,8 @@ The public API headers are grouped by topics:
   [log](@ref rte_log.h),
   [errno](@ref rte_errno.h),
   [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),
+  [pmu](@ref rte_pmu.h)
 
 - **misc**:
   [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
index f0886c3bd1..920e615996 100644
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
                           @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +8 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
doc/guides/rel_notes/release_23_03.rst +9 -0
index 73f5d94e14..733541d56c 100644
@@ -55,10 +55,19 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added PMU library.**
+
+  Added a new PMU (performance measurement unit) library which allows applications
+  to perform self monitoring activities without depending on external utilities like perf.
+  After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters
+  can be stored in CTF format for further analysis.
+
 * **Updated AMD axgbe driver.**
 
   * Added multi-process support.
 
+* **Added multi-process support for axgbe PMD.**
+
 * **Updated Corigine nfp driver.**
 
   * Added support for meter options.
lib/meson.build +1 -0
index a90fee31b7..7132131b5c 100644
@@ -11,6 +11,7 @@
 libraries = [
         'kvargs', # eal depends on kvargs
         'telemetry', # basic info querying
+        'pmu',
         'eal', # everything depends on eal
         'ring',
         'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
new file mode 100644
index 0000000000..a4160b494e
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2023 Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +29 -0
new file mode 100644
index 0000000000..849549b125
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +464 -0

464 lines not on this page · raw message

lib/pmu/rte_pmu.h new file +205 -0

205 lines not on this page · raw message

lib/pmu/version.map new file +20 -0
new file mode 100644
index 0000000000..50fb0f354e
@@ -0,0 +1,20 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
+
+INTERNAL {
+	global:
+
+	rte_pmu_enable_group;
+};
Signature · 1 line
2.34.1
#84 lore raw

Re: [PATCH v9 1/4] lib: add generic support for reading PMU events

Hello,

Quoted 7 lines · Tomasz Duszynski

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

For my understanding, what OS capability/permission are required to use this library?

Quoted 46 lines
---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   1 +
 app/test/test_pmu.c                    |  55 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   9 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 464 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 205 +++++++++++
 lib/pmu/version.map                    |  20 ++
 13 files changed, 813 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
index 9a0f416d2e..9f13eafd95 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build +0 -0
index f34d19e3c3..7b6b69dcf1 100644

app/test/meson.build test_sources = files( · line 117

Quoted 7 lines
app/test/meson.build +1 -0
@@ -111,6 +111,7 @@ test_sources = files(
         'test_reciprocal_division_perf.c',
         'test_red.c',
         'test_pie.c',
+        'test_pmu.c',
         'test_reorder.c',
         'test_rib.c',
         'test_rib6.c',

This code adds a new test.
This test should be added to an existing testsuite, like fast-tests etc...

Quoted 93 lines
app/test/test_pmu.c new file +55 -0
new file mode 100644
index 0000000000..a9bfb1a427
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include "test.h"
+
+#ifndef RTE_EXEC_ENV_LINUX
+
+static int
+test_pmu(void)
+{
+       printf("pmu_autotest only supported on Linux, skipping test\n");
+       return TEST_SKIPPED;
+}
+
+#else
+
+#include <rte_pmu.h>
+
+static int
+test_pmu_read(void)
+{
+       int tries = 10, event = -1;
+       uint64_t val = 0;
+
+       if (rte_pmu_init() < 0)
+               return TEST_FAILED;
+
+       while (tries--)
+               val += rte_pmu_read(event);
+
+       rte_pmu_fini();
+
+       return val ? TEST_SUCCESS : TEST_FAILED;
+}
+
+static struct unit_test_suite pmu_tests = {
+       .suite_name = "pmu autotest",
+       .setup = NULL,
+       .teardown = NULL,
+       .unit_test_cases = {
+               TEST_CASE(test_pmu_read),
+               TEST_CASES_END()
+       }
+};
+
+static int
+test_pmu(void)
+{
+       return unit_test_suite_runner(&pmu_tests);
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/api/doxy-api-index.md +2 -1
index de488c7abf..7f1938f92f 100644
@@ -222,7 +222,8 @@ The public API headers are grouped by topics:
   [log](@ref rte_log.h),
   [errno](@ref rte_errno.h),
   [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),
+  [pmu](@ref rte_pmu.h)
 
 - **misc**:
   [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
index f0886c3bd1..920e615996 100644
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
                           @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +0 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on

doc/guides/prog_guide/profile_app.rst Profile Your Application · line 17

Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +8 -0
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.

We need a common definition in the documentation of what PMU stands for and use it consistently.
I am not sure this documentation is the best place, but at least, I'd prefer we go with "Performance Monitoring Unit" and stick to it.

Plus, this block is a bit hard to read too, what do you think of:

"""
A majority of architectures support some performance monitoring unit (PMU). Such unit provides programmable counters that monitor specific events.

Different tools gather that information, like for example perf. However, in some scenarios when CPU cores are isolated (nohz_full) and run dedicated tasks, interrupting those tasks with perf may be undesirable.
In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``.
"""

And, a double newline is used between sections in this doc.

Quoted 8 lines
doc/guides/prog_guide/profile_app.rst +0 -0
 Profiling on x86
 ----------------
doc/guides/rel_notes/release_23_03.rst +0 -0
index 73f5d94e14..733541d56c 100644

doc/guides/rel_notes/release_23_03.rst New Features · line 60

Quoted 6 lines
doc/guides/rel_notes/release_23_03.rst +3 -0
@@ -55,10 +55,19 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added PMU library.**
+
+  Added a new PMU (performance measurement unit) library which allows applications

Performance Monitoring Unit.

doc/guides/rel_notes/release_23_03.rst New Features · line 63

Quoted 4 lines
doc/guides/rel_notes/release_23_03.rst +3 -0
+  to perform self monitoring activities without depending on external utilities like perf.
 
+  After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters
+  can be stored in CTF format for further analysis.

Afaiu, this integration comes later in the series. This part of the RN update should go with it.

doc/guides/rel_notes/release_23_03.rst New Features · line 72

Quoted 9 lines
doc/guides/rel_notes/release_23_03.rst +3 -0
+
 * **Updated AMD axgbe driver.**
 
   * Added multi-process support.
 
+* **Added multi-process support for axgbe PMD.**
+
 * **Updated Corigine nfp driver.**
 

Unrelated rebase damage.. please pay attention to such detail.

Quoted 52 lines

* Added support for meter options.

lib/meson.build +1 -0
index a90fee31b7..7132131b5c 100644
@@ -11,6 +11,7 @@
 libraries = [
         'kvargs', # eal depends on kvargs
         'telemetry', # basic info querying
+        'pmu',
         'eal', # everything depends on eal
         'ring',
         'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
new file mode 100644
index 0000000000..a4160b494e
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2023 Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +14 -0
new file mode 100644
index 0000000000..849549b125
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int

lib/pmu/pmu_private.h line 24

Quoted 10 lines
lib/pmu/pmu_private.h new file +10 -0
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.

Please describe config[].

Quoted 16 lines
lib/pmu/pmu_private.h new file +5 -0
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +5 -0
new file mode 100644
index 0000000000..4cf3161155
@@ -0,0 +1,464 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>

lib/pmu/rte_pmu.c line 15

Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>

Asking to be sure because I did not check:
do we need all those includes, or is this just copy/pasted from somewhere else?

lib/pmu/rte_pmu.c line 27

Quoted 12 lines
lib/pmu/rte_pmu.c new file +12 -0
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL

This macro is copy/pasted all over the dpdk tree...
This is worth a cleanup later, read: I am not asking for it as part of this series.

However, here, there is no need for protecting against its definition.

lib/pmu/rte_pmu.c line 33

Quoted 6 lines
lib/pmu/rte_pmu.c new file +6 -0
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h)))))
+#endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m))
+#endif

Idem.

lib/pmu/rte_pmu.c line 41

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op. Architectures should override them if
+ * necessary.
+ */

Prefer using per architectures #ifdef.
It is easier to get a simple link error than use weak symbols that make it look like it could work on some arch.

lib/pmu/rte_pmu.c line 46

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+
+int
+__rte_weak pmu_arch_init(void)
+{
+       return 0;

Add a debug log message indicating that this arch does not support PMU.

Quoted 9 lines
lib/pmu/rte_pmu.c new file +9 -0
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3])

lib/pmu/rte_pmu.c line 65

Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+{
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask)
+{
+       char *config = NULL;
+       char path[PATH_MAX];
+       int high, low, ret;
+       FILE *fp;

Reverse xmas tree when possible.

lib/pmu/rte_pmu.c line 67

Quoted 2 lines
lib/pmu/rte_pmu.c new file +2 -0
+
+       /* quiesce -Wmaybe-uninitialized warning */

This comment just seems to be a note for yourself. What was the issue exactly?

Quoted 190 lines
lib/pmu/rte_pmu.c new file +190 -0
+       *num = 0;
+       *mask = 0;
+
+       snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name, name);
+       fp = fopen(path, "r");
+       if (fp == NULL)
+               return -errno;
+
+       errno = 0;
+       ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+       if (ret < 2) {
+               ret = -ENODATA;
+               goto out;
+       }
+       if (errno) {
+               ret = -errno;
+               goto out;
+       }
+
+       if (ret == 2)
+               high = low;
+
+       *mask = GENMASK_ULL(high, low);
+       /* Last digit should be [012]. If last digit is missing 0 is implied. */
+       *num = config[strlen(config) - 1];
+       *num = isdigit(*num) ? *num - '0' : 0;
+
+       ret = 0;
+out:
+       free(config);
+       fclose(fp);
+
+       return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3])
+{
+       char *token, *term;
+       int num, ret, val;
+       uint64_t mask;
+
+       config[0] = config[1] = config[2] = 0;
+
+       token = strtok(buf, ",");
+       while (token) {
+               errno = 0;
+               /* <term>=<value> */
+               ret = sscanf(token, "%m[^=]=%i", &term, &val);
+               if (ret < 1)
+                       return -ENODATA;
+               if (errno)
+                       return -errno;
+               if (ret == 1)
+                       val = 1;
+
+               ret = get_term_format(term, &num, &mask);
+               free(term);
+               if (ret)
+                       return ret;
+
+               config[num] |= FIELD_PREP(mask, val);
+               token = strtok(NULL, ",");
+       }
+
+       return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3])
+{
+       char path[PATH_MAX], buf[BUFSIZ];
+       FILE *fp;
+       int ret;
+
+       snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+       fp = fopen(path, "r");
+       if (fp == NULL)
+               return -errno;
+
+       ret = fread(buf, 1, sizeof(buf), fp);
+       if (ret == 0) {
+               fclose(fp);
+
+               return -EINVAL;
+       }
+       fclose(fp);
+       buf[ret] = '\0';
+
+       return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd)
+{
+       struct perf_event_attr attr = {
+               .size = sizeof(struct perf_event_attr),
+               .type = PERF_TYPE_RAW,
+               .exclude_kernel = 1,
+               .exclude_hv = 1,
+               .disabled = 1,
+       };
+
+       pmu_arch_fixup_config(config);
+
+       attr.config = config[0];
+       attr.config1 = config[1];
+       attr.config2 = config[2];
+
+       return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0);
+}
+
+static int
+open_events(struct rte_pmu_event_group *group)
+{
+       struct rte_pmu_event *event;
+       uint64_t config[3];
+       int num = 0, ret;
+
+       /* group leader gets created first, with fd = -1 */
+       group->fds[0] = -1;
+
+       TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+               ret = get_event_config(event->name, config);
+               if (ret)
+                       continue;
+
+               ret = do_perf_event_open(config, group->fds[0]);
+               if (ret == -1) {
+                       ret = -errno;
+                       goto out;
+               }
+
+               group->fds[event->index] = ret;
+               num++;
+       }
+
+       return 0;
+out:
+       for (--num; num >= 0; num--) {
+               close(group->fds[num]);
+               group->fds[num] = -1;
+       }
+
+
+       return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group)
+{
+       long page_size = sysconf(_SC_PAGE_SIZE);
+       unsigned int i;
+       void *addr;
+       int ret;
+
+       for (i = 0; i < rte_pmu.num_group_events; i++) {
+               addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+               if (addr == MAP_FAILED) {
+                       ret = -errno;
+                       goto out;
+               }
+
+               group->mmap_pages[i] = addr;
+       }
+
+       return 0;
+out:
+       for (; i; i--) {
+               munmap(group->mmap_pages[i - 1], page_size);
+               group->mmap_pages[i - 1] = NULL;
+       }
+
+       return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group)
+{
+       unsigned int i;
+
+       if (group->fds[0] != -1)
+               ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+       for (i = 0; i < rte_pmu.num_group_events; i++) {
+               if (group->mmap_pages[i]) {
+                       munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+                       group->mmap_pages[i] = NULL;
+               }
+

lib/pmu/rte_pmu.c line 267

Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+               if (group->fds[i] != -1) {
+                       close(group->fds[i]);
+                       group->fds[i] = -1;
+               }
+       }
+
+       group->enabled = false;
+}
+
+int __rte_noinline

This symbol is exported out of this library, no need for noinline.

Quoted 137 lines
lib/pmu/rte_pmu.c new file +137 -0
+rte_pmu_enable_group(void)
+{
+       struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+       int ret;
+
+       if (rte_pmu.num_group_events == 0)
+               return -ENODEV;
+
+       ret = open_events(group);
+       if (ret)
+               goto out;
+
+       ret = mmap_events(group);
+       if (ret)
+               goto out;
+
+       if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+               ret = -errno;
+               goto out;
+       }
+
+       if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+               ret = -errno;
+               goto out;
+       }
+
+       rte_spinlock_lock(&rte_pmu.lock);
+       TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);
+       rte_spinlock_unlock(&rte_pmu.lock);
+       group->enabled = true;
+
+       return 0;
+
+out:
+       cleanup_events(group);
+
+       return ret;
+}
+
+static int
+scan_pmus(void)
+{
+       char path[PATH_MAX];
+       struct dirent *dent;
+       const char *name;
+       DIR *dirp;
+
+       dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+       if (dirp == NULL)
+               return -errno;
+
+       while ((dent = readdir(dirp))) {
+               name = dent->d_name;
+               if (name[0] == '.')
+                       continue;
+
+               /* sysfs entry should either contain cpus or be a cpu */
+               if (!strcmp(name, "cpu"))
+                       break;
+
+               snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+               if (access(path, F_OK) == 0)
+                       break;
+       }
+
+       if (dent) {
+               rte_pmu.name = strdup(name);
+               if (rte_pmu.name == NULL) {
+                       closedir(dirp);
+
+                       return -ENOMEM;
+               }
+       }
+
+       closedir(dirp);
+
+       return rte_pmu.name ? 0 : -ENODEV;
+}
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+       struct rte_pmu_event *event;
+
+       event = calloc(1, sizeof(*event));
+       if (event == NULL)
+               goto out;
+
+       event->name = strdup(name);
+       if (event->name == NULL) {
+               free(event);
+               event = NULL;
+       }
+
+out:
+       return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event)
+{
+       free(event->name);
+       free(event);
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+       struct rte_pmu_event *event;
+       char path[PATH_MAX];
+
+       if (rte_pmu.name == NULL)
+               return -ENODEV;
+
+       if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+               return -ENOSPC;
+
+       snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+       if (access(path, R_OK))
+               return -ENODEV;
+
+       TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+               if (!strcmp(event->name, name))
+                       return event->index;
+               continue;
+       }
+
+       event = new_event(name);
+       if (event == NULL)
+               return -ENOMEM;
+
+       event->index = rte_pmu.num_group_events++;
+       TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+       return event->index;
+}
+

lib/pmu/rte_pmu.c line 414

Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+int
+rte_pmu_init(void)
+{
+       int ret;
+
+       /* Allow calling init from multiple contexts within a single thread. This simplifies
+        * resource management a bit e.g in case fast-path tracepoint has already been enabled
+        * via command line but application doesn't care enough and performs init/fini again.
+        */
+       if (rte_pmu.initialized) {

This is an integer so check against 0 explicitly (there may be other cases in this patch, I did not recheck the whole patch).

Quoted 86 lines
lib/pmu/rte_pmu.c new file +50 -0
+               rte_pmu.initialized++;
+               return 0;
+       }
+
+       ret = scan_pmus();
+       if (ret)
+               goto out;
+
+       ret = pmu_arch_init();
+       if (ret)
+               goto out;
+
+       TAILQ_INIT(&rte_pmu.event_list);
+       TAILQ_INIT(&rte_pmu.event_group_list);
+       rte_spinlock_init(&rte_pmu.lock);
+       rte_pmu.initialized = 1;
+
+       return 0;
+out:
+       free(rte_pmu.name);
+       rte_pmu.name = NULL;
+
+       return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+       struct rte_pmu_event_group *group, *tmp_group;
+       struct rte_pmu_event *event, *tmp_event;
+
+       /* cleanup once init count drops to zero */
+       if (!rte_pmu.initialized || --rte_pmu.initialized)
+               return;
+
+       RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+               TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+               free_event(event);
+       }
+
+       RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+               TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+               cleanup_events(group);
+       }
+
+       pmu_arch_fini();
+       free(rte_pmu.name);
+       rte_pmu.name = NULL;
+       rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file +30 -0
new file mode 100644
index 0000000000..e360375a0c
@@ -0,0 +1,205 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_compat.h>
+#include <rte_spinlock.h>
+
+/** Maximum number of events in a group */
+#define MAX_NUM_GROUP_EVENTS 8

lib/pmu/rte_pmu.h line 40

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+       struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */
+       int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+       bool enabled; /**< true if group was enabled on particular lcore */
+       TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */
+} __rte_cache_aligned;

One problem for the future is that we have a fixed size fd array. Do we need to expose this whole structure to the application?

lib/pmu/rte_pmu.h line 47

Quoted 7 lines
lib/pmu/rte_pmu.h new file +7 -0
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+       char *name; /**< name of an event */
+       unsigned int index; /**< event index into fds/mmap_pages */

This is an internal consideration.
Do we need to expose this to the application?

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+       TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */
+};
+
+/**

lib/pmu/rte_pmu.h line 61

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+ * A PMU state container.
+ */
+struct rte_pmu {
+       char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+       rte_spinlock_t lock; /**< serialize access to event group list */
+       TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+       unsigned int num_group_events; /**< number of events in a group */
+       TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+       unsigned int initialized; /**< initialization counter */
+};

Idem, do we need to expose this to the application?

Quoted 53 lines
lib/pmu/rte_pmu.h new file +53 -0
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version */
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+       uint64_t width, offset;
+       uint32_t seq, index;
+       int64_t pmc;
+
+       for (;;) {
+               seq = pc->lock;
+               rte_compiler_barrier();
+               index = pc->index;
+               offset = pc->offset;
+               width = pc->pmc_width;
+
+               /* index set to 0 means that particular counter cannot be used */
+               if (likely(pc->cap_user_rdpmc && index)) {
+                       pmc = rte_pmu_pmc_read(index - 1);
+                       pmc <<= 64 - width;
+                       pmc >>= 64 - width;
+                       offset += pmc;
+               }
+
+               rte_compiler_barrier();
+
+               if (likely(pc->lock == seq))
+                       return offset;
+       }
+
+       return 0;
+}

lib/pmu/rte_pmu.h line 124

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+
+/**
+ * @internal
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal

Unless I missed something, this symbol is called from rte_pmu_read() so this makes rte_pmu_read() itself internal.
So external applications won't be able to use the PMU API.

This can probably be confirmed by adding some call to the PMU API in an examples/.

Quoted 110 lines
lib/pmu/rte_pmu.h new file +81 -0
+int
+rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+       struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+       int ret;
+
+       if (unlikely(!rte_pmu.initialized))
+               return 0;
+
+       if (unlikely(!group->enabled)) {
+               ret = rte_pmu_enable_group();
+               if (ret)
+                       return 0;
+       }
+
+       if (unlikely(index >= rte_pmu.num_group_events))
+               return 0;
+
+       return rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/pmu/version.map new file +20 -1
new file mode 100644
index 0000000000..50fb0f354e
@@ -0,0 +1,20 @@
+DPDK_23 {
+       local: *;
+};
+
+EXPERIMENTAL {
+       global:
+
+       per_lcore__event_group;
+       rte_pmu;
+       rte_pmu_add_event;
+       rte_pmu_fini;
+       rte_pmu_init;
+       rte_pmu_read;
+};
+
+INTERNAL {
+       global:
+
+       rte_pmu_enable_group;
+};
--

2.34.1

Signature · 1 line
David Marchand
#85 lore raw

RE: [EXT] Re: [PATCH v9 1/4] lib: add generic support for reading PMU events

Hi David,

Thanks for review. Comments inline.

Quoted 16 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, February 6, 2023 12:03 PM
To: Tomasz Duszynski <[email protected]>
Cc: [email protected]; Thomas Monjalon <[email protected]>; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v9 1/4] lib: add generic support for reading PMU events

External Email

----------------------------------------------------------------------
Hello,

Quoted 1 line · Tomasz Duszynski
Quoted 9 lines
Quoted 6 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

For my understanding, what OS capability/permission are required to use this library?

On x86 it sufficient for self-monitoring to have kernel built with perf events enabled and /proc/sys/kernel/perf_event_paranoid knob should be set to 2, which should be a default value anyway, unless changed by some scripts.

On ARM64 you need to additionally set /proc/sys/kernel/perf_user_access to bypass kernel when accessing hw counters.

Quoted 160 lines
Quoted 45 lines
---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   1 +
 app/test/test_pmu.c                    |  55 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   9 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 464 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 205 +++++++++++
 lib/pmu/version.map                    |  20 ++
 13 files changed, 813 insertions(+), 1 deletion(-)  create mode

100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.map

MAINTAINERS index 9a0f416d2e..9f13eafd95 renamed from MAINTAINERS +0 -0

100644

MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build index renamed from app/test/meson.build +0 -0

f34d19e3c3..7b6b69dcf1 100644

app/test/meson.build test_sources = files( · line 117

Quoted 7 lines
app/test/meson.build +1 -0
@@ -111,6 +111,7 @@ test_sources = files(
         'test_reciprocal_division_perf.c',
         'test_red.c',
         'test_pie.c',
+        'test_pmu.c',
         'test_reorder.c',
         'test_rib.c',
         'test_rib6.c',

This code adds a new test.
This test should be added to an existing testsuite, like fast-tests etc...

Quoted 93 lines
app/test/test_pmu.c new file mode renamed from app/test/test_pmu.c +0 -0

100644 index 0000000000..a9bfb1a427

app/test/test_pmu.c new file +54 -0
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include "test.h"
+
+#ifndef RTE_EXEC_ENV_LINUX
+
+static int
+test_pmu(void)
+{
+       printf("pmu_autotest only supported on Linux, skipping test\n");
+       return TEST_SKIPPED;
+}
+
+#else
+
+#include <rte_pmu.h>
+
+static int
+test_pmu_read(void)
+{
+       int tries = 10, event = -1;
+       uint64_t val = 0;
+
+       if (rte_pmu_init() < 0)
+               return TEST_FAILED;
+
+       while (tries--)
+               val += rte_pmu_read(event);
+
+       rte_pmu_fini();
+
+       return val ? TEST_SUCCESS : TEST_FAILED; }
+
+static struct unit_test_suite pmu_tests = {
+       .suite_name = "pmu autotest",
+       .setup = NULL,
+       .teardown = NULL,
+       .unit_test_cases = {
+               TEST_CASE(test_pmu_read),
+               TEST_CASES_END()
+       }
+};
+
+static int
+test_pmu(void)
+{
+       return unit_test_suite_runner(&pmu_tests);
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/api/doxy-api-index.md +1 -1
index de488c7abf..7f1938f92f 100644
@@ -222,7 +222,8 @@ The public API headers are grouped by topics:
   [log](@ref rte_log.h),
   [errno](@ref rte_errno.h),
   [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),  [pmu](@ref rte_pmu.h)
 
 - **misc**:
   [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0 -0

f0886c3bd1..920e615996 100644

doc/api/doxy-api.conf.in +1 -0
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
                           @TOPDIR@/lib/rawdev \ diff --git

a/doc/guides/prog_guide/profile_app.rst
b/doc/guides/prog_guide/profile_app.rst
index 14292d4c25..a8b501fe0c 100644

doc/guides/prog_guide/profile_app.rst +0 -0
@@ -7,6 +7,14 @@ Profile Your Application  The following sections

describe methods of profiling DPDK applications on different architectures.

doc/guides/prog_guide/profile_app.rst

Quoted 8 lines
doc/guides/prog_guide/profile_app.rst +8 -0
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit
+which provides a set of programmable counters that monitor specific
+events. There are different tools which can gather that information,
+perf being an example here. Though in some scenarios, eg. when CPU
+cores are isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such

cases one can read specific events directly from application via ``rte_pmu_read()``.

Quoted 3 lines

We need a common definition in the documentation of what PMU stands for and use it consistently. I am not sure this documentation is the best place, but at least, I'd prefer we go with "Performance Monitoring Unit" and stick to it.

For the time being I think it's good enough. Frankly I don’t have better idea where to put that.

Quoted 7 lines

Plus, this block is a bit hard to read too, what do you think of:

"""
A majority of architectures support some performance monitoring unit (PMU). Such unit provides programmable counters that monitor specific events.

Quoted 9 lines

Different tools gather that information, like for example perf.
However, in some scenarios when CPU cores are isolated (nohz_full) and run dedicated tasks, interrupting those tasks with perf may be undesirable.
In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``. """

And, a double newline is used between sections in this doc.

No problem.

Quoted 31 lines
Quoted 9 lines

Profiling on x86
----------------

doc/guides/rel_notes/release_23_03.rst +0 -0
index 73f5d94e14..733541d56c 100644

doc/guides/rel_notes/release_23_03.rst New Features · line 61

Quoted 7 lines
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -55,10 +55,19 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added PMU library.**
+
+  Added a new PMU (performance measurement unit) library which allows
+ applications

Performance Monitoring Unit.

doc/guides/rel_notes/release_23_03.rst New Features

Quoted 4 lines
doc/guides/rel_notes/release_23_03.rst +3 -0
+  to perform self monitoring activities without depending on external utilities like perf.
 
+  After integration with :doc:`../prog_guide/trace_lib` data gathered
+ from hardware counters  can be stored in CTF format for further analysis.

Afaiu, this integration comes later in the series. This part of the RN update should go with it.

Quoted 3 lines
doc/guides/rel_notes/release_23_03.rst +1 -0
+
 * **Updated AMD axgbe driver.**
 
Quoted 9 lines

doc/guides/rel_notes/release_23_03.rst New Features · line 71

Quoted 6 lines
doc/guides/rel_notes/release_23_03.rst +2 -0
   * Added multi-process support.
 
+* **Added multi-process support for axgbe PMD.**
+
 * **Updated Corigine nfp driver.**

Unrelated rebase damage.. please pay attention to such detail.

Thanks, that creeped in somehow.

Quoted 55 lines
Quoted 55 lines
doc/guides/rel_notes/release_23_03.rst +0 -0
   * Added support for meter options.
lib/meson.build index renamed from lib/meson.build +0 -0

a90fee31b7..7132131b5c 100644

lib/meson.build +1 -0
@@ -11,6 +11,7 @@
 libraries = [
         'kvargs', # eal depends on kvargs
         'telemetry', # basic info querying
+        'pmu',
         'eal', # everything depends on eal
         'ring',
         'rcu', # rcu depends on ring
lib/pmu/meson.build new file mode renamed from lib/pmu/meson.build +0 -0

100644 index 0000000000..a4160b494e

lib/pmu/meson.build new file +13 -0
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(C) 2023 Marvell
+International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file renamed from lib/pmu/pmu_private.h +0 -0

mode 100644 index 0000000000..849549b125

lib/pmu/pmu_private.h new file +18 -0
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
Quoted 9 lines

lib/pmu/pmu_private.h line 24

Quoted 6 lines
lib/pmu/pmu_private.h new file +6 -0
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.

Please describe config[].

Well, the problem here is that each and every arch may expect different config so anyone adding new stuff would need to consult kernel sources anyway.

What kind of documentation would you like to see here?

Quoted 21 lines
Quoted 21 lines
lib/pmu/pmu_private.h new file +5 -0
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..4cf3161155

lib/pmu/rte_pmu.c new file +10 -0
@@ -0,0 +1,464 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
Quoted 9 lines

lib/pmu/rte_pmu.c line 15

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>

Asking to be sure because I did not check:
do we need all those includes, or is this just copy/pasted from somewhere else?

No, it was not copy/pasted. Each header exports something used in this file. I'll double check if all still required.

Quoted 17 lines

lib/pmu/rte_pmu.c line 27

Quoted 13 lines
lib/pmu/rte_pmu.c new file +12 -0
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL

This macro is copy/pasted all over the dpdk tree...
This is worth a cleanup later, read: I am not asking for it as part of this series.

Quoted 2 lines

However, here, there is no need for protecting against its definition.

Okay.

Quoted 11 lines

lib/pmu/rte_pmu.c line 33

Quoted 7 lines
lib/pmu/rte_pmu.c new file +6 -0
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1))
+& (m)) #endif

Idem.

Quoted 14 lines

lib/pmu/rte_pmu.c line 41

Quoted 9 lines
lib/pmu/rte_pmu.c new file +9 -0
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */

Prefer using per architectures #ifdef.
It is easier to get a simple link error than use weak symbols that make it look like it could work on some arch.

Rationale for that was actually to not break anything. That means you can compile that code for every arch, except on unsupported ones you'll see stream of zeros inside a trace file.

Quoted 9 lines

lib/pmu/rte_pmu.c line 46

Quoted 6 lines
lib/pmu/rte_pmu.c new file +5 -0
+
+int
+__rte_weak pmu_arch_init(void)
+{
+       return 0;

Add a debug log message indicating that this arch does not support PMU.

Prove me wrong but logs are part of eal and unless they are moved to a separate library this lib shouldn't call these APIs. Otherwise we'll introduce dependency on eal which we wanted to avoid in the first place.

Quoted 11 lines
Quoted 11 lines
lib/pmu/rte_pmu.c new file +10 -0
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { }
+
Quoted 9 lines

lib/pmu/rte_pmu.c line 65

Quoted 6 lines
lib/pmu/rte_pmu.c new file +6 -0
+static int
+get_term_format(const char *name, int *num, uint64_t *mask) {
+       char *config = NULL;
+       char path[PATH_MAX];
+       int high, low, ret;
+       FILE *fp;

Reverse xmas tree when possible.

Okay.

Quoted 7 lines

lib/pmu/rte_pmu.c line 65

Quoted 3 lines
lib/pmu/rte_pmu.c new file +2 -0
+
+       /* quiesce -Wmaybe-uninitialized warning */

This comment just seems to be a note for yourself. What was the issue exactly?

Generally speaking, compiler thinks that function calling this function may use 'num' even though callee returned an error.

[1/198] Compiling C object lib/librte_pmu.a.p/pmu_rte_pmu.c.o In function _parse_event_,

    inlined from _get_event_config_ at ../lib/pmu/rte_pmu.c:157:9:
../lib/pmu/rte_pmu.c:129:23: warning: _num_ may be used uninitialized [-Wmaybe-uninitialized]
  129 |                 config[num] |= FIELD_PREP(mask, val);
      |                 ~~~~~~^~~~~
../lib/pmu/rte_pmu.c: In function _get_event_config_:
../lib/pmu/rte_pmu.c:107:13: note: _num_ was declared here
  107 |         int num, ret, val;
      |
Quoted 191 lines

lib/pmu/rte_pmu.c

Quoted 5 lines
lib/pmu/rte_pmu.c new file +4 -0
+       *num = 0;
+       *mask = 0;
+
+       snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name,

name);

Quoted 60 lines
lib/pmu/rte_pmu.c new file +60 -0
+       fp = fopen(path, "r");
+       if (fp == NULL)
+               return -errno;
+
+       errno = 0;
+       ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+       if (ret < 2) {
+               ret = -ENODATA;
+               goto out;
+       }
+       if (errno) {
+               ret = -errno;
+               goto out;
+       }
+
+       if (ret == 2)
+               high = low;
+
+       *mask = GENMASK_ULL(high, low);
+       /* Last digit should be [012]. If last digit is missing 0 is implied. */
+       *num = config[strlen(config) - 1];
+       *num = isdigit(*num) ? *num - '0' : 0;
+
+       ret = 0;
+out:
+       free(config);
+       fclose(fp);
+
+       return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+       char *token, *term;
+       int num, ret, val;
+       uint64_t mask;
+
+       config[0] = config[1] = config[2] = 0;
+
+       token = strtok(buf, ",");
+       while (token) {
+               errno = 0;
+               /* <term>=<value> */
+               ret = sscanf(token, "%m[^=]=%i", &term, &val);
+               if (ret < 1)
+                       return -ENODATA;
+               if (errno)
+                       return -errno;
+               if (ret == 1)
+                       val = 1;
+
+               ret = get_term_format(term, &num, &mask);
+               free(term);
+               if (ret)
+                       return ret;
+
+               config[num] |= FIELD_PREP(mask, val);
+               token = strtok(NULL, ",");
+       }
+

lib/pmu/rte_pmu.c

Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+       return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+       char path[PATH_MAX], buf[BUFSIZ];
+       FILE *fp;
+       int ret;
+
+       snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,

name);

Quoted 114 lines
lib/pmu/rte_pmu.c new file +114 -0
+       fp = fopen(path, "r");
+       if (fp == NULL)
+               return -errno;
+
+       ret = fread(buf, 1, sizeof(buf), fp);
+       if (ret == 0) {
+               fclose(fp);
+
+               return -EINVAL;
+       }
+       fclose(fp);
+       buf[ret] = '\0';
+
+       return parse_event(buf, config); }
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd) {
+       struct perf_event_attr attr = {
+               .size = sizeof(struct perf_event_attr),
+               .type = PERF_TYPE_RAW,
+               .exclude_kernel = 1,
+               .exclude_hv = 1,
+               .disabled = 1,
+       };
+
+       pmu_arch_fixup_config(config);
+
+       attr.config = config[0];
+       attr.config1 = config[1];
+       attr.config2 = config[2];
+
+       return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd,
+0); }
+
+static int
+open_events(struct rte_pmu_event_group *group) {
+       struct rte_pmu_event *event;
+       uint64_t config[3];
+       int num = 0, ret;
+
+       /* group leader gets created first, with fd = -1 */
+       group->fds[0] = -1;
+
+       TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+               ret = get_event_config(event->name, config);
+               if (ret)
+                       continue;
+
+               ret = do_perf_event_open(config, group->fds[0]);
+               if (ret == -1) {
+                       ret = -errno;
+                       goto out;
+               }
+
+               group->fds[event->index] = ret;
+               num++;
+       }
+
+       return 0;
+out:
+       for (--num; num >= 0; num--) {
+               close(group->fds[num]);
+               group->fds[num] = -1;
+       }
+
+
+       return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group) {
+       long page_size = sysconf(_SC_PAGE_SIZE);
+       unsigned int i;
+       void *addr;
+       int ret;
+
+       for (i = 0; i < rte_pmu.num_group_events; i++) {
+               addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+               if (addr == MAP_FAILED) {
+                       ret = -errno;
+                       goto out;
+               }
+
+               group->mmap_pages[i] = addr;
+       }
+
+       return 0;
+out:
+       for (; i; i--) {
+               munmap(group->mmap_pages[i - 1], page_size);
+               group->mmap_pages[i - 1] = NULL;
+       }
+
+       return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group) {
+       unsigned int i;
+
+       if (group->fds[0] != -1)
+               ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE,
+ PERF_IOC_FLAG_GROUP);
+
+       for (i = 0; i < rte_pmu.num_group_events; i++) {
+               if (group->mmap_pages[i]) {
+                       munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+                       group->mmap_pages[i] = NULL;
+               }
+
+               if (group->fds[i] != -1) {
+                       close(group->fds[i]);
+                       group->fds[i] = -1;
+               }
Quoted 9 lines

lib/pmu/rte_pmu.c line 267

Quoted 6 lines
lib/pmu/rte_pmu.c new file +6 -0
+       }
+
+       group->enabled = false;
+}
+
+int __rte_noinline

This symbol is exported out of this library, no need for noinline.

I recall that it was added deliberately because this function was actually being inlined. But given code changed a bit through multiple revisions this may not be necessary anymore.

Quoted 142 lines
Quoted 107 lines
lib/pmu/rte_pmu.c new file +106 -0
+rte_pmu_enable_group(void)
+{
+       struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+       int ret;
+
+       if (rte_pmu.num_group_events == 0)
+               return -ENODEV;
+
+       ret = open_events(group);
+       if (ret)
+               goto out;
+
+       ret = mmap_events(group);
+       if (ret)
+               goto out;
+
+       if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+               ret = -errno;
+               goto out;
+       }
+
+       if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+               ret = -errno;
+               goto out;
+       }
+
+       rte_spinlock_lock(&rte_pmu.lock);
+       TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);
+       rte_spinlock_unlock(&rte_pmu.lock);
+       group->enabled = true;
+
+       return 0;
+
+out:
+       cleanup_events(group);
+
+       return ret;
+}
+
+static int
+scan_pmus(void)
+{
+       char path[PATH_MAX];
+       struct dirent *dent;
+       const char *name;
+       DIR *dirp;
+
+       dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+       if (dirp == NULL)
+               return -errno;
+
+       while ((dent = readdir(dirp))) {
+               name = dent->d_name;
+               if (name[0] == '.')
+                       continue;
+
+               /* sysfs entry should either contain cpus or be a cpu */
+               if (!strcmp(name, "cpu"))
+                       break;
+
+               snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+               if (access(path, F_OK) == 0)
+                       break;
+       }
+
+       if (dent) {
+               rte_pmu.name = strdup(name);
+               if (rte_pmu.name == NULL) {
+                       closedir(dirp);
+
+                       return -ENOMEM;
+               }
+       }
+
+       closedir(dirp);
+
+       return rte_pmu.name ? 0 : -ENODEV; }
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+       struct rte_pmu_event *event;
+
+       event = calloc(1, sizeof(*event));
+       if (event == NULL)
+               goto out;
+
+       event->name = strdup(name);
+       if (event->name == NULL) {
+               free(event);
+               event = NULL;
+       }
+
+out:
+       return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event) {
+       free(event->name);
+       free(event);
+}
+
+int
+rte_pmu_add_event(const char *name)
+{

lib/pmu/rte_pmu.c

Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+       struct rte_pmu_event *event;
+       char path[PATH_MAX];
+
+       if (rte_pmu.name == NULL)
+               return -ENODEV;
+
+       if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+               return -ENOSPC;
+
+       snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,

name);

Quoted 24 lines
lib/pmu/rte_pmu.c new file +24 -0
+       if (access(path, R_OK))
+               return -ENODEV;
+
+       TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+               if (!strcmp(event->name, name))
+                       return event->index;
+               continue;
+       }
+
+       event = new_event(name);
+       if (event == NULL)
+               return -ENOMEM;
+
+       event->index = rte_pmu.num_group_events++;
+       TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+       return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+       int ret;
+
Quoted 9 lines

lib/pmu/rte_pmu.c line 414

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+       /* Allow calling init from multiple contexts within a single thread. This simplifies
+        * resource management a bit e.g in case fast-path tracepoint has already been enabled
+        * via command line but application doesn't care enough and performs init/fini again.
+        */
+       if (rte_pmu.initialized) {

This is an integer so check against 0 explicitly (there may be other cases in this patch, I did not recheck the whole patch).

Okay.

Quoted 93 lines
Quoted 82 lines
lib/pmu/rte_pmu.c new file +50 -0
+               rte_pmu.initialized++;
+               return 0;
+       }
+
+       ret = scan_pmus();
+       if (ret)
+               goto out;
+
+       ret = pmu_arch_init();
+       if (ret)
+               goto out;
+
+       TAILQ_INIT(&rte_pmu.event_list);
+       TAILQ_INIT(&rte_pmu.event_group_list);
+       rte_spinlock_init(&rte_pmu.lock);
+       rte_pmu.initialized = 1;
+
+       return 0;
+out:
+       free(rte_pmu.name);
+       rte_pmu.name = NULL;
+
+       return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+       struct rte_pmu_event_group *group, *tmp_group;
+       struct rte_pmu_event *event, *tmp_event;
+
+       /* cleanup once init count drops to zero */
+       if (!rte_pmu.initialized || --rte_pmu.initialized)
+               return;
+
+       RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+               TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+               free_event(event);
+       }
+
+       RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+               TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+               cleanup_events(group);
+       }
+
+       pmu_arch_fini();
+       free(rte_pmu.name);
+       rte_pmu.name = NULL;
+       rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..e360375a0c

lib/pmu/rte_pmu.h new file +26 -0
@@ -0,0 +1,205 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_compat.h>

lib/pmu/rte_pmu.h line 36

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+#include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+       struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages

*/

Quoted 6 lines

lib/pmu/rte_pmu.h

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+       int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+       bool enabled; /**< true if group was enabled on particular lcore */
+       TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;

One problem for the future is that we have a fixed size fd array.

This number can be increased if needed but rationale for that was PMUs come with relatively small number of hw counters. Here events are grouped together (i.e scheduled together) which means there must be enough hw counters for all events for things to work.

Quoted 2 lines

Do we need to expose this whole structure to the application?

Probably some part of it could have been hidden but all those structures are so small that presumably such partitioning would not bring much to the table.

Quoted 95 lines

lib/pmu/rte_pmu.h line 47

Quoted 8 lines
lib/pmu/rte_pmu.h new file +7 -0
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+       char *name; /**< name of an event */
+       unsigned int index; /**< event index into fds/mmap_pages */

This is an internal consideration.
Do we need to expose this to the application?

lib/pmu/rte_pmu.h

Quoted 12 lines
lib/pmu/rte_pmu.h new file +12 -0
+       TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+       char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+       rte_spinlock_t lock; /**< serialize access to event group list */
+       TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+       unsigned int num_group_events; /**< number of events in a group */
+       TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+       unsigned int initialized; /**< initialization counter */ };

Idem, do we need to expose this to the application?

Quoted 51 lines
lib/pmu/rte_pmu.h new file +51 -0
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version
+*/ #ifndef rte_pmu_pmc_read #define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+       uint64_t width, offset;
+       uint32_t seq, index;
+       int64_t pmc;
+
+       for (;;) {
+               seq = pc->lock;
+               rte_compiler_barrier();
+               index = pc->index;
+               offset = pc->offset;
+               width = pc->pmc_width;
+
+               /* index set to 0 means that particular counter cannot be used */
+               if (likely(pc->cap_user_rdpmc && index)) {
+                       pmc = rte_pmu_pmc_read(index - 1);
+                       pmc <<= 64 - width;
+                       pmc >>= 64 - width;
+                       offset += pmc;
+               }
+
+               rte_compiler_barrier();
+
+               if (likely(pc->lock == seq))
+                       return offset;
+       }
+
+       return 0;
+}

lib/pmu/rte_pmu.h line 124

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+
+/**
+ * @internal
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal

Unless I missed something, this symbol is called from rte_pmu_read() so this makes rte_pmu_read() itself internal.
So external applications won't be able to use the PMU API.

Quoted 2 lines

This can probably be confirmed by adding some call to the PMU API in an examples/.

Good point actually. This was not that obvious when I looked at the patch introducing that change. So in this case it needs to be exported but given app should not call that itself maybe I'll just make the intent clear by renaming it perhaps to __rte_pmu_enable_group() or something alike.

Quoted 114 lines
Quoted 108 lines
lib/pmu/rte_pmu.h new file +81 -0
+int
+rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+       struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+       int ret;
+
+       if (unlikely(!rte_pmu.initialized))
+               return 0;
+
+       if (unlikely(!group->enabled)) {
+               ret = rte_pmu_enable_group();
+               if (ret)
+                       return 0;
+       }
+
+       if (unlikely(index >= rte_pmu.num_group_events))
+               return 0;
+
+       return rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/pmu/version.map new file mode renamed from lib/pmu/version.map +0 -0

100644 index 0000000000..50fb0f354e

lib/pmu/version.map new file +20 -1
@@ -0,0 +1,20 @@
+DPDK_23 {
+       local: *;
+};
+
+EXPERIMENTAL {
+       global:
+
+       per_lcore__event_group;
+       rte_pmu;
+       rte_pmu_add_event;
+       rte_pmu_fini;
+       rte_pmu_init;
+       rte_pmu_read;
+};
+
+INTERNAL {
+       global:
+
+       rte_pmu_enable_group;
+};
--
Quoted 2 lines

2.34.1

--
David Marchand

#86 lore raw

PATCH v9 2/4 pmu: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c         |  4 ++
 lib/pmu/meson.build         |  7 +++
 lib/pmu/pmu_arm64.c         | 94 +++++++++++++++++++++++++++++++++++++
 lib/pmu/rte_pmu.h           |  4 ++
 lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++
 5 files changed, 139 insertions(+)
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
app/test/test_pmu.c +4 -0
index a9bfb1a427..623e04b691 100644
@@ -26,6 +26,10 @@ test_pmu_read(void)
 	if (rte_pmu_init() < 0)
 		return TEST_FAILED;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/pmu/meson.build +7 -0
index a4160b494e..e857681137 100644
@@ -11,3 +11,10 @@ includes = [global_inc]
 
 sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
+indirect_headers += files(
+        'rte_pmu_pmc_arm64.h',
+)
+
+if dpdk_conf.has('RTE_ARCH_ARM64')
+    sources += files('pmu_arm64.c')
+endif
lib/pmu/pmu_arm64.c new file +94 -0

94 lines not on this page · raw message

lib/pmu/rte_pmu.h +4 -0
index e360375a0c..b18938dab1 100644
@@ -26,6 +26,10 @@ extern "C" {
 #include <rte_compat.h>
 #include <rte_spinlock.h>
 
+#if defined(RTE_ARCH_ARM64)
+#include "rte_pmu_pmc_arm64.h"
+#endif
+
 /** Maximum number of events in a group */
 #define MAX_NUM_GROUP_EVENTS 8
 
lib/pmu/rte_pmu_pmc_arm64.h new file +30 -0
new file mode 100644
index 0000000000..10648f0c5f
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_ARM64_H_
+#define _RTE_PMU_PMC_ARM64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_ARM64_H_ */
Signature · 1 line
2.34.1
#87 lore raw

PATCH v9 3/4 pmu: support reading Intel x86_64 PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel x86_64 PMU events in runtime.

---
 app/test/test_pmu.c          |  2 ++
 lib/pmu/meson.build          |  1 +
 lib/pmu/rte_pmu.h            |  2 ++
 lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++++++++++++++++++++++++
 4 files changed, 29 insertions(+)
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
app/test/test_pmu.c +2 -0
index 623e04b691..614395482f 100644
@@ -28,6 +28,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/pmu/meson.build +1 -0
index e857681137..5b92e5c4e3 100644
@@ -13,6 +13,7 @@ sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
 indirect_headers += files(
         'rte_pmu_pmc_arm64.h',
+        'rte_pmu_pmc_x86_64.h',
 )
 
 if dpdk_conf.has('RTE_ARCH_ARM64')
lib/pmu/rte_pmu.h +2 -0
index b18938dab1..0f7004c31c 100644
@@ -28,6 +28,8 @@ extern "C" {
 
 #if defined(RTE_ARCH_ARM64)
 #include "rte_pmu_pmc_arm64.h"
+#elif defined(RTE_ARCH_X86_64)
+#include "rte_pmu_pmc_x86_64.h"
 #endif
 
 /** Maximum number of events in a group */
lib/pmu/rte_pmu_pmc_x86_64.h new file +24 -0
new file mode 100644
index 0000000000..7b67466960
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_X86_64_H_
+#define _RTE_PMU_PMC_X86_64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t low, high;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return low | (high << 32);
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_X86_64_H_ */
Signature · 1 line
2.34.1
#88 lore raw

PATCH v9 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               | 10 ++++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 +++++++++++++
 lib/eal/common/eal_common_trace.c        | 13 ++++-
 lib/eal/common/eal_common_trace_points.c |  5 ++
 lib/eal/include/rte_eal_trace.h          | 13 +++++
 lib/eal/meson.build                      |  3 ++
 lib/eal/version.map                      |  1 +
 lib/pmu/rte_pmu.c                        | 61 ++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 14 ++++++
 lib/pmu/version.map                      |  1 +
 11 files changed, 157 insertions(+), 1 deletion(-)
app/test/test_trace_perf.c +10 -0
index 46ae7d8074..f1929f2734 100644
@@ -114,6 +114,10 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+#ifdef RTE_EXEC_ENV_LINUX
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
+#endif
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +126,9 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+#ifdef RTE_EXEC_ENV_LINUX
+WORKER_DEFINE(READ_PMU)
+#endif
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +181,9 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+#ifdef RTE_EXEC_ENV_LINUX
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
+#endif
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index a8b501fe0c..6a53341c6b 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..a8e97ee1ec 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86-64 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='.*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace.c +12 -1
index 75162b722d..8796052d0c 100644
@@ -11,6 +11,9 @@
 #include <rte_errno.h>
 #include <rte_lcore.h>
 #include <rte_per_lcore.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_string_fns.h>
 
 #include "eal_trace.h"
@@ -71,8 +74,13 @@ eal_trace_init(void)
 		goto free_meta;
 
 	/* Apply global configurations */
-	STAILQ_FOREACH(arg, &trace.args, next)
+	STAILQ_FOREACH(arg, &trace.args, next) {
 		trace_args_apply(arg->val);
+#ifdef RTE_EXEC_ENV_LINUX
+		if (rte_pmu_init() == 0)
+			rte_pmu_add_events_by_pattern(arg->val);
+#endif
+	}
 
 	rte_trace_mode_set(trace.mode);
 
@@ -88,6 +96,9 @@ eal_trace_init(void)
 void
 eal_trace_fini(void)
 {
+#ifdef RTE_EXEC_ENV_LINUX
+	rte_pmu_fini();
+#endif
 	trace_mem_free();
 	trace_metadata_destroy();
 	eal_trace_args_free();
lib/eal/common/eal_common_trace_points.c +5 -0
index 0b0b254615..1e46ce549a 100644
@@ -75,3 +75,8 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
+#endif
lib/eal/include/rte_eal_trace.h +13 -0
index 5ef4398230..afb459b198 100644
@@ -17,6 +17,9 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +282,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(unsigned int index),
+	uint64_t val;
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+#endif
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/meson.build +3 -0
index 056beb9461..f5865dbcd9 100644
@@ -26,6 +26,9 @@ deps += ['kvargs']
 if not is_windows
     deps += ['telemetry']
 endif
+if is_linux
+    deps += ['pmu']
+endif
 if dpdk_conf.has('RTE_USE_LIBBSD')
     ext_deps += libbsd
 endif
lib/eal/version.map +1 -0
index 6523102157..2f8f66874b 100644
@@ -441,6 +441,7 @@ EXPERIMENTAL {
 	rte_thread_join;
 
 	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 	rte_thread_set_name;
 };
 
lib/pmu/rte_pmu.c +61 -0

67 lines not on this page · raw message

lib/pmu/rte_pmu.h +14 -0
index 0f7004c31c..0f6250e81f 100644
@@ -169,6 +169,20 @@ __rte_experimental
 int
 rte_pmu_add_event(const char *name);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add events matching pattern to the group of enabled events.
+ *
+ * @param pattern
+ *   Pattern e=ev1[,ev2,...] matching events, where evX is a placeholder for an event listed under
+ *   /sys/bus/event_source/devices/pmu/events.
+ */
+__rte_experimental
+int
+rte_pmu_add_events_by_pattern(const char *pattern);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice
lib/pmu/version.map +1 -0
index 50fb0f354e..20a27d085c 100644
@@ -8,6 +8,7 @@ EXPERIMENTAL {
 	per_lcore__event_group;
 	rte_pmu;
 	rte_pmu_add_event;
+	rte_pmu_add_events_by_pattern;
 	rte_pmu_fini;
 	rte_pmu_init;
 	rte_pmu_read;
Signature · 1 line
2.34.1
#89 lore raw

v10 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain advantages when application runs on isolated cores running dedicated tasks.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v10:
- check permissions before using counters
- do not use internal symbols in exported functions
- address review comments
v9:
- fix 'maybe-uninitialized' warning reported by CI
v8:
- just rebase series
v7:
- use per-lcore event group instead of global table index by lcore-id
- don't add pmu_autotest to fast tests because due to lack of suported on every arch
v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there
v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library

 MAINTAINERS                              |   5 +
 app/test/meson.build                     |   2 +
 app/test/test_pmu.c                      |  68 +++
 app/test/test_trace_perf.c               |  10 +
 doc/api/doxy-api-index.md                |   3 +-
 doc/api/doxy-api.conf.in                 |   1 +
 doc/guides/prog_guide/profile_app.rst    |  17 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 doc/guides/rel_notes/release_23_03.rst   |   7 +
 lib/eal/common/eal_common_trace.c        |  13 +-
 lib/eal/common/eal_common_trace_points.c |   5 +
 lib/eal/include/rte_eal_trace.h          |  13 +
 lib/eal/meson.build                      |   3 +
 lib/eal/version.map                      |   1 +
 lib/meson.build                          |   1 +
 lib/pmu/meson.build                      |  21 +
 lib/pmu/pmu_arm64.c                      |  94 ++++
 lib/pmu/pmu_private.h                    |  32 ++
 lib/pmu/rte_pmu.c                        | 521 +++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 232 ++++++++++
 lib/pmu/rte_pmu_pmc_arm64.h              |  30 ++
 lib/pmu/rte_pmu_pmc_x86_64.h             |  24 ++
 lib/pmu/version.map                      |  16 +
 23 files changed, 1149 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
 create mode 100644 lib/pmu/version.map

--
2.34.1

#90 lore raw

PATCH v10 1/4 lib: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   2 +
 app/test/test_pmu.c                    |  62 ++++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |  12 +
 doc/guides/rel_notes/release_23_03.rst |   7 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  32 ++
 lib/pmu/rte_pmu.c                      | 460 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 212 ++++++++++++
 lib/pmu/version.map                    |  15 +
 13 files changed, 824 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
index 3495946d0f..d37f242120 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build +2 -0
index f34d19e3c3..6b61b7fc32 100644
@@ -111,6 +111,7 @@ test_sources = files(
         'test_reciprocal_division_perf.c',
         'test_red.c',
         'test_pie.c',
+        'test_pmu.c',
         'test_reorder.c',
         'test_rib.c',
         'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [
         ['kni_autotest', false, true],
         ['kvargs_autotest', true, true],
         ['member_autotest', true, true],
+        ['pmu_autotest', true, true],
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true],
app/test/test_pmu.c new file +62 -0

62 lines not on this page · raw message

doc/api/doxy-api-index.md +2 -1
index 2deec7ea19..a8e04a195d 100644
@@ -223,7 +223,8 @@ The public API headers are grouped by topics:
   [log](@ref rte_log.h),
   [errno](@ref rte_errno.h),
   [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),
+  [pmu](@ref rte_pmu.h)
 
 - **misc**:
   [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
index e859426099..350b5a8c94 100644
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
                           @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +12 -0
index 14292d4c25..89e38cd301 100644
@@ -7,6 +7,18 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some performance monitoring unit (PMU).
+Such unit provides programmable counters that monitor specific events.
+
+Different tools gather that information, like for example perf.
+However, in some scenarios when CPU cores are isolated and run
+dedicated tasks interrupting those tasks with perf may be undesirable.
+
+In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``.
+
 
 Profiling on x86
 ----------------
doc/guides/rel_notes/release_23_03.rst +7 -0
index ab998a5357..20622efe58 100644
@@ -147,6 +147,13 @@ New Features
   * Added support to capture packets at each graph node with packet metadata and
     node name.
 
+* **Added PMU library.**
+
+  Added a new performance monitoring unit (PMU) library which allows applications
+  to perform self monitoring activities without depending on external utilities like perf.
+  After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters
+  can be stored in CTF format for further analysis.
+
 
 Removed Items
 -------------
lib/meson.build +1 -0
index 450c061d2b..8a42d45d20 100644
@@ -11,6 +11,7 @@
 libraries = [
         'kvargs', # eal depends on kvargs
         'telemetry', # basic info querying
+        'pmu',
         'eal', # everything depends on eal
         'ring',
         'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
new file mode 100644
index 0000000000..a4160b494e
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2023 Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +32 -0
new file mode 100644
index 0000000000..b9f8c1ddc8
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ *
+ * @param config
+ *   Architecture specific event configuration. Consult kernel sources for available options.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +460 -0

460 lines not on this page · raw message

lib/pmu/rte_pmu.h new file +212 -0

212 lines not on this page · raw message

lib/pmu/version.map new file +15 -0
new file mode 100644
index 0000000000..39a4f279c1
@@ -0,0 +1,15 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	__rte_pmu_enable_group;
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};

--
2.34.1

#91 lore raw

RE: [PATCH v10 1/4] lib: add generic support for reading PMU events

Quoted 100 lines

-----Original Message-----
From: Tomasz Duszynski <[email protected]>
Sent: Monday, February 13, 2023 7:32 PM
To: [email protected]; [email protected]; Tomasz Duszynski <[email protected]>
Cc: [email protected]; Ruifeng Wang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v10 1/4] lib: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   2 +
 app/test/test_pmu.c                    |  62 ++++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |  12 +
 doc/guides/rel_notes/release_23_03.rst |   7 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  32 ++
 lib/pmu/rte_pmu.c                      | 460 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 212 ++++++++++++
 lib/pmu/version.map                    |  15 +
 13 files changed, 824 insertions(+), 1 deletion(-)  create mode 100644

app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.map

MAINTAINERS +5 -0
index 3495946d0f..d37f242120 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build index f34d19e3c3..6b61b7fc32 renamed from app/test/meson.build +0 -0

100644

app/test/meson.build +2 -0
@@ -111,6 +111,7 @@ test_sources = files(
         'test_reciprocal_division_perf.c',
         'test_red.c',
         'test_pie.c',
+        'test_pmu.c',
         'test_reorder.c',
         'test_rib.c',
         'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [
         ['kni_autotest', false, true],
         ['kvargs_autotest', true, true],
         ['member_autotest', true, true],
+        ['pmu_autotest', true, true],
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true], diff --git a/app/test/test_pmu.c

b/app/test/test_pmu.c new file mode 100644 index 0000000000..a64564b5f5

app/test/test_pmu.c new file +23 -0
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include "test.h"
+
+#ifndef RTE_EXEC_ENV_LINUX
+
+static int
+test_pmu(void)
+{
+	printf("pmu_autotest only supported on Linux, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+
+#include <rte_pmu.h>
+
+static int
+test_pmu_read(void)
+{
+	const char *name = NULL;

app/test/test_pmu.c line 33

Quoted 10 lines
app/test/test_pmu.c new file +10 -0
+	int tries = 10, event;
+	uint64_t val = 0;
+
+	if (name == NULL) {
+		printf("PMU not supported on this arch\n");
+		return TEST_SKIPPED;
+	}
+
+	if (rte_pmu_init() < 0)
+		return TEST_FAILED;

Can we return TEST_SKIPPED here?
On aarch64, this feature requires kernel version >= 5.17. CI setups doesn't meet this requirement will start to report failure when running fast_tests.

Quoted 19 lines
app/test/test_pmu.c new file +19 -0
+
+	event = rte_pmu_add_event(name);
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	rte_pmu_fini();
+
+	return val ? TEST_SUCCESS : TEST_FAILED; }
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+

<snip>

#92 lore raw

RE: [PATCH v10 1/4] lib: add generic support for reading PMU events

[...]

Quoted 8 lines

app/test/test_pmu.c line 33

Quoted 3 lines
app/test/test_pmu.c new file +3 -0
+
+	if (rte_pmu_init() < 0)
+		return TEST_FAILED;

Can we return TEST_SKIPPED here?
On aarch64, this feature requires kernel version >= 5.17. CI setups doesn't meet this requirement will start to report failure when running fast_tests.

Okay. I think that's good enough for CI.

Quoted 20 lines
Quoted 19 lines
app/test/test_pmu.c new file +19 -0
+
+	event = rte_pmu_add_event(name);
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	rte_pmu_fini();
+
+	return val ? TEST_SUCCESS : TEST_FAILED; }
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+

<snip>

#93 lore raw

PATCH v10 2/4 pmu: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c         |  4 ++
 lib/pmu/meson.build         |  7 +++
 lib/pmu/pmu_arm64.c         | 94 +++++++++++++++++++++++++++++++++++++
 lib/pmu/rte_pmu.h           |  4 ++
 lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++
 5 files changed, 139 insertions(+)
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
app/test/test_pmu.c +4 -0
index a64564b5f5..7550223dc0 100644
@@ -24,6 +24,10 @@ test_pmu_read(void)
 	int tries = 10, event;
 	uint64_t val = 0;
 
+#if defined(RTE_ARCH_ARM64)
+	name = "cpu_cycles";
+#endif
+
 	if (name == NULL) {
 		printf("PMU not supported on this arch\n");
 		return TEST_SKIPPED;
lib/pmu/meson.build +7 -0
index a4160b494e..e857681137 100644
@@ -11,3 +11,10 @@ includes = [global_inc]
 
 sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
+indirect_headers += files(
+        'rte_pmu_pmc_arm64.h',
+)
+
+if dpdk_conf.has('RTE_ARCH_ARM64')
+    sources += files('pmu_arm64.c')
+endif
lib/pmu/pmu_arm64.c new file +94 -0

94 lines not on this page · raw message

lib/pmu/rte_pmu.h +4 -0
index 6b664c3336..bcc8e3b22d 100644
@@ -26,6 +26,10 @@ extern "C" {
 #include <rte_compat.h>
 #include <rte_spinlock.h>
 
+#if defined(RTE_ARCH_ARM64)
+#include "rte_pmu_pmc_arm64.h"
+#endif
+
 /** Maximum number of events in a group */
 #define MAX_NUM_GROUP_EVENTS 8
 
lib/pmu/rte_pmu_pmc_arm64.h new file +30 -0
new file mode 100644
index 0000000000..10648f0c5f
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_ARM64_H_
+#define _RTE_PMU_PMC_ARM64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_ARM64_H_ */
Signature · 1 line
2.34.1
Acked-by on v10 2/4
#94 lore raw

RE: [PATCH v10 2/4] pmu: support reading ARM PMU events in runtime

Quoted 15 lines

-----Original Message-----
From: Tomasz Duszynski <[email protected]>
Sent: Monday, February 13, 2023 7:32 PM
To: [email protected]; Tomasz Duszynski <[email protected]>; Ruifeng Wang <[email protected]>
Cc: [email protected]; Ruifeng Wang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v10 2/4] pmu: support reading ARM PMU events in runtime

Add support for reading ARM PMU events in runtime.

Quoted 10 lines
---
 app/test/test_pmu.c         |  4 ++
 lib/pmu/meson.build         |  7 +++
 lib/pmu/pmu_arm64.c         | 94 +++++++++++++++++++++++++++++++++++++
 lib/pmu/rte_pmu.h           |  4 ++
 lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++
 5 files changed, 139 insertions(+)
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h

Thanks.

#95 lore raw

PATCH v10 3/4 pmu: support reading Intel x86_64 PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel x86_64 PMU events in runtime.

---
 app/test/test_pmu.c          |  2 ++
 lib/pmu/meson.build          |  1 +
 lib/pmu/rte_pmu.h            |  2 ++
 lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++++++++++++++++++++++++
 4 files changed, 29 insertions(+)
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
app/test/test_pmu.c +2 -0
index 7550223dc0..cf6de85d88 100644
@@ -26,6 +26,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	name = "cpu_cycles";
+#elif defined(RTE_ARCH_X86_64)
+	name = "cpu-cycles";
 #endif
 
 	if (name == NULL) {
lib/pmu/meson.build +1 -0
index e857681137..5b92e5c4e3 100644
@@ -13,6 +13,7 @@ sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
 indirect_headers += files(
         'rte_pmu_pmc_arm64.h',
+        'rte_pmu_pmc_x86_64.h',
 )
 
 if dpdk_conf.has('RTE_ARCH_ARM64')
lib/pmu/rte_pmu.h +2 -0
index bcc8e3b22d..b1d1c17bc5 100644
@@ -28,6 +28,8 @@ extern "C" {
 
 #if defined(RTE_ARCH_ARM64)
 #include "rte_pmu_pmc_arm64.h"
+#elif defined(RTE_ARCH_X86_64)
+#include "rte_pmu_pmc_x86_64.h"
 #endif
 
 /** Maximum number of events in a group */
lib/pmu/rte_pmu_pmc_x86_64.h new file +24 -0
new file mode 100644
index 0000000000..7b67466960
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_X86_64_H_
+#define _RTE_PMU_PMC_X86_64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t low, high;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return low | (high << 32);
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_X86_64_H_ */
Signature · 1 line
2.34.1
#96 lore raw

PATCH v10 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               | 10 ++++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 +++++++++++++
 lib/eal/common/eal_common_trace.c        | 13 ++++-
 lib/eal/common/eal_common_trace_points.c |  5 ++
 lib/eal/include/rte_eal_trace.h          | 13 +++++
 lib/eal/meson.build                      |  3 ++
 lib/eal/version.map                      |  1 +
 lib/pmu/rte_pmu.c                        | 61 ++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 14 ++++++
 lib/pmu/version.map                      |  1 +
 11 files changed, 157 insertions(+), 1 deletion(-)
app/test/test_trace_perf.c +10 -0
index 46ae7d8074..f1929f2734 100644
@@ -114,6 +114,10 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+#ifdef RTE_EXEC_ENV_LINUX
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
+#endif
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +126,9 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+#ifdef RTE_EXEC_ENV_LINUX
+WORKER_DEFINE(READ_PMU)
+#endif
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +181,9 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+#ifdef RTE_EXEC_ENV_LINUX
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
+#endif
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index 89e38cd301..c4dfe85c3b 100644
@@ -19,6 +19,11 @@ dedicated tasks interrupting those tasks with perf may be undesirable.
 
 In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 
 Profiling on x86
 ----------------
doc/guides/prog_guide/trace_lib.rst +32 -0
index 3e0ea5835c..9c81936e35 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86-64 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance monitoring unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='.*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace.c +12 -1
index 75162b722d..8796052d0c 100644
@@ -11,6 +11,9 @@
 #include <rte_errno.h>
 #include <rte_lcore.h>
 #include <rte_per_lcore.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_string_fns.h>
 
 #include "eal_trace.h"
@@ -71,8 +74,13 @@ eal_trace_init(void)
 		goto free_meta;
 
 	/* Apply global configurations */
-	STAILQ_FOREACH(arg, &trace.args, next)
+	STAILQ_FOREACH(arg, &trace.args, next) {
 		trace_args_apply(arg->val);
+#ifdef RTE_EXEC_ENV_LINUX
+		if (rte_pmu_init() == 0)
+			rte_pmu_add_events_by_pattern(arg->val);
+#endif
+	}
 
 	rte_trace_mode_set(trace.mode);
 
@@ -88,6 +96,9 @@ eal_trace_init(void)
 void
 eal_trace_fini(void)
 {
+#ifdef RTE_EXEC_ENV_LINUX
+	rte_pmu_fini();
+#endif
 	trace_mem_free();
 	trace_metadata_destroy();
 	eal_trace_args_free();
lib/eal/common/eal_common_trace_points.c +5 -0
index 051f89809c..9d6faa19ed 100644
@@ -77,3 +77,8 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
+#endif
lib/eal/include/rte_eal_trace.h +13 -0
index 6f5c022558..c7da83c480 100644
@@ -17,6 +17,9 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -285,6 +288,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(unsigned int index),
+	uint64_t val;
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+#endif
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/meson.build +3 -0
index 056beb9461..f5865dbcd9 100644
@@ -26,6 +26,9 @@ deps += ['kvargs']
 if not is_windows
     deps += ['telemetry']
 endif
+if is_linux
+    deps += ['pmu']
+endif
 if dpdk_conf.has('RTE_USE_LIBBSD')
     ext_deps += libbsd
 endif
lib/eal/version.map +1 -0
index 2ae57ee78a..01e7a099d2 100644
@@ -441,6 +441,7 @@ EXPERIMENTAL {
 	rte_thread_join;
 
 	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 	rte_lcore_register_usage_cb;
 	rte_thread_create_control;
 	rte_thread_set_name;
lib/pmu/rte_pmu.c +61 -0

67 lines not on this page · raw message

lib/pmu/rte_pmu.h +14 -0
index b1d1c17bc5..e1c3bb5e56 100644
@@ -176,6 +176,20 @@ __rte_experimental
 int
 rte_pmu_add_event(const char *name);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add events matching pattern to the group of enabled events.
+ *
+ * @param pattern
+ *   Pattern e=ev1[,ev2,...] matching events, where evX is a placeholder for an event listed under
+ *   /sys/bus/event_source/devices/pmu/events.
+ */
+__rte_experimental
+int
+rte_pmu_add_events_by_pattern(const char *pattern);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice
lib/pmu/version.map +1 -0
index 39a4f279c1..e16b3ff009 100644
@@ -9,6 +9,7 @@ EXPERIMENTAL {
 	per_lcore__event_group;
 	rte_pmu;
 	rte_pmu_add_event;
+	rte_pmu_add_events_by_pattern;
 	rte_pmu_fini;
 	rte_pmu_init;
 	rte_pmu_read;
Signature · 1 line
2.34.1
#97 lore raw

v11 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain advantages when application runs on isolated cores running dedicated tasks.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v11:
- skip fast test in case init fails
v10:
- check permissions before using counters
- do not use internal symbols in exported functions
- address review comments
v9:
- fix 'maybe-uninitialized' warning reported by CI
v8:
- just rebase series
v7:
- use per-lcore event group instead of global table index by lcore-id
- don't add pmu_autotest to fast tests because due to lack of suported on every arch
v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there
v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library

 MAINTAINERS                              |   5 +
 app/test/meson.build                     |   2 +
 app/test/test_pmu.c                      |  68 +++
 app/test/test_trace_perf.c               |  10 +
 doc/api/doxy-api-index.md                |   3 +-
 doc/api/doxy-api.conf.in                 |   1 +
 doc/guides/prog_guide/profile_app.rst    |  17 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 doc/guides/rel_notes/release_23_03.rst   |   7 +
 lib/eal/common/eal_common_trace.c        |  13 +-
 lib/eal/common/eal_common_trace_points.c |   5 +
 lib/eal/include/rte_eal_trace.h          |  13 +
 lib/eal/meson.build                      |   3 +
 lib/eal/version.map                      |   1 +
 lib/meson.build                          |   1 +
 lib/pmu/meson.build                      |  21 +
 lib/pmu/pmu_arm64.c                      |  94 ++++
 lib/pmu/pmu_private.h                    |  32 ++
 lib/pmu/rte_pmu.c                        | 521 +++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 232 ++++++++++
 lib/pmu/rte_pmu_pmc_arm64.h              |  30 ++
 lib/pmu/rte_pmu_pmc_x86_64.h             |  24 ++
 lib/pmu/version.map                      |  16 +
 23 files changed, 1149 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
 create mode 100644 lib/pmu/version.map

--
2.34.1

#98 lore raw

PATCH v11 1/4 lib: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   2 +
 app/test/test_pmu.c                    |  62 ++++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |  12 +
 doc/guides/rel_notes/release_23_03.rst |   7 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  32 ++
 lib/pmu/rte_pmu.c                      | 460 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 212 ++++++++++++
 lib/pmu/version.map                    |  15 +
 13 files changed, 824 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
index 3495946d0f..d37f242120 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build +2 -0
index f34d19e3c3..6b61b7fc32 100644
@@ -111,6 +111,7 @@ test_sources = files(
         'test_reciprocal_division_perf.c',
         'test_red.c',
         'test_pie.c',
+        'test_pmu.c',
         'test_reorder.c',
         'test_rib.c',
         'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [
         ['kni_autotest', false, true],
         ['kvargs_autotest', true, true],
         ['member_autotest', true, true],
+        ['pmu_autotest', true, true],
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true],
app/test/test_pmu.c new file +62 -0

62 lines not on this page · raw message

doc/api/doxy-api-index.md +2 -1
index 2deec7ea19..a8e04a195d 100644
@@ -223,7 +223,8 @@ The public API headers are grouped by topics:
   [log](@ref rte_log.h),
   [errno](@ref rte_errno.h),
   [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),
+  [pmu](@ref rte_pmu.h)
 
 - **misc**:
   [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
index e859426099..350b5a8c94 100644
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
                           @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +12 -0
index 14292d4c25..89e38cd301 100644
@@ -7,6 +7,18 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some performance monitoring unit (PMU).
+Such unit provides programmable counters that monitor specific events.
+
+Different tools gather that information, like for example perf.
+However, in some scenarios when CPU cores are isolated and run
+dedicated tasks interrupting those tasks with perf may be undesirable.
+
+In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``.
+
 
 Profiling on x86
 ----------------
doc/guides/rel_notes/release_23_03.rst +7 -0
index ab998a5357..20622efe58 100644
@@ -147,6 +147,13 @@ New Features
   * Added support to capture packets at each graph node with packet metadata and
     node name.
 
+* **Added PMU library.**
+
+  Added a new performance monitoring unit (PMU) library which allows applications
+  to perform self monitoring activities without depending on external utilities like perf.
+  After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters
+  can be stored in CTF format for further analysis.
+
 
 Removed Items
 -------------
lib/meson.build +1 -0
index 450c061d2b..8a42d45d20 100644
@@ -11,6 +11,7 @@
 libraries = [
         'kvargs', # eal depends on kvargs
         'telemetry', # basic info querying
+        'pmu',
         'eal', # everything depends on eal
         'ring',
         'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
new file mode 100644
index 0000000000..a4160b494e
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2023 Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +32 -0
new file mode 100644
index 0000000000..b9f8c1ddc8
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ *
+ * @param config
+ *   Architecture specific event configuration. Consult kernel sources for available options.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +460 -0

460 lines not on this page · raw message

lib/pmu/rte_pmu.h new file +212 -0

212 lines not on this page · raw message

lib/pmu/version.map new file +15 -0
new file mode 100644
index 0000000000..39a4f279c1
@@ -0,0 +1,15 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	__rte_pmu_enable_group;
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
Signature · 1 line
2.34.1
#99 lore raw

Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

16/02/2023 17:54, Tomasz Duszynski пишет:

Quoted 292 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
  MAINTAINERS                            |   5 +
  app/test/meson.build                   |   2 +
  app/test/test_pmu.c                    |  62 ++++
  doc/api/doxy-api-index.md              |   3 +-
  doc/api/doxy-api.conf.in               |   1 +
  doc/guides/prog_guide/profile_app.rst  |  12 +
  doc/guides/rel_notes/release_23_03.rst |   7 +
  lib/meson.build                        |   1 +
  lib/pmu/meson.build                    |  13 +
  lib/pmu/pmu_private.h                  |  32 ++
  lib/pmu/rte_pmu.c                      | 460 +++++++++++++++++++++++++
  lib/pmu/rte_pmu.h                      | 212 ++++++++++++
  lib/pmu/version.map                    |  15 +
  13 files changed, 824 insertions(+), 1 deletion(-)

create mode 100644 app/test/test_pmu.c
create mode 100644 lib/pmu/meson.build
create mode 100644 lib/pmu/pmu_private.h
create mode 100644 lib/pmu/rte_pmu.c
create mode 100644 lib/pmu/rte_pmu.h
create mode 100644 lib/pmu/version.map

MAINTAINERS +5 -0
index 3495946d0f..d37f242120 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
  M: Pavan Nikhilesh <[email protected]>
  F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
  Test Applications
  -----------------
app/test/meson.build +2 -0
index f34d19e3c3..6b61b7fc32 100644
@@ -111,6 +111,7 @@ test_sources = files(
          'test_reciprocal_division_perf.c',
          'test_red.c',
          'test_pie.c',
+        'test_pmu.c',
          'test_reorder.c',
          'test_rib.c',
          'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [
          ['kni_autotest', false, true],
          ['kvargs_autotest', true, true],
          ['member_autotest', true, true],
+        ['pmu_autotest', true, true],
          ['power_cpufreq_autotest', false, true],
          ['power_autotest', true, true],
          ['power_kvm_vm_autotest', false, true],
app/test/test_pmu.c new file +62 -0
new file mode 100644
index 0000000000..c257638e8b
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include "test.h"
+
+#ifndef RTE_EXEC_ENV_LINUX
+
+static int
+test_pmu(void)
+{
+	printf("pmu_autotest only supported on Linux, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+
+#include <rte_pmu.h>
+
+static int
+test_pmu_read(void)
+{
+	const char *name = NULL;
+	int tries = 10, event;
+	uint64_t val = 0;
+
+	if (name == NULL) {
+		printf("PMU not supported on this arch\n");
+		return TEST_SKIPPED;
+	}
+
+	if (rte_pmu_init() < 0)
+		return TEST_SKIPPED;
+
+	event = rte_pmu_add_event(name);
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	rte_pmu_fini();
+
+	return val ? TEST_SUCCESS : TEST_FAILED;
+}
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_pmu(void)
+{
+	return unit_test_suite_runner(&pmu_tests);
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/api/doxy-api-index.md +2 -1
index 2deec7ea19..a8e04a195d 100644
@@ -223,7 +223,8 @@ The public API headers are grouped by topics:
    [log](@ref rte_log.h),
    [errno](@ref rte_errno.h),
    [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),
+  [pmu](@ref rte_pmu.h)
 
  - **misc**:
    [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
index e859426099..350b5a8c94 100644
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                            @TOPDIR@/lib/pci \
                            @TOPDIR@/lib/pdump \
                            @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                            @TOPDIR@/lib/port \
                            @TOPDIR@/lib/power \
                            @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +12 -0
index 14292d4c25..89e38cd301 100644
@@ -7,6 +7,18 @@ Profile Your Application
  The following sections describe methods of profiling DPDK applications on
  different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some performance monitoring unit (PMU).
+Such unit provides programmable counters that monitor specific events.
+
+Different tools gather that information, like for example perf.
+However, in some scenarios when CPU cores are isolated and run
+dedicated tasks interrupting those tasks with perf may be undesirable.
+
+In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``.
+
 
  Profiling on x86
  ----------------
doc/guides/rel_notes/release_23_03.rst +7 -0
index ab998a5357..20622efe58 100644
@@ -147,6 +147,13 @@ New Features
    * Added support to capture packets at each graph node with packet metadata and
      node name.
 
+* **Added PMU library.**
+
+  Added a new performance monitoring unit (PMU) library which allows applications
+  to perform self monitoring activities without depending on external utilities like perf.
+  After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters
+  can be stored in CTF format for further analysis.
+
 
  Removed Items
  -------------
lib/meson.build +1 -0
index 450c061d2b..8a42d45d20 100644
@@ -11,6 +11,7 @@
  libraries = [
          'kvargs', # eal depends on kvargs
          'telemetry', # basic info querying
+        'pmu',
          'eal', # everything depends on eal
          'ring',
          'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
new file mode 100644
index 0000000000..a4160b494e
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2023 Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +32 -0
new file mode 100644
index 0000000000..b9f8c1ddc8
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ *
+ * @param config
+ *   Architecture specific event configuration. Consult kernel sources for available options.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +15 -0
new file mode 100644
index 0000000000..950f999cb7
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>

lib/pmu/rte_pmu.c line 25

Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"

I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?

lib/pmu/rte_pmu.c line 31

Quoted 6 lines
lib/pmu/rte_pmu.c new file +6 -0
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h)))))
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;

Do we really need struct declaration here?

Quoted 520 lines
lib/pmu/rte_pmu.c new file +428 -0
+/*
+ * Following __rte_weak functions provide default no-op. Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3])
+{
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask)
+{
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3])
+{
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3])
+{
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd)
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0);
+}
+
+static int
+open_events(struct rte_pmu_event_group *group)
+{
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group)
+{
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group)
+{
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV;
+}
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event)
+{
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file +86 -0
new file mode 100644
index 0000000000..6b664c3336
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_compat.h>
+#include <rte_spinlock.h>
+
+/** Maximum number of events in a group */
+#define MAX_NUM_GROUP_EVENTS 8
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */
+} __rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */
+};
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */
+};
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version */
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */

lib/pmu/rte_pmu.h line 96

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

Quoted 137 lines
lib/pmu/rte_pmu.h new file +116 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/pmu/version.map new file +15 -0
new file mode 100644
index 0000000000..39a4f279c1
@@ -0,0 +1,15 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	__rte_pmu_enable_group;
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
#100 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 308 lines

-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Friday, February 17, 2023 12:51 AM
To: [email protected]
Subject: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

External Email

----------------------------------------------------------------------
16/02/2023 17:54, Tomasz Duszynski пишет:

Quoted 168 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
  MAINTAINERS                            |   5 +
  app/test/meson.build                   |   2 +
  app/test/test_pmu.c                    |  62 ++++
  doc/api/doxy-api-index.md              |   3 +-
  doc/api/doxy-api.conf.in               |   1 +
  doc/guides/prog_guide/profile_app.rst  |  12 +
  doc/guides/rel_notes/release_23_03.rst |   7 +
  lib/meson.build                        |   1 +
  lib/pmu/meson.build                    |  13 +
  lib/pmu/pmu_private.h                  |  32 ++
  lib/pmu/rte_pmu.c                      | 460 +++++++++++++++++++++++++
  lib/pmu/rte_pmu.h                      | 212 ++++++++++++
  lib/pmu/version.map                    |  15 +
  13 files changed, 824 insertions(+), 1 deletion(-)

create mode 100644 app/test/test_pmu.c
create mode 100644 lib/pmu/meson.build
create mode 100644 lib/pmu/pmu_private.h
create mode 100644 lib/pmu/rte_pmu.c
create mode 100644 lib/pmu/rte_pmu.h
create mode 100644 lib/pmu/version.map

MAINTAINERS index 3495946d0f..d37f242120 renamed from MAINTAINERS +0 -0

100644

MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
  M: Pavan Nikhilesh <[email protected]>
  F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
  Test Applications
  -----------------
app/test/meson.build index renamed from app/test/meson.build +0 -0

f34d19e3c3..6b61b7fc32 100644

app/test/meson.build +2 -0
@@ -111,6 +111,7 @@ test_sources = files(
          'test_reciprocal_division_perf.c',
          'test_red.c',
          'test_pie.c',
+        'test_pmu.c',
          'test_reorder.c',
          'test_rib.c',
          'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [
          ['kni_autotest', false, true],
          ['kvargs_autotest', true, true],
          ['member_autotest', true, true],
+        ['pmu_autotest', true, true],
          ['power_cpufreq_autotest', false, true],
          ['power_autotest', true, true],
          ['power_kvm_vm_autotest', false, true], diff --git

a/app/test/test_pmu.c b/app/test/test_pmu.c new file mode 100644 index 0000000000..c257638e8b

app/test/test_pmu.c new file +61 -0
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include "test.h"
+
+#ifndef RTE_EXEC_ENV_LINUX
+
+static int
+test_pmu(void)
+{
+	printf("pmu_autotest only supported on Linux, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+
+#include <rte_pmu.h>
+
+static int
+test_pmu_read(void)
+{
+	const char *name = NULL;
+	int tries = 10, event;
+	uint64_t val = 0;
+
+	if (name == NULL) {
+		printf("PMU not supported on this arch\n");
+		return TEST_SKIPPED;
+	}
+
+	if (rte_pmu_init() < 0)
+		return TEST_SKIPPED;
+
+	event = rte_pmu_add_event(name);
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	rte_pmu_fini();
+
+	return val ? TEST_SUCCESS : TEST_FAILED; }
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_pmu(void)
+{
+	return unit_test_suite_runner(&pmu_tests);
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/api/doxy-api-index.md +1 -1
index 2deec7ea19..a8e04a195d 100644
@@ -223,7 +223,8 @@ The public API headers are grouped by topics:
    [log](@ref rte_log.h),
    [errno](@ref rte_errno.h),
    [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),  [pmu](@ref rte_pmu.h)
 
  - **misc**:
    [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0 -0

e859426099..350b5a8c94 100644

doc/api/doxy-api.conf.in +1 -0
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                            @TOPDIR@/lib/pci \
                            @TOPDIR@/lib/pdump \
                            @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                            @TOPDIR@/lib/port \
                            @TOPDIR@/lib/power \
                            @TOPDIR@/lib/rawdev \ diff --git

a/doc/guides/prog_guide/profile_app.rst
b/doc/guides/prog_guide/profile_app.rst
index 14292d4c25..89e38cd301 100644

doc/guides/prog_guide/profile_app.rst +1 -0
@@ -7,6 +7,18 @@ Profile Your Application
  The following sections describe methods of profiling DPDK applications on
  different architectures.
 
+Performance counter based profiling

doc/guides/prog_guide/profile_app.rst Profile Your Application · line 20

Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +10 -0
+-----------------------------------
+
+Majority of architectures support some performance monitoring unit (PMU).
+Such unit provides programmable counters that monitor specific events.
+
+Different tools gather that information, like for example perf.
+However, in some scenarios when CPU cores are isolated and run
+dedicated tasks interrupting those tasks with perf may be undesirable.
+
+In such cases, an application can use the PMU library to read such events via

``rte_pmu_read()``.

Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +1 -0
+
 
  Profiling on x86
  ----------------
doc/guides/rel_notes/release_23_03.rst +0 -0
index ab998a5357..20622efe58 100644

doc/guides/rel_notes/release_23_03.rst New Features · line 153

Quoted 7 lines
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -147,6 +147,13 @@ New Features
    * Added support to capture packets at each graph node with packet metadata and
      node name.
 
+* **Added PMU library.**
+
+  Added a new performance monitoring unit (PMU) library which allows
+ applications  to perform self monitoring activities without depending on external utilities

like perf.

Quoted 99 lines
doc/guides/rel_notes/release_23_03.rst +3 -0
+  After integration with :doc:`../prog_guide/trace_lib` data gathered
+ from hardware counters  can be stored in CTF format for further analysis.
+
 
  Removed Items
  -------------
lib/meson.build index renamed from lib/meson.build +0 -0

450c061d2b..8a42d45d20 100644

lib/meson.build +1 -0
@@ -11,6 +11,7 @@
  libraries = [
          'kvargs', # eal depends on kvargs
          'telemetry', # basic info querying
+        'pmu',
          'eal', # everything depends on eal
          'ring',
          'rcu', # rcu depends on ring diff --git

a/lib/pmu/meson.build b/lib/pmu/meson.build new file mode 100644 index 0000000000..a4160b494e

lib/pmu/meson.build new file +13 -0
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(C) 2023 Marvell
+International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file renamed from lib/pmu/pmu_private.h +0 -0

mode 100644 index 0000000000..b9f8c1ddc8

lib/pmu/pmu_private.h new file +32 -0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ *
+ * @param config
+ *   Architecture specific event configuration. Consult kernel sources for available options.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..950f999cb7

lib/pmu/rte_pmu.c new file +21 -0
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
Quoted 9 lines

lib/pmu/rte_pmu.c line 25

Quoted 4 lines
lib/pmu/rte_pmu.c new file +4 -0
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"

I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?

There are not any plans to support that elsewhere currently so flat directory structure is good enough.

Quoted 10 lines

lib/pmu/rte_pmu.c line 31

Quoted 7 lines
lib/pmu/rte_pmu.c new file +7 -0
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) <<
+(__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;

Do we really need struct declaration here?

What’s the problem with this placement precisely?

Quoted 512 lines
Quoted 512 lines
lib/pmu/rte_pmu.c new file +418 -0
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { }
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0); }
+
+static int
+open_events(struct rte_pmu_event_group *group) {
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group) {
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group) {
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV;
+}
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event) {
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +88 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_compat.h>
+#include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version
+*/ #ifndef rte_pmu_pmc_read #define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
Quoted 9 lines

lib/pmu/rte_pmu.h line 96

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Quoted 136 lines
Quoted 136 lines
lib/pmu/rte_pmu.h new file +116 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/pmu/version.map new file mode renamed from lib/pmu/version.map +0 -0

100644 index 0000000000..39a4f279c1

lib/pmu/version.map new file +15 -0
@@ -0,0 +1,15 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	__rte_pmu_enable_group;
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
#101 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 320 lines
Quoted 296 lines
Quoted 168 lines

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
  MAINTAINERS                            |   5 +
  app/test/meson.build                   |   2 +
  app/test/test_pmu.c                    |  62 ++++
  doc/api/doxy-api-index.md              |   3 +-
  doc/api/doxy-api.conf.in               |   1 +
  doc/guides/prog_guide/profile_app.rst  |  12 +
  doc/guides/rel_notes/release_23_03.rst |   7 +
  lib/meson.build                        |   1 +
  lib/pmu/meson.build                    |  13 +
  lib/pmu/pmu_private.h                  |  32 ++
  lib/pmu/rte_pmu.c                      | 460 +++++++++++++++++++++++++
  lib/pmu/rte_pmu.h                      | 212 ++++++++++++
  lib/pmu/version.map                    |  15 +
  13 files changed, 824 insertions(+), 1 deletion(-)

create mode 100644 app/test/test_pmu.c
create mode 100644 lib/pmu/meson.build
create mode 100644 lib/pmu/pmu_private.h
create mode 100644 lib/pmu/rte_pmu.c
create mode 100644 lib/pmu/rte_pmu.h
create mode 100644 lib/pmu/version.map

MAINTAINERS index 3495946d0f..d37f242120 renamed from MAINTAINERS +0 -0

100644

MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
  M: Pavan Nikhilesh <[email protected]>
  F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
  Test Applications
  -----------------
app/test/meson.build index renamed from app/test/meson.build +0 -0

f34d19e3c3..6b61b7fc32 100644

app/test/meson.build +2 -0
@@ -111,6 +111,7 @@ test_sources = files(
          'test_reciprocal_division_perf.c',
          'test_red.c',
          'test_pie.c',
+        'test_pmu.c',
          'test_reorder.c',
          'test_rib.c',
          'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [
          ['kni_autotest', false, true],
          ['kvargs_autotest', true, true],
          ['member_autotest', true, true],
+        ['pmu_autotest', true, true],
          ['power_cpufreq_autotest', false, true],
          ['power_autotest', true, true],
          ['power_kvm_vm_autotest', false, true], diff --git

a/app/test/test_pmu.c b/app/test/test_pmu.c new file mode 100644 index 0000000000..c257638e8b

app/test/test_pmu.c new file +61 -0
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include "test.h"
+
+#ifndef RTE_EXEC_ENV_LINUX
+
+static int
+test_pmu(void)
+{
+	printf("pmu_autotest only supported on Linux, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+
+#include <rte_pmu.h>
+
+static int
+test_pmu_read(void)
+{
+	const char *name = NULL;
+	int tries = 10, event;
+	uint64_t val = 0;
+
+	if (name == NULL) {
+		printf("PMU not supported on this arch\n");
+		return TEST_SKIPPED;
+	}
+
+	if (rte_pmu_init() < 0)
+		return TEST_SKIPPED;
+
+	event = rte_pmu_add_event(name);
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	rte_pmu_fini();
+
+	return val ? TEST_SUCCESS : TEST_FAILED; }
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_pmu(void)
+{
+	return unit_test_suite_runner(&pmu_tests);
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/api/doxy-api-index.md +1 -1
index 2deec7ea19..a8e04a195d 100644
@@ -223,7 +223,8 @@ The public API headers are grouped by topics:
    [log](@ref rte_log.h),
    [errno](@ref rte_errno.h),
    [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),  [pmu](@ref rte_pmu.h)
 
  - **misc**:
    [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0 -0

e859426099..350b5a8c94 100644

doc/api/doxy-api.conf.in +1 -0
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                            @TOPDIR@/lib/pci \
                            @TOPDIR@/lib/pdump \
                            @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                            @TOPDIR@/lib/port \
                            @TOPDIR@/lib/power \
                            @TOPDIR@/lib/rawdev \ diff --git

a/doc/guides/prog_guide/profile_app.rst
b/doc/guides/prog_guide/profile_app.rst
index 14292d4c25..89e38cd301 100644

doc/guides/prog_guide/profile_app.rst +1 -0
@@ -7,6 +7,18 @@ Profile Your Application
  The following sections describe methods of profiling DPDK applications on
  different architectures.
 
+Performance counter based profiling

doc/guides/prog_guide/profile_app.rst Profile Your Application · line 20

Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +10 -0
+-----------------------------------
+
+Majority of architectures support some performance monitoring unit (PMU).
+Such unit provides programmable counters that monitor specific events.
+
+Different tools gather that information, like for example perf.
+However, in some scenarios when CPU cores are isolated and run
+dedicated tasks interrupting those tasks with perf may be undesirable.
+
+In such cases, an application can use the PMU library to read such events via

``rte_pmu_read()``.

Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +1 -0
+
 
  Profiling on x86
  ----------------
doc/guides/rel_notes/release_23_03.rst +0 -0
index ab998a5357..20622efe58 100644

doc/guides/rel_notes/release_23_03.rst New Features · line 153

Quoted 7 lines
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -147,6 +147,13 @@ New Features
    * Added support to capture packets at each graph node with packet metadata and
      node name.
 
+* **Added PMU library.**
+
+  Added a new performance monitoring unit (PMU) library which allows
+ applications  to perform self monitoring activities without depending on external utilities

like perf.

Quoted 99 lines
doc/guides/rel_notes/release_23_03.rst +3 -0
+  After integration with :doc:`../prog_guide/trace_lib` data gathered
+ from hardware counters  can be stored in CTF format for further analysis.
+
 
  Removed Items
  -------------
lib/meson.build index renamed from lib/meson.build +0 -0

450c061d2b..8a42d45d20 100644

lib/meson.build +1 -0
@@ -11,6 +11,7 @@
  libraries = [
          'kvargs', # eal depends on kvargs
          'telemetry', # basic info querying
+        'pmu',
          'eal', # everything depends on eal
          'ring',
          'rcu', # rcu depends on ring diff --git

a/lib/pmu/meson.build b/lib/pmu/meson.build new file mode 100644 index 0000000000..a4160b494e

lib/pmu/meson.build new file +13 -0
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(C) 2023 Marvell
+International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file renamed from lib/pmu/pmu_private.h +0 -0

mode 100644 index 0000000000..b9f8c1ddc8

lib/pmu/pmu_private.h new file +32 -0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ *
+ * @param config
+ *   Architecture specific event configuration. Consult kernel sources for available options.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..950f999cb7

lib/pmu/rte_pmu.c new file +21 -0
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
Quoted 9 lines

lib/pmu/rte_pmu.c line 25

Quoted 4 lines
lib/pmu/rte_pmu.c new file +4 -0
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"

I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?

There are not any plans to support that elsewhere currently so flat directory structure is good enough.

Quoted 10 lines

lib/pmu/rte_pmu.c line 31

Quoted 7 lines
lib/pmu/rte_pmu.c new file +7 -0
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) <<
+(__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;

Do we really need struct declaration here?

Quoted 1 line

What’s the problem with this placement precisely?

Not a big deal, but It seems excessive for me.
As I understand you do have include just above for the whole .h that contains definition of that struct anyway.

Quoted 523 lines
Quoted 513 lines
Quoted 513 lines
lib/pmu/rte_pmu.c new file +418 -0
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { }
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0); }
+
+static int
+open_events(struct rte_pmu_event_group *group) {
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group) {
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group) {
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV;
+}
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event) {
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +88 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_compat.h>
+#include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version
+*/ #ifndef rte_pmu_pmc_read #define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
Quoted 9 lines

lib/pmu/rte_pmu.h line 96

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

Quoted 3 lines

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

Quoted 137 lines
Quoted 137 lines
Quoted 137 lines
lib/pmu/rte_pmu.h new file +116 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/pmu/version.map new file mode renamed from lib/pmu/version.map +0 -0

100644 index 0000000000..39a4f279c1

lib/pmu/version.map new file +15 -0
@@ -0,0 +1,15 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	__rte_pmu_enable_group;
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
#102 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 864 lines

-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Friday, February 17, 2023 11:15 AM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 321 lines
Quoted 297 lines
Quoted 168 lines

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
  MAINTAINERS                            |   5 +
  app/test/meson.build                   |   2 +
  app/test/test_pmu.c                    |  62 ++++
  doc/api/doxy-api-index.md              |   3 +-
  doc/api/doxy-api.conf.in               |   1 +
  doc/guides/prog_guide/profile_app.rst  |  12 +
  doc/guides/rel_notes/release_23_03.rst |   7 +
  lib/meson.build                        |   1 +
  lib/pmu/meson.build                    |  13 +
  lib/pmu/pmu_private.h                  |  32 ++
  lib/pmu/rte_pmu.c                      | 460 +++++++++++++++++++++++++
  lib/pmu/rte_pmu.h                      | 212 ++++++++++++
  lib/pmu/version.map                    |  15 +
  13 files changed, 824 insertions(+), 1 deletion(-)

create mode 100644 app/test/test_pmu.c
create mode 100644 lib/pmu/meson.build
create mode 100644 lib/pmu/pmu_private.h
create mode 100644 lib/pmu/rte_pmu.c
create mode 100644 lib/pmu/rte_pmu.h
create mode 100644 lib/pmu/version.map

MAINTAINERS index 3495946d0f..d37f242120 renamed from MAINTAINERS +0 -0

100644

MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
  M: Pavan Nikhilesh <[email protected]>
  F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
  Test Applications
  -----------------
app/test/meson.build index renamed from app/test/meson.build +0 -0

f34d19e3c3..6b61b7fc32 100644

app/test/meson.build +2 -0
@@ -111,6 +111,7 @@ test_sources = files(
          'test_reciprocal_division_perf.c',
          'test_red.c',
          'test_pie.c',
+        'test_pmu.c',
          'test_reorder.c',
          'test_rib.c',
          'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [
          ['kni_autotest', false, true],
          ['kvargs_autotest', true, true],
          ['member_autotest', true, true],
+        ['pmu_autotest', true, true],
          ['power_cpufreq_autotest', false, true],
          ['power_autotest', true, true],
          ['power_kvm_vm_autotest', false, true], diff --git

a/app/test/test_pmu.c b/app/test/test_pmu.c new file mode 100644 index 0000000000..c257638e8b

app/test/test_pmu.c new file +61 -0
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include "test.h"
+
+#ifndef RTE_EXEC_ENV_LINUX
+
+static int
+test_pmu(void)
+{
+	printf("pmu_autotest only supported on Linux, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+
+#include <rte_pmu.h>
+
+static int
+test_pmu_read(void)
+{
+	const char *name = NULL;
+	int tries = 10, event;
+	uint64_t val = 0;
+
+	if (name == NULL) {
+		printf("PMU not supported on this arch\n");
+		return TEST_SKIPPED;
+	}
+
+	if (rte_pmu_init() < 0)
+		return TEST_SKIPPED;
+
+	event = rte_pmu_add_event(name);
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	rte_pmu_fini();
+
+	return val ? TEST_SUCCESS : TEST_FAILED; }
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_pmu(void)
+{
+	return unit_test_suite_runner(&pmu_tests);
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/api/doxy-api-index.md +1 -1
index 2deec7ea19..a8e04a195d 100644
@@ -223,7 +223,8 @@ The public API headers are grouped by topics:
    [log](@ref rte_log.h),
    [errno](@ref rte_errno.h),
    [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),  [pmu](@ref rte_pmu.h)
 
  - **misc**:
    [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +0 -0

index
e859426099..350b5a8c94 100644

doc/api/doxy-api.conf.in +1 -0
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                            @TOPDIR@/lib/pci \
                            @TOPDIR@/lib/pdump \
                            @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                            @TOPDIR@/lib/port \
                            @TOPDIR@/lib/power \
                            @TOPDIR@/lib/rawdev \ diff --git

a/doc/guides/prog_guide/profile_app.rst
b/doc/guides/prog_guide/profile_app.rst
index 14292d4c25..89e38cd301 100644

doc/guides/prog_guide/profile_app.rst +2 -0
@@ -7,6 +7,18 @@ Profile Your Application
  The following sections describe methods of profiling DPDK applications on
  different architectures.
 
+Performance counter based profiling
+-----------------------------------

doc/guides/prog_guide/profile_app.rst Profile Your Application · line 21

Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +10 -0
+
+Majority of architectures support some performance monitoring unit (PMU).
+Such unit provides programmable counters that monitor specific events.
+
+Different tools gather that information, like for example perf.
+However, in some scenarios when CPU cores are isolated and run
+dedicated tasks interrupting those tasks with perf may be undesirable.
+
+In such cases, an application can use the PMU library to read such
+events via

``rte_pmu_read()``.

Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +1 -0
+
 
  Profiling on x86
  ----------------
doc/guides/rel_notes/release_23_03.rst +0 -0
index ab998a5357..20622efe58 100644

doc/guides/rel_notes/release_23_03.rst New Features · line 154

Quoted 8 lines
doc/guides/rel_notes/release_23_03.rst +5 -0
@@ -147,6 +147,13 @@ New Features
    * Added support to capture packets at each graph node with packet metadata and
      node name.
 
+* **Added PMU library.**
+
+  Added a new performance monitoring unit (PMU) library which
+ allows applications  to perform self monitoring activities
+ without depending on external utilities

like perf.

Quoted 99 lines
doc/guides/rel_notes/release_23_03.rst +3 -0
+  After integration with :doc:`../prog_guide/trace_lib` data
+ gathered from hardware counters  can be stored in CTF format for further analysis.
+
 
  Removed Items
  -------------
lib/meson.build index renamed from lib/meson.build +0 -0

450c061d2b..8a42d45d20 100644

lib/meson.build +1 -0
@@ -11,6 +11,7 @@
  libraries = [
          'kvargs', # eal depends on kvargs
          'telemetry', # basic info querying
+        'pmu',
          'eal', # everything depends on eal
          'ring',
          'rcu', # rcu depends on ring diff --git

a/lib/pmu/meson.build b/lib/pmu/meson.build new file mode 100644 index 0000000000..a4160b494e

lib/pmu/meson.build new file +13 -0
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(C) 2023
+Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file renamed from lib/pmu/pmu_private.h +0 -0

mode 100644 index 0000000000..b9f8c1ddc8

lib/pmu/pmu_private.h new file +32 -0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ *
+ * @param config
+ *   Architecture specific event configuration. Consult kernel sources for available options.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..950f999cb7

lib/pmu/rte_pmu.c new file +21 -0
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
Quoted 9 lines

lib/pmu/rte_pmu.c line 25

Quoted 4 lines
lib/pmu/rte_pmu.c new file +4 -0
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"

I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?

There are not any plans to support that elsewhere currently so flat directory structure is good enough.

Quoted 10 lines

lib/pmu/rte_pmu.c line 31

Quoted 7 lines
lib/pmu/rte_pmu.c new file +7 -0
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) <<
+(__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;

Do we really need struct declaration here?

Quoted 1 line

What’s the problem with this placement precisely?

Not a big deal, but It seems excessive for me.
As I understand you do have include just above for the whole .h that contains definition of that struct anyway.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
lib/pmu/rte_pmu.c new file +22 -0
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3])
+{ }
+
+static int
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name,

name);

Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
lib/pmu/rte_pmu.c new file +62 -0
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,

name);

Quoted 231 lines
Quoted 231 lines
Quoted 231 lines
lib/pmu/rte_pmu.c new file +231 -0
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0); }
+
+static int
+open_events(struct rte_pmu_event_group *group) {
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group) {
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group) {
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE,
+PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV; }
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event) {
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name) {
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,

name);

Quoted 111 lines
Quoted 111 lines
Quoted 111 lines
lib/pmu/rte_pmu.c new file +79 -0
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +27 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU
+and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages

*/

Quoted 62 lines
Quoted 61 lines
Quoted 46 lines
lib/pmu/rte_pmu.h new file +46 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own
+version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.

lib/pmu/rte_pmu.h line 96

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+ */
+static __rte_always_inline uint64_t __rte_pmu_read_userpage(struct
+perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

Quoted 8 lines
Quoted 3 lines

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

This number is just an indicator whether kernel did change something or not.
If cpu reordering will come into play then this will not change anything from pov of this loop. All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.

Quoted 137 lines
Quoted 137 lines
Quoted 137 lines
Quoted 137 lines
lib/pmu/rte_pmu.h new file +115 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(unsigned int
+index) {
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/pmu/version.map new file renamed from lib/pmu/version.map +0 -0

mode
100644 index 0000000000..39a4f279c1

lib/pmu/version.map new file +15 -0
@@ -0,0 +1,15 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	__rte_pmu_enable_group;
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
#103 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 107 lines
Quoted 107 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +27 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU
+and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages

*/

Quoted 65 lines
Quoted 52 lines
Quoted 52 lines
lib/pmu/rte_pmu.h new file +52 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own
+version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t __rte_pmu_read_userpage(struct
+perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
Quoted 9 lines

lib/pmu/rte_pmu.h line 96

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Quoted 6 lines
Quoted 4 lines

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

This number is just an indicator whether kernel did change something or not.

You are talking about pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual for perf_event_open() also has a code snippet with compiler barrier only...

Quoted 3 lines

If cpu reordering will come into play then this will not change anything from pov of this loop. All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.

Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:

      cpu #1 kernel vent_update_userpage:
      pc->lock++; // pc->lock==1
      pc->index=I2;
      pc->offset=O2;
      ...
      pc->lock++; //pc->lock==2
T3:
      cpu #0 continue with read pmu:
      seq=pc->lock; //seq == 2
       offset=pc->offset; // offset == O2
       ....
       pmc = rte_pmu_pmc_read(index - 1);  // Note that we read at I1, not I2
       offset += pmc; //offset == O2 + pmcread(I1-1);       
       if (pc->lock == seq) // they are equal, return
             return offset;

Or, it can happen, but by some reason we don't care much?

Quoted 82 lines
Quoted 82 lines
Quoted 82 lines
Quoted 82 lines
Quoted 82 lines
lib/pmu/rte_pmu.h new file +81 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 185

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */

Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header? As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(unsigned int
+index) {
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
#104 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 157 lines

-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Monday, February 20, 2023 3:31 PM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 109 lines
Quoted 109 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +28 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup
+PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+MAX_NUM_GROUP_EVENTS 8
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS];
+/**< array of user pages

*/

Quoted 66 lines
Quoted 53 lines
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group,
+_event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own
+version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
Quoted 9 lines

lib/pmu/rte_pmu.h line 96

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Quoted 6 lines
Quoted 4 lines

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

This number is just an indicator whether kernel did change something or not.

You are talking about pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual for perf_event_open() also has a code snippet with compiler barrier only...

Quoted 3 lines

If cpu reordering will come into play then this will not change anything from pov of this loop. All we want is fresh data when needed and no involvement of compiler
when it comes to reordering code.

Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:

     cpu #1 kernel vent_update_userpage:
     pc->lock++; // pc->lock==1
     pc->index=I2;
     pc->offset=O2;
     ...
     pc->lock++; //pc->lock==2
T3:
     cpu #0 continue with read pmu:
     seq=pc->lock; //seq == 2
      offset=pc->offset; // offset == O2
      ....
      pmc = rte_pmu_pmc_read(index - 1);  // Note that we read at I1, not I2
      offset += pmc; //offset == O2 + pmcread(I1-1);
      if (pc->lock == seq) // they are equal, return
            return offset;
Quoted 2 lines

Or, it can happen, but by some reason we don't care much?

This code does self-monitoring and user page (whole group actually) is per thread running on current cpu. Hence I am not sure what are you trying to prove with that example.

Quoted 122 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
lib/pmu/rte_pmu.h new file +48 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being

read.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25 -0
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 185

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */

Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(unsigned int
+index) {
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
#105 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 149 lines
Quoted 149 lines
Quoted 110 lines
Quoted 110 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +28 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup
+PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 36

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+MAX_NUM_GROUP_EVENTS 8
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS];
+/**< array of user pages

*/

Quoted 66 lines
Quoted 53 lines
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group,
+_event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own
+version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
Quoted 9 lines

lib/pmu/rte_pmu.h line 93

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Quoted 6 lines
Quoted 4 lines

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

This number is just an indicator whether kernel did change something or not.

You are talking about pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual for perf_event_open() also has a code snippet with compiler barrier only...

Quoted 3 lines

If cpu reordering will come into play then this will not change anything from pov of this loop. All we want is fresh data when needed and no involvement of compiler
when it comes to reordering code.

Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:

     cpu #1 kernel vent_update_userpage:
     pc->lock++; // pc->lock==1
     pc->index=I2;
     pc->offset=O2;
     ...
     pc->lock++; //pc->lock==2
T3:
     cpu #0 continue with read pmu:
     seq=pc->lock; //seq == 2
      offset=pc->offset; // offset == O2
      ....
      pmc = rte_pmu_pmc_read(index - 1);  // Note that we read at I1, not I2
      offset += pmc; //offset == O2 + pmcread(I1-1);
      if (pc->lock == seq) // they are equal, return
            return offset;
Quoted 5 lines
Quoted 2 lines

Or, it can happen, but by some reason we don't care much?

This code does self-monitoring and user page (whole group actually) is per thread running on current cpu. Hence I am not sure what are you trying to prove with that example.

I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why?
My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them.
So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?

Also there was another question below, which you probably missed, so I copied it here:
Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

Quoted 123 lines
Quoted 123 lines
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
lib/pmu/rte_pmu.h new file +48 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 149

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being

read.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25 -0
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 182

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */

Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(unsigned int
+index) {
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
#106 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 167 lines

-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Monday, February 20, 2023 6:21 PM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 115 lines
Quoted 115 lines
Quoted 115 lines
Quoted 114 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
lib/pmu/rte_pmu.h new file renamed from lib/pmu/rte_pmu.h +0 -0

mode
100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +28 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell  */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to
+setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 36

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page
+*mmap_pages[MAX_NUM_GROUP_EVENTS];
+/**< array of user pages

*/

Quoted 63 lines
Quoted 53 lines
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group,
+_event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its
+own version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
Quoted 9 lines

lib/pmu/rte_pmu.h line 93

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

Quoted 3 lines

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

This number is just an indicator whether kernel did change something or not.

You are talking about pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual for perf_event_open() also has a code snippet with compiler barrier only...

Quoted 1 line

If cpu reordering will come into play then this will not change anything from pov of this

loop.

Quoted 26 lines
Quoted 26 lines
Quoted 2 lines

All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.

Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:

     cpu #1 kernel vent_update_userpage:
     pc->lock++; // pc->lock==1
     pc->index=I2;
     pc->offset=O2;
     ...
     pc->lock++; //pc->lock==2
T3:
     cpu #0 continue with read pmu:
     seq=pc->lock; //seq == 2
      offset=pc->offset; // offset == O2
      ....
      pmc = rte_pmu_pmc_read(index - 1);  // Note that we read at I1, not I2
      offset += pmc; //offset == O2 + pmcread(I1-1);
      if (pc->lock == seq) // they are equal, return
            return offset;
Quoted 5 lines
Quoted 2 lines

Or, it can happen, but by some reason we don't care much?

This code does self-monitoring and user page (whole group actually) is
per thread running on current cpu. Hence I am not sure what are you trying to prove with that

example.

Quoted 8 lines

I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why?
My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them.
So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?

Look. I'll reiterate that.

1. That user page/group/PMU config is per process. Other processes do not access that.
All this happens on the very same CPU where current thread is running.
2. Suppose you've already read seq. Now for some reason kernel updates data in page seq was read from.
3. Kernel will enter critical section during update. seq changes along with other data without app knowing about it. If you want nitty gritty details consult kernel sources.
4. app resumes and has some stale data but *WILL* read new seq. Code loops again because values do not match.
5. Otherwise seq values match and data is valid.

Quoted 6 lines

Also there was another question below, which you probably missed, so I copied it here:
Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

These functions need to be inlined otherwise performance takes a hit.

Quoted 125 lines
Quoted 125 lines
Quoted 125 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
lib/pmu/rte_pmu.h new file +49 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 150

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU
+counters are no longer being

read.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25 -0
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 183

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */

Another question - do we really need to have
__rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header? As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
lib/pmu/rte_pmu.h new file +26 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(unsigned
+int
+index) {
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
#107 lore raw

Re: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 167 lines
Quoted 158 lines
Quoted 116 lines
Quoted 116 lines
Quoted 116 lines
Quoted 115 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
lib/pmu/rte_pmu.h new file renamed from lib/pmu/rte_pmu.h +0 -0

mode
100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +28 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell  */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to
+setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 36

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page
+*mmap_pages[MAX_NUM_GROUP_EVENTS];
+/**< array of user pages

*/

Quoted 63 lines
Quoted 53 lines
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group,
+_event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its
+own version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
Quoted 9 lines

lib/pmu/rte_pmu.h line 93

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

Quoted 3 lines

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

This number is just an indicator whether kernel did change something or not.

You are talking about pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual for perf_event_open() also has a code snippet with compiler barrier only...

Quoted 1 line

If cpu reordering will come into play then this will not change anything from pov of this

loop.

Quoted 26 lines
Quoted 26 lines
Quoted 2 lines

All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.

Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:

      cpu #1 kernel vent_update_userpage:
      pc->lock++; // pc->lock==1
      pc->index=I2;
      pc->offset=O2;
      ...
      pc->lock++; //pc->lock==2
T3:
      cpu #0 continue with read pmu:
      seq=pc->lock; //seq == 2
       offset=pc->offset; // offset == O2
       ....
       pmc = rte_pmu_pmc_read(index - 1);  // Note that we read at I1, not I2
       offset += pmc; //offset == O2 + pmcread(I1-1);
       if (pc->lock == seq) // they are equal, return
             return offset;
Quoted 5 lines
Quoted 2 lines

Or, it can happen, but by some reason we don't care much?

This code does self-monitoring and user page (whole group actually) is
per thread running on current cpu. Hence I am not sure what are you trying to prove with that

example.

Quoted 8 lines

I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why?
My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them.
So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?

Quoted 3 lines

Look. I'll reiterate that.

1. That user page/group/PMU config is per process. Other processes do not access that.

Ok, that's clear.

Quoted 1 line

All this happens on the very same CPU where current thread is running.

Ok... but can't this page be updated by kernel thread running simultaneously on different CPU?

Quoted 3 lines

2. Suppose you've already read seq. Now for some reason kernel updates data in page seq was read from.
3. Kernel will enter critical section during update. seq changes along with other data without app knowing about it. If you want nitty gritty details consult kernel sources.

Look, I don't have to beg you to answer these questions.
In fact, I expect library author to document all such narrow things clearly either in in PG, or in source code comments (ideally in both). If not, then from my perspective the patch is not ready stage and shouldn't be accepted.
I don't know is compiler-barrier is enough here or not, but I think it is definitely worth a clear explanation in the docs. I suppose it wouldn't be only me who will get confused here.
So please take an effort and document it clearly why you believe there is no race-condition.

Quoted 1 line

4. app resumes and has some stale data but *WILL* read new seq. Code loops again because values do not match.

If the kernel will always execute update for this page in the same thread context, then yes, - user code will always note the difference after resume.
But why it can't happen that your user-thread reads this page on one CPU, while some kernel code on other CPU updates it simultaneously?

Quoted 10 lines

5. Otherwise seq values match and data is valid.

Quoted 6 lines

Also there was another question below, which you probably missed, so I copied it here:
Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

These functions need to be inlined otherwise performance takes a hit.

I understand that perfomance might be affected, but how big is hit?
I expect actual PMU read will not be free anyway, right?
If the diff is small, might be it is worth to go for such change,
removing unneeded structures from public headers would help a lot in future in terms of ABI/API stability.

Quoted 126 lines
Quoted 125 lines
Quoted 125 lines
Quoted 125 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
lib/pmu/rte_pmu.h new file +49 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 150

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU
+counters are no longer being

read.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25 -0
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 183

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */

Another question - do we really need to have
__rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header? As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
lib/pmu/rte_pmu.h new file +26 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(unsigned
+int
+index) {
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
#108 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 180 lines

-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Tuesday, February 21, 2023 1:48 AM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: Re: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 170 lines
Quoted 159 lines
Quoted 117 lines
Quoted 117 lines
Quoted 116 lines
Quoted 106 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
lib/pmu/rte_pmu.h new file renamed from lib/pmu/rte_pmu.h +0 -0

mode
100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +28 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell  */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup
+PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 36

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page
+*mmap_pages[MAX_NUM_GROUP_EVENTS];
+/**< array of user pages

*/

Quoted 63 lines
Quoted 62 lines
Quoted 47 lines
lib/pmu/rte_pmu.h new file +47 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group,
+_event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own
+version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.

lib/pmu/rte_pmu.h line 93

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

Quoted 8 lines
Quoted 3 lines

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

This number is just an indicator whether kernel did change something or not.

Quoted 7 lines
Quoted 7 lines

You are talking about pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual for perf_event_open() also has a code snippet with compiler barrier only...

Quoted 2 lines

If cpu reordering will come into play then this will not change anything from pov of this

loop.

Quoted 26 lines
Quoted 26 lines
Quoted 2 lines

All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.

Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:

      cpu #1 kernel vent_update_userpage:
      pc->lock++; // pc->lock==1
      pc->index=I2;
      pc->offset=O2;
      ...
      pc->lock++; //pc->lock==2
T3:
      cpu #0 continue with read pmu:
      seq=pc->lock; //seq == 2
       offset=pc->offset; // offset == O2
       ....
       pmc = rte_pmu_pmc_read(index - 1);  // Note that we read at I1, not I2
       offset += pmc; //offset == O2 + pmcread(I1-1);
       if (pc->lock == seq) // they are equal, return
             return offset;
Quoted 6 lines
Quoted 2 lines

Or, it can happen, but by some reason we don't care much?

This code does self-monitoring and user page (whole group actually) is per thread running on current cpu. Hence I am not sure what are you trying to prove with that

example.

Quoted 8 lines

I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why? My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them. So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?

Look. I'll reiterate that.

Quoted 9 lines
Quoted 1 line

1. That user page/group/PMU config is per process. Other processes do not access that.

Ok, that's clear.

Quoted 1 line

All this happens on the very same CPU where current thread is running.

Ok... but can't this page be updated by kernel thread running simultaneously on different CPU?

I already pointed out that event/counter configuration is bound to current cpu. How can possibly other cpu update that configuration? This cannot work.

If you think that there's some problem with the code (or is simply broken on your setup) and logic has obvious flaw and you can provide meaningful evidence of that then I'd be more than happy to apply that fix. Otherwise that discussion will get us nowhere.

Quoted 173 lines
Quoted 2 lines

2. Suppose you've already read seq. Now for some reason kernel updates data in page seq was read

from.

Quoted 1 line

3. Kernel will enter critical section during update. seq changes along with other data without

app knowing about it.

Quoted 1 line

If you want nitty gritty details consult kernel sources.

Look, I don't have to beg you to answer these questions.
In fact, I expect library author to document all such narrow things clearly either in in PG, or in source code comments (ideally in both). If not, then from my perspective the patch is not ready stage and shouldn't be accepted.
I don't know is compiler-barrier is enough here or not, but I think it is definitely worth a clear explanation in the docs.
I suppose it wouldn't be only me who will get confused here. So please take an effort and document it clearly why you believe there is no race-condition.

Quoted 1 line

4. app resumes and has some stale data but *WILL* read new seq. Code loops again because values

do not match.

If the kernel will always execute update for this page in the same thread context, then yes, - user code will always note the difference after resume.
But why it can't happen that your user-thread reads this page on one CPU, while some kernel code on other CPU updates it simultaneously?

Quoted 10 lines

5. Otherwise seq values match and data is valid.

Quoted 6 lines

Also there was another question below, which you probably missed, so I copied it here:
Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

These functions need to be inlined otherwise performance takes a hit.

I understand that perfomance might be affected, but how big is hit? I expect actual PMU read will not be free anyway, right? If the diff is small, might be it is worth to go for such change, removing unneeded structures from public headers would help a lot in future in terms of ABI/API stability.

Quoted 126 lines
Quoted 125 lines
Quoted 125 lines
Quoted 125 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
lib/pmu/rte_pmu.h new file +49 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 150

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU
+counters are no longer being

read.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25 -0
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 183

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */

Another question - do we really need to have
__rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header? As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
lib/pmu/rte_pmu.h new file +26 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(unsigned
+int
+index) {
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
#109 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 177 lines
Quoted 177 lines
Quoted 170 lines
Quoted 161 lines
Quoted 119 lines
Quoted 119 lines
Quoted 118 lines
Quoted 108 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
lib/pmu/rte_pmu.h new file renamed from lib/pmu/rte_pmu.h +0 -0

mode
100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +28 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell  */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup
+PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 36

Quoted 8 lines
lib/pmu/rte_pmu.h new file +8 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page
+*mmap_pages[MAX_NUM_GROUP_EVENTS];
+/**< array of user pages

*/

Quoted 63 lines
Quoted 62 lines
Quoted 47 lines
lib/pmu/rte_pmu.h new file +47 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group,
+_event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own
+version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.

lib/pmu/rte_pmu.h line 93

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?

Quoted 8 lines
Quoted 3 lines

It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.

Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?

This number is just an indicator whether kernel did change something or not.

Quoted 7 lines
Quoted 7 lines

You are talking about pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual for perf_event_open() also has a code snippet with compiler barrier only...

Quoted 2 lines

If cpu reordering will come into play then this will not change anything from pov of this

loop.

Quoted 26 lines
Quoted 26 lines
Quoted 2 lines

All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.

Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:

      cpu #1 kernel vent_update_userpage:
      pc->lock++; // pc->lock==1
      pc->index=I2;
      pc->offset=O2;
      ...
      pc->lock++; //pc->lock==2
T3:
      cpu #0 continue with read pmu:
      seq=pc->lock; //seq == 2
       offset=pc->offset; // offset == O2
       ....
       pmc = rte_pmu_pmc_read(index - 1);  // Note that we read at I1, not I2
       offset += pmc; //offset == O2 + pmcread(I1-1);
       if (pc->lock == seq) // they are equal, return
             return offset;
Quoted 6 lines
Quoted 2 lines

Or, it can happen, but by some reason we don't care much?

This code does self-monitoring and user page (whole group actually) is per thread running on current cpu. Hence I am not sure what are you trying to prove with that

example.

Quoted 8 lines

I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why? My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them. So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?

Quoted 3 lines

Look. I'll reiterate that.

1. That user page/group/PMU config is per process. Other processes do not access that.

Ok, that's clear.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

All this happens on the very same CPU where current thread is running.

Ok... but can't this page be updated by kernel thread running simultaneously on different CPU?

I already pointed out that event/counter configuration is bound to current cpu. How can possibly other cpu update that configuration? This cannot work.

Can you elaborate a bit what you mean with 'event/counter configuration is bound to current cpu'? If that means it could be updated only by code running on given, CPU - yes it is clear. But can this page be read by user-space from different CPU?
Or you just assume that your user-space thread will *always* be bounded just to one particular CPU and would never switch?

Quoted 6 lines

If you think that there's some problem with the code (or is simply broken on your setup) and logic[] a bit has obvious flaw and you can provide meaningful evidence of that then I'd be more than happy to apply that fix. Otherwise that discussion will get us nowhere.

Yes, we are going in cycles here.
I keep asking you same questions about library function internals, you keep refusing to explain things to me insisting that it is 'way too obvious'.
Well, sorry but it is not obvious to me.
So I still insist that a clearly documented internal design and expected usage is required for that patch before it can be accepted.

Konstantin

#110 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

In reply to Tomasz Duszynski
Quoted 37 lines
Quoted 27 lines
Quoted 27 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..950f999cb7

lib/pmu/rte_pmu.c new file +21 -0
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
Quoted 9 lines

lib/pmu/rte_pmu.c line 25

Quoted 4 lines
lib/pmu/rte_pmu.c new file +4 -0
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"

I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?

Quoted 2 lines

There are not any plans to support that elsewhere currently so flat directory structure is good enough.

Ok, I suppose then best choice is to ask freebsd and windows maintainers. Guys, any thoughts here?
Thanks
Konstantin

#111 lore raw

Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

In reply to Tomasz Duszynski
Quoted 10 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Few more comments/questions below.

Quoted 287 lines
lib/pmu/rte_pmu.c new file +281 -0
new file mode 100644
index 0000000000..950f999cb7
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h)))))
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op. Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3])
+{
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask)
+{
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3])
+{
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3])
+{
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd)
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0);
+}
+
+static int
+open_events(struct rte_pmu_event_group *group)
+{
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group)
+{
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group)
+{
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;

lib/pmu/rte_pmu.c line 291

Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Can memory from its TLS block get re-used (by other thread or for other purposes)?

Quoted 195 lines
lib/pmu/rte_pmu.c new file +169 -0
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV;
+}
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event)
+{
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file +20 -0
new file mode 100644
index 0000000000..6b664c3336
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+

lib/pmu/rte_pmu.h line 30

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_compat.h>
+#include <rte_spinlock.h>
+
+/** Maximum number of events in a group */
+#define MAX_NUM_GROUP_EVENTS 8

forgot RTE_ prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically by enable_group()?

lib/pmu/rte_pmu.h line 41

Quoted 11 lines
lib/pmu/rte_pmu.h new file +11 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */
+} __rte_cache_aligned;
+

Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea), why these two struct below (rte_pmu_event and rte_pmu) have to be public?
I think both can be safely moved away from public headers.

Quoted 52 lines
lib/pmu/rte_pmu.h new file +51 -0
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */
+};
 
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */
+};
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version */
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;

lib/pmu/rte_pmu.h line 102

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {

In mmap_events() you return EPERM if cap_user_rdpmc is not enabled.
Do you need another check here? Or this capability can be disabled by kernel at run-time?

Quoted 12 lines
lib/pmu/rte_pmu.h new file +12 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+

lib/pmu/rte_pmu.h line 124

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.

__rte_internal ?

Quoted 5 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

lib/pmu/rte_pmu.h new file +1 -0
+__rte_experimental

lib/pmu/rte_pmu.h line 139

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.

Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?

Quoted 4 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

Probably worth to mention that this function is not MT safe.
Same for _fini_ and add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.

Quoted 90 lines
lib/pmu/rte_pmu.h new file +69 -0
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/pmu/version.map new file +15 -0
new file mode 100644
index 0000000000..39a4f279c1
@@ -0,0 +1,15 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	__rte_pmu_enable_group;
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
#112 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 308 lines

-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Tuesday, February 21, 2023 3:17 AM
To: [email protected]
Subject: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 9 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Few more comments/questions below.

Quoted 283 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..950f999cb7

lib/pmu/rte_pmu.c new file +278 -0
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) <<
+(__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { }
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0); }
+
+static int
+open_events(struct rte_pmu_event_group *group) {
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group) {
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group) {
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
Quoted 8 lines

lib/pmu/rte_pmu.c line 291

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Nothing special. Any pointers to that thread-local in that thread are invalided.

Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for other purposes)?

Why would any other thread reuse that? Eventually main thread will need that data to do the cleanup.

Quoted 202 lines
Quoted 202 lines
lib/pmu/rte_pmu.c new file +169 -0
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV;
+}
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event)
+{
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file +26 -0
new file mode 100644
index 0000000000..6b664c3336
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_compat.h>
Quoted 9 lines

lib/pmu/rte_pmu.h line 30

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+#include <rte_spinlock.h>
+
+/** Maximum number of events in a group */
+#define MAX_NUM_GROUP_EVENTS 8

forgot RTE_ prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically by enable_group()?

8 is reasonable number I think. X86/ARM have actually less that that (was that something like 4?). Moreover events are scheduled as a group so there must be enough hw counters available for that to succeed. So this number should cover current needs.

Quoted 6 lines
Quoted 6 lines
lib/pmu/rte_pmu.h new file +6 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */
Quoted 8 lines

lib/pmu/rte_pmu.h line 41

Quoted 5 lines
lib/pmu/rte_pmu.h new file +5 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */
+} __rte_cache_aligned;
+

Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea),

We want to save as much cpu cycles as we possibly can and inlining does helps in that matter.

Quoted 3 lines

why these two struct below (rte_pmu_event and rte_pmu) have to be public? I think both can be safely moved away from public headers.

struct rte_pmu_event can be hidden I guess.
struct rte_pmu is used in this header hence cannot be moved elsewhere.

Quoted 59 lines
Quoted 59 lines
lib/pmu/rte_pmu.h new file +57 -0
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */
+};
 
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */
+};
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version */
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
Quoted 9 lines

lib/pmu/rte_pmu.h line 102

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {

In mmap_events() you return EPERM if cap_user_rdpmc is not enabled. Do you need another check here? Or this capability can be disabled by kernel at run-time?

That extra check in mmap_event() may be removed actually. Some archs allow disabling reading rdpmc (I think that on x86 one can do that) so this check needs to stay.

Quoted 17 lines
Quoted 17 lines
lib/pmu/rte_pmu.h new file +16 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
Quoted 9 lines
Quoted 6 lines

+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.

__rte_internal ?

No this cannot be internal because that will make functions calling it internal as well hence apps won't be able to use that. This has already been brought up by one of the reviewers.

Quoted 10 lines
Quoted 10 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

lib/pmu/rte_pmu.h new file +6 -0
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
Quoted 9 lines
Quoted 5 lines

+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.

Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?

This is inlined and has one caller i.e rte_pmu_read().

Quoted 10 lines
Quoted 4 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

Probably worth to mention that this function is not MT safe. Same for _fini_ and add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.

Yes they are meant to be called from main thread.

Quoted 90 lines
Quoted 90 lines
lib/pmu/rte_pmu.h new file +69 -0
+__rte_experimental
+int
+rte_pmu_init(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Finalize PMU library. This should be called after PMU counters are no longer being read.
+ */
+__rte_experimental
+void
+rte_pmu_fini(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (unlikely(!rte_pmu.initialized))
+		return 0;
+
+	if (unlikely(!group->enabled)) {
+		ret = __rte_pmu_enable_group();
+		if (ret)
+			return 0;
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return __rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/pmu/version.map new file +15 -0
new file mode 100644
index 0000000000..39a4f279c1
@@ -0,0 +1,15 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	__rte_pmu_enable_group;
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
#113 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 307 lines
Quoted 298 lines
Quoted 11 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Few more comments/questions below.

Quoted 283 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..950f999cb7

lib/pmu/rte_pmu.c new file +278 -0
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) <<
+(__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { }
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0); }
+
+static int
+open_events(struct rte_pmu_event_group *group) {
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group) {
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group) {
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
Quoted 8 lines

lib/pmu/rte_pmu.c line 291

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Quoted 7 lines

Nothing special. Any pointers to that thread-local in that thread are invalided.

Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for other purposes)?

Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.

I understand that main thread would need to access that data.
I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while program continues to run. As I understand address of its RTE_PER_LCORE(_event_group) will still remain in rte_pmu.event_group_list, even if it is probably not valid any more.

Quoted 212 lines
Quoted 202 lines
Quoted 202 lines
lib/pmu/rte_pmu.c new file +169 -0
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV;
+}
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event)
+{
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file +26 -0
new file mode 100644
index 0000000000..6b664c3336
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_compat.h>
Quoted 9 lines

lib/pmu/rte_pmu.h line 30

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+#include <rte_spinlock.h>
+
+/** Maximum number of events in a group */
+#define MAX_NUM_GROUP_EVENTS 8

forgot RTE_ prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically by enable_group()?

Quoted 3 lines

8 is reasonable number I think. X86/ARM have actually less that that (was that something like 4?). Moreover events are scheduled as a group so there must be enough hw counters available for that to succeed. So this number should cover current needs.

If you think 8 will be enough to cover all possible future cases - I am ok either way. Still need RTE_ prefix for it.

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
lib/pmu/rte_pmu.h new file +6 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */
Quoted 8 lines

lib/pmu/rte_pmu.h line 41

Quoted 5 lines
lib/pmu/rte_pmu.h new file +5 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */
+} __rte_cache_aligned;
+

Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea),

Quoted 2 lines

We want to save as much cpu cycles as we possibly can and inlining does helps in that matter.

Ok, so asking same question from different thread: how many cycles it will save? What is the difference in terms of performance when you have this function inlined vs not inlined?

Quoted 6 lines
Quoted 3 lines

why these two struct below (rte_pmu_event and rte_pmu) have to be public? I think both can be safely moved away from public headers.

struct rte_pmu_event can be hidden I guess.
struct rte_pmu is used in this header hence cannot be moved elsewhere.

Not sure why?
Is that because you use it inside rte_pmu_read()?
But that check I think can be safely moved into __rte_pmu_enable_group() or probably even into rte_pmu_add_event().

Quoted 99 lines
Quoted 59 lines
Quoted 59 lines
lib/pmu/rte_pmu.h new file +57 -0
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */
+};
 
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */
+};
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version */
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t
+__rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
Quoted 9 lines

lib/pmu/rte_pmu.h line 102

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {

In mmap_events() you return EPERM if cap_user_rdpmc is not enabled. Do you need another check here? Or this capability can be disabled by kernel at run-time?

That extra check in mmap_event() may be removed actually. Some archs allow disabling reading rdpmc (I think that on x86 one can do that) so this check needs to stay.

Quoted 17 lines
Quoted 17 lines
lib/pmu/rte_pmu.h new file +16 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
Quoted 9 lines
Quoted 6 lines

+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.

__rte_internal ?

Quoted 3 lines

No this cannot be internal because that will make functions calling it internal as well hence apps won't be able to use that. This has already been brought up by one of the reviewers.

Ok, then we probably can mark it with ' @internal' tag in formal comments?

Quoted 21 lines
Quoted 11 lines
Quoted 11 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

lib/pmu/rte_pmu.h new file +6 -0
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
Quoted 9 lines
Quoted 5 lines

+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.

Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?

Quoted 1 line

This is inlined and has one caller i.e rte_pmu_read().

I thought we are talking here about rte_pmu_init().
I don't see where it is inlined and still not clear why it can't be called directly.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

Quoted 7 lines
Quoted 5 lines

Probably worth to mention that this function is not MT safe. Same for _fini_ and add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.

Yes they are meant to be called from main thread.

Ok, then please add that into formal API comments.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+__rte_experimental
+int
+rte_pmu_init(void);
+
#114 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 13 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 27 February 2023 21.53

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

[...]

Quoted 42 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -

1) {

Quoted 5 lines
Quoted 5 lines

lib/pmu/rte_pmu.c

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) ==

-1) {

Quoted 11 lines
Quoted 9 lines

lib/pmu/rte_pmu.c line 291

Quoted 6 lines
lib/pmu/rte_pmu.c new file +6 -0
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Nothing special. Any pointers to that thread-local in that thread are

invalided.

Quoted 2 lines
Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for other

purposes)?

Quoted 2 lines
Quoted 1 line
Quoted 7 lines
Quoted 2 lines

Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.

I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while program continues to run.

Is the example you describe here (i.e. a thread terminating in the middle of doing something) really a scenario DPDK is supposed to support?

Quoted 3 lines

As I understand address of its RTE_PER_LCORE(_event_group) will still remain in rte_pmu.event_group_list,
even if it is probably not valid any more.

There should be a "destructor/done/finish" function available to remove this from the list.

[...]

Quoted 11 lines
Quoted 4 lines
Quoted 2 lines

Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea),

We want to save as much cpu cycles as we possibly can and inlining does

helps

Quoted 1 line

in that matter.

Ok, so asking same question from different thread: how many cycles it will save?
What is the difference in terms of performance when you have this function inlined vs not inlined?

We expect to use this in our in-house profiler library. For this reason, I have a very strong preference for absolute maximum performance.

Reading PMU events is for performance profiling, so I expect other potential users of the PMU library to share my opinion on this.

#115 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 59 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

[...]

Quoted 45 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -

1) {

Quoted 5 lines
Quoted 5 lines

lib/pmu/rte_pmu.c

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) ==

-1) {

Quoted 11 lines
Quoted 9 lines

lib/pmu/rte_pmu.c line 291

Quoted 6 lines
lib/pmu/rte_pmu.c new file +6 -0
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Nothing special. Any pointers to that thread-local in that thread are

invalided.

Quoted 2 lines
Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for other

purposes)?

Quoted 4 lines
Quoted 1 line

Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.

Quoted 7 lines
Quoted 4 lines

I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while program continues to run.

Is the example you describe here (i.e. a thread terminating in the middle of doing something) really a scenario DPDK is supposed to support?

I am not talking about some abnormal termination.
We do have ability to spawn control threads, user can spawn his own thread, all these threads can have limited life-time.
Not to mention about rte_thread_register()/rte_thread_unregister().

Quoted 20 lines
Quoted 3 lines

As I understand address of its RTE_PER_LCORE(_event_group) will still remain in rte_pmu.event_group_list,
even if it is probably not valid any more.

There should be a "destructor/done/finish" function available to remove this from the list.

[...]

Quoted 11 lines
Quoted 4 lines
Quoted 2 lines

Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea),

We want to save as much cpu cycles as we possibly can and inlining does

helps

Quoted 1 line

in that matter.

Ok, so asking same question from different thread: how many cycles it will save?
What is the difference in terms of performance when you have this function inlined vs not inlined?

Quoted 4 lines

We expect to use this in our in-house profiler library. For this reason, I have a very strong preference for absolute maximum performance.

Reading PMU events is for performance profiling, so I expect other potential users of the PMU library to share my opinion on this.

Well, from my perspective 14 cycles are not that much... Though yes, it would be good to hear more opinions here.

#116 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 68 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Tuesday, 28 February 2023 13.05

Quoted 23 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

[...]

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET,

PERF_IOC_FLAG_GROUP) == -

Quoted 6 lines
Quoted 6 lines

1) {

Quoted 5 lines
Quoted 5 lines

lib/pmu/rte_pmu.c

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE,

PERF_IOC_FLAG_GROUP) ==

Quoted 26 lines
Quoted 25 lines

-1) {

Quoted 11 lines
Quoted 9 lines

lib/pmu/rte_pmu.c line 291

Quoted 6 lines
lib/pmu/rte_pmu.c new file +6 -0
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Nothing special. Any pointers to that thread-local in that thread are

invalided.

Quoted 2 lines
Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for other

purposes)?

Quoted 4 lines
Quoted 1 line

Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.

I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while program continues to run.

Quoted 5 lines
Quoted 1 line

Is the example you describe here (i.e. a thread terminating in the middle of

doing something) really a scenario DPDK is supposed to

Quoted 1 line

support?

I am not talking about some abnormal termination.

Then I misunderstood your example; I thought you meant the tread was terminated while inside the rte_pmu_read() function.

Quoted 5 lines

We do have ability to spawn control threads, user can spawn his own thread, all these
threads can have limited life-time.
Not to mention about rte_thread_register()/rte_thread_unregister().

I agree that normal thread termination should be supported.

Quoted 22 lines
Quoted 1 line
Quoted 1 line

As I understand address of its RTE_PER_LCORE(_event_group) will still

remain

Quoted 4 lines
Quoted 2 lines

in rte_pmu.event_group_list,
even if it is probably not valid any more.

There should be a "destructor/done/finish" function available to remove this

from the list.

Quoted 15 lines

[...]

Quoted 11 lines
Quoted 4 lines
Quoted 2 lines

Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea),

We want to save as much cpu cycles as we possibly can and inlining does

helps

Quoted 1 line

in that matter.

Ok, so asking same question from different thread: how many cycles it will save?
What is the difference in terms of performance when you have this function inlined vs not inlined?

Quoted 8 lines
Quoted 1 line

We expect to use this in our in-house profiler library. For this reason, I

have a very strong preference for absolute maximum

Quoted 3 lines

performance.

Reading PMU events is for performance profiling, so I expect other potential

users of the PMU library to share my opinion on this.

Well, from my perspective 14 cycles are not that much...

For reference, the i40e testpmd per-core performance report shows that it uses 36 cycles per packet.

This is a total of 1152 cycles per burst of 32 packets. 14 cycles overhead per burst / 1152 cycles per burst = 1.2 % overhead.

But that is not all: If the application's pipeline has three stages, where the PMU counters are read for each stage, the per-invocation overhead of 14 cycles adds up, and the overhead per burst is now 3 * 14 / 1152 = 3.6 %.

Generalizing...

In my example here, the same function with 14 wasted cycles is called three times. It might as well be three individual libraries each wasting 14 cycles in its individual fast path processing function, due to a similarly relaxed attitude regarding wasting 14 cycles.

My point is:

Real applications do much more work than testpmd, so all this "insignificant" extra overhead in the libraries adds up!

Generally, I would like the DPDK Project to remain loyal to its original philosophy, where performance is considered a Key Performance Indicator, and overhead in the fast path is kept at an absolute minimum.

Quoted 1 line

Though yes, it would be good to hear more opinions here.

#117 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

In reply to Konstantin Ananyev
Quoted 132 lines

From: Morten Brørup
Sent: Tuesday, 28 February 2023 14.16

Quoted 4 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Tuesday, 28 February 2023 13.05

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Add support for programming PMU counters and reading their values

in

Quoted 31 lines
Quoted 22 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

[...]

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10 -0
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET,

PERF_IOC_FLAG_GROUP) == -

Quoted 22 lines
Quoted 6 lines
Quoted 6 lines

1) {

Quoted 5 lines
Quoted 5 lines

lib/pmu/rte_pmu.c

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE,

PERF_IOC_FLAG_GROUP) ==

Quoted 15 lines
Quoted 15 lines

-1) {

Quoted 11 lines
Quoted 9 lines

lib/pmu/rte_pmu.c line 291

Quoted 6 lines
lib/pmu/rte_pmu.c new file +6 -0
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Nothing special. Any pointers to that thread-local in that thread are

invalided.

Quoted 2 lines
Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for

other

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

purposes)?

Quoted 4 lines
Quoted 1 line

Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.

I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while

program

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

continues to run.

Is the example you describe here (i.e. a thread terminating in the middle

of

Quoted 4 lines

doing something) really a scenario DPDK is supposed to

Quoted 1 line

support?

I am not talking about some abnormal termination.

Then I misunderstood your example; I thought you meant the tread was terminated while inside the rte_pmu_read() function.

Quoted 5 lines

We do have ability to spawn control threads, user can spawn his own thread, all these
threads can have limited life-time.
Not to mention about rte_thread_register()/rte_thread_unregister().

I agree that normal thread termination should be supported.

Quoted 6 lines
Quoted 1 line
Quoted 1 line

As I understand address of its RTE_PER_LCORE(_event_group) will still

remain

Quoted 4 lines
Quoted 2 lines

in rte_pmu.event_group_list,
even if it is probably not valid any more.

There should be a "destructor/done/finish" function available to remove

this

Quoted 5 lines

from the list.

Quoted 4 lines

[...]

Quoted 1 line
Quoted 1 line
Quoted 1 line

Even if we'd decide to keep rte_pmu_read() as static inline (still

not

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

sure it is a good idea),

We want to save as much cpu cycles as we possibly can and inlining

does

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

helps

Quoted 1 line

in that matter.

Ok, so asking same question from different thread: how many cycles it

will

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

save?
What is the difference in terms of performance when you have this

function

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

inlined vs not inlined?

We expect to use this in our in-house profiler library. For this reason, I

have a very strong preference for absolute maximum

Quoted 3 lines

performance.

Reading PMU events is for performance profiling, so I expect other

potential

Quoted 3 lines

users of the PMU library to share my opinion on this.

Well, from my perspective 14 cycles are not that much...

For reference, the i40e testpmd per-core performance report shows that it uses 36 cycles per packet.

Quoted 6 lines

This is a total of 1152 cycles per burst of 32 packets. 14 cycles overhead per burst / 1152 cycles per burst = 1.2 % overhead.

But that is not all: If the application's pipeline has three stages, where the PMU counters are read for each stage, the per-invocation overhead of 14 cycles adds up, and the overhead per burst is now 3 * 14 / 1152 = 3.6 %.

I was too fast on the keyboard here... If the application does more work than testpmd, it certainly also uses more than 1152 cycles to do that work. So please ignore the 3.6 % as a wild exaggeration from an invalid example, and just stick with the 1.2 % overhead - which I still consider significant, and thus worth avoiding.

Quoted 18 lines

Generalizing...

In my example here, the same function with 14 wasted cycles is called three times. It might as well be three individual libraries each wasting 14 cycles in its individual fast path processing function, due to a similarly relaxed attitude regarding wasting 14 cycles.

My point is:

Real applications do much more work than testpmd, so all this "insignificant" extra overhead in the libraries adds up!

Generally, I would like the DPDK Project to remain loyal to its original philosophy, where performance is considered a Key Performance Indicator, and overhead in the fast path is kept at an absolute minimum.

Quoted 1 line

Though yes, it would be good to hear more opinions here.

#118 lore raw

Re: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 127 lines
Quoted 127 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Add support for programming PMU counters and reading their values

in

Quoted 31 lines
Quoted 22 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

[...]

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
(quoted diff) +10 -0
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET,

PERF_IOC_FLAG_GROUP) == -

Quoted 22 lines
Quoted 6 lines
Quoted 6 lines

1) {

Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE,

PERF_IOC_FLAG_GROUP) ==

Quoted 15 lines
Quoted 15 lines

-1) {

Quoted 11 lines
Quoted 9 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+		ret = -errno;
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Nothing special. Any pointers to that thread-local in that thread are

invalided.

Quoted 2 lines
Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for

other

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

purposes)?

Quoted 4 lines
Quoted 1 line

Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.

I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while

program

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

continues to run.

Is the example you describe here (i.e. a thread terminating in the middle

of

Quoted 4 lines

doing something) really a scenario DPDK is supposed to

Quoted 1 line

support?

I am not talking about some abnormal termination.

Then I misunderstood your example; I thought you meant the tread was terminated while inside the rte_pmu_read() function.

Quoted 5 lines

We do have ability to spawn control threads, user can spawn his own thread, all these
threads can have limited life-time.
Not to mention about rte_thread_register()/rte_thread_unregister().

I agree that normal thread termination should be supported.

Quoted 6 lines
Quoted 1 line
Quoted 1 line

As I understand address of its RTE_PER_LCORE(_event_group) will still

remain

Quoted 4 lines
Quoted 2 lines

in rte_pmu.event_group_list,
even if it is probably not valid any more.

There should be a "destructor/done/finish" function available to remove

this

Quoted 5 lines

from the list.

Quoted 4 lines

[...]

Quoted 1 line
Quoted 1 line
Quoted 1 line

Even if we'd decide to keep rte_pmu_read() as static inline (still

not

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

sure it is a good idea),

We want to save as much cpu cycles as we possibly can and inlining

does

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

helps

Quoted 1 line

in that matter.

Ok, so asking same question from different thread: how many cycles it

will

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

save?
What is the difference in terms of performance when you have this

function

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

inlined vs not inlined?

We expect to use this in our in-house profiler library. For this reason, I

have a very strong preference for absolute maximum

Quoted 3 lines

performance.

Reading PMU events is for performance profiling, so I expect other

potential

Quoted 3 lines

users of the PMU library to share my opinion on this.

Well, from my perspective 14 cycles are not that much...

For reference, the i40e testpmd per-core performance report shows that it uses 36 cycles per packet.

Quoted 8 lines
Quoted 6 lines

This is a total of 1152 cycles per burst of 32 packets. 14 cycles overhead per burst / 1152 cycles per burst = 1.2 % overhead.

But that is not all: If the application's pipeline has three stages, where the PMU counters are read for each stage, the per-invocation overhead of 14 cycles adds up, and the overhead per burst is now 3 * 14 / 1152 = 3.6 %.

I was too fast on the keyboard here... If the application does more work than testpmd, it certainly also uses more than 1152 cycles to do that work. So please ignore the 3.6 % as a wild exaggeration from an invalid example, and just stick with the 1.2 % overhead - which I still consider significant, and thus worth avoiding.

Wonder can we do both - hide struct rte_pmu_event_group from public API and have inline function to read pmu stats? if we can add a separate function that will allow user to get
struct perf_event_mmap_page * for given event index (or event name),
then later user can use
__rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
directly.

Quoted 19 lines
Quoted 19 lines

Generalizing...

In my example here, the same function with 14 wasted cycles is called three times. It might as well be three individual libraries each wasting 14 cycles in its individual fast path processing function, due to a similarly relaxed attitude regarding wasting 14 cycles.

My point is:

Real applications do much more work than testpmd, so all this "insignificant" extra overhead in the libraries adds up!

Generally, I would like the DPDK Project to remain loyal to its original philosophy, where performance is considered a Key Performance Indicator, and overhead in the fast path is kept at an absolute minimum.

Quoted 1 line

Though yes, it would be good to hear more opinions here.

#119 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

In reply to Konstantin Ananyev
Quoted 325 lines

-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Monday, February 27, 2023 9:53 PM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 73 lines
Quoted 73 lines
Quoted 9 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Few more comments/questions below.

Quoted 60 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..950f999cb7

lib/pmu/rte_pmu.c new file +55 -0
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) <<
+(__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3])
+{ }
+
+static int
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name,

name);

Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
lib/pmu/rte_pmu.c new file +62 -0
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,

name);

Quoted 161 lines
Quoted 147 lines
Quoted 147 lines
lib/pmu/rte_pmu.c new file +147 -0
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0); }
+
+static int
+open_events(struct rte_pmu_event_group *group) {
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group) {
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group) {
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE,
+PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
Quoted 8 lines

lib/pmu/rte_pmu.c line 291

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Nothing special. Any pointers to that thread-local in that thread are invalided.

Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for other purposes)?

Quoted 9 lines
Quoted 2 lines

Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.

I understand that main thread would need to access that data.
I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while program continues to run. As I understand address of its RTE_PER_LCORE(_event_group) will still remain in rte_pmu.event_group_list, even if it is probably not valid any more.

Okay got your point. In DPDK that will not happen. We do not spawn/kill lcores in runtime. In other scenarios such approach will not work because once thread terminates it's per-thread-data becomes invalid.

Quoted 208 lines
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
lib/pmu/rte_pmu.c new file +79 -0
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV; }
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event) {
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name) {
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,

name);

Quoted 119 lines
Quoted 118 lines
Quoted 103 lines
lib/pmu/rte_pmu.c new file +79 -0
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +19 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU
+and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {

lib/pmu/rte_pmu.h line 29

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8

forgot RTE_ prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically by enable_group()?

Quoted 8 lines
Quoted 1 line

8 is reasonable number I think. X86/ARM have actually less that that (was that something like

4?).

Quoted 2 lines

Moreover events are scheduled as a group so there must be enough hw counters available for that to succeed. So this number should cover current needs.

If you think 8 will be enough to cover all possible future cases - I am ok either way. Still need RTE_ prefix for it.

Okay that can be added.

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines

lib/pmu/rte_pmu.h

Quoted 6 lines
lib/pmu/rte_pmu.h new file +6 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages

*/

Quoted 9 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 41

Quoted 5 lines
lib/pmu/rte_pmu.h new file +5 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+

Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea),

Quoted 6 lines
Quoted 2 lines

We want to save as much cpu cycles as we possibly can and inlining does helps in that matter.

Ok, so asking same question from different thread: how many cycles it will save? What is the difference in terms of performance when you have this function inlined vs not inlined?

On x86 setup which is not under load, no cpusets configured, etc *just* not inlining rte_pmu_read() decreases performance by roughly 24% (44 vs 58 cpu cycles). At least that is reported by trace_perf_autotest.

Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

why these two struct below (rte_pmu_event and rte_pmu) have to be public? I think both can be safely moved away from public headers.

struct rte_pmu_event can be hidden I guess.
struct rte_pmu is used in this header hence cannot be moved elsewhere.

Not sure why?
Is that because you use it inside rte_pmu_read()?
But that check I think can be safely moved into __rte_pmu_enable_group() or probably even into rte_pmu_add_event().

No, we should not do that. Otherwise we'll need to call function. Even though check will happen early on still function prologue/epilogue will happen. This takes cycles.

Quoted 96 lines
Quoted 96 lines
Quoted 56 lines
Quoted 56 lines
lib/pmu/rte_pmu.h new file +53 -0
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
 
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own
+version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t __rte_pmu_read_userpage(struct
+perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
Quoted 9 lines

lib/pmu/rte_pmu.h line 102

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {

In mmap_events() you return EPERM if cap_user_rdpmc is not enabled. Do you need another check here? Or this capability can be disabled by kernel at run-time?

That extra check in mmap_event() may be removed actually. Some archs
allow disabling reading rdpmc (I think that on x86 one can do that) so this check needs to stay.

Quoted 26 lines
Quoted 13 lines
lib/pmu/rte_pmu.h new file +12 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+

lib/pmu/rte_pmu.h line 124

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+	return 0;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.

__rte_internal ?

Quoted 6 lines
Quoted 3 lines

No this cannot be internal because that will make functions calling it internal as well hence apps won't be able to use that. This has already been brought up by one of the reviewers.

Ok, then we probably can mark it with ' @internal' tag in formal comments?

I added a warning not to call that directly. Since function is not internal (in DPDK parlance) per se I don’t think we should add more confusion that extra tag.

Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 6 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

lib/pmu/rte_pmu.h new file +1 -0
+__rte_experimental

lib/pmu/rte_pmu.h line 139

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.

Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?

Quoted 5 lines
Quoted 1 line

This is inlined and has one caller i.e rte_pmu_read().

I thought we are talking here about rte_pmu_init().
I don't see where it is inlined and still not clear why it can't be called directly.

No this cannot be called by init because groups are configured in runtime. That is why __rte_pmu_enable_group() is called once in rte_pmu_read().

*Other* code should not call that directly. And yes, that is not inlined - my mistake.

Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

Quoted 7 lines
Quoted 5 lines

Probably worth to mention that this function is not MT safe. Same for _fini_ and add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.

Yes they are meant to be called from main thread.

Ok, then please add that into formal API comments.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+__rte_experimental
+int
+rte_pmu_init(void);
+
#120 lore raw

RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events

Quoted 319 lines
Quoted 319 lines
Quoted 75 lines
Quoted 75 lines
Quoted 11 lines

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Few more comments/questions below.

Quoted 60 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0 -0

100644 index 0000000000..950f999cb7

lib/pmu/rte_pmu.c new file +55 -0
@@ -0,0 +1,460 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2023 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_atomic.h>
+#include <rte_per_lcore.h>
+#include <rte_pmu.h>
+#include <rte_spinlock.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) <<
+(__builtin_ffsll(m) - 1)) & (m))
+
+RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+struct rte_pmu rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3])
+{ }
+
+static int
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char path[PATH_MAX];
+	char *config = NULL;
+	int high, low, ret;
+	FILE *fp;
+
+	*num = *mask = 0;
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name,

name);

Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
lib/pmu/rte_pmu.c new file +62 -0
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,

name);

Quoted 163 lines
Quoted 147 lines
Quoted 147 lines
lib/pmu/rte_pmu.c new file +147 -0
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0, -1, group_fd, 0); }
+
+static int
+open_events(struct rte_pmu_event_group *group) {
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret)
+			continue;
+
+		ret = do_perf_event_open(config, group->fds[0]);
+		if (ret == -1) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(struct rte_pmu_event_group *group) {
+	long page_size = sysconf(_SC_PAGE_SIZE);
+	unsigned int i;
+	void *addr;
+	int ret;
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		addr = mmap(0, page_size, PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+		if (!group->mmap_pages[i]->cap_user_rdpmc) {
+			ret = -EPERM;
+			goto out;
+		}
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], page_size);
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(struct rte_pmu_event_group *group) {
+	unsigned int i;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE,
+PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu.num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], sysconf(_SC_PAGE_SIZE));
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	group->enabled = false;
+}
+
+int
+__rte_pmu_enable_group(void)
+{
+	struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group);
+	int ret;
+
+	if (rte_pmu.num_group_events == 0)
+		return -ENODEV;
+
+	ret = open_events(group);
+	if (ret)
+		goto out;
+
+	ret = mmap_events(group);
+	if (ret)
+		goto out;
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		ret = -errno;
Quoted 8 lines

lib/pmu/rte_pmu.c line 291

Quoted 5 lines
lib/pmu/rte_pmu.c new file +5 -0
+		goto out;
+	}
+
+	rte_spinlock_lock(&rte_pmu.lock);
+	TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);

Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?

Nothing special. Any pointers to that thread-local in that thread are invalided.

Quoted 2 lines

Can memory from its TLS block get re-used (by other thread or for other purposes)?

Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.

Quoted 8 lines
Quoted 6 lines

I understand that main thread would need to access that data.
I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while program continues to run. As I understand address of its RTE_PER_LCORE(_event_group) will still remain in rte_pmu.event_group_list, even if it is probably not valid any more.

Okay got your point. In DPDK that will not happen. We do not spawn/kill lcores in runtime.

Well, yes usually DPDK app doesn't do that, but in theory there is an API to register/unregister non-eal threads as lcores: rte_thread_register()/rte_thread_unregister().
Also besides of lcores there are control threads, some house-keeping threads, plus user is free to spawn/kill his own threads.
Are you saying that this library doesn't support none of them?
If so, then at least that should be very clearly documented.
Though I think a proper way is handle this situation somehow -
either return error at __rte_pmu_enable_group(), or change the code to allow it to work properly from any thread. I don't think it is that hard.

Quoted 241 lines

In other scenarios such approach will not work because once thread terminates it's per-thread-data becomes invalid.

Quoted 208 lines
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
lib/pmu/rte_pmu.c new file +79 -0
+	rte_spinlock_unlock(&rte_pmu.lock);
+	group->enabled = true;
+
+	return 0;
+
+out:
+	cleanup_events(group);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (dirp == NULL)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	if (dent) {
+		rte_pmu.name = strdup(name);
+		if (rte_pmu.name == NULL) {
+			closedir(dirp);
+
+			return -ENOMEM;
+		}
+	}
+
+	closedir(dirp);
+
+	return rte_pmu.name ? 0 : -ENODEV; }
+
+static struct rte_pmu_event *
+new_event(const char *name)
+{
+	struct rte_pmu_event *event;
+
+	event = calloc(1, sizeof(*event));
+	if (event == NULL)
+		goto out;
+
+	event->name = strdup(name);
+	if (event->name == NULL) {
+		free(event);
+		event = NULL;
+	}
+
+out:
+	return event;
+}
+
+static void
+free_event(struct rte_pmu_event *event) {
+	free(event->name);
+	free(event);
+}
+
+int
+rte_pmu_add_event(const char *name) {
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
Quoted 8 lines
Quoted 8 lines

lib/pmu/rte_pmu.c

Quoted 8 lines
lib/pmu/rte_pmu.c new file +8 -0
+
+	if (rte_pmu.name == NULL)
+		return -ENODEV;
+
+	if (rte_pmu.num_group_events + 1 >= MAX_NUM_GROUP_EVENTS)
+		return -ENOSPC;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,

name);

Quoted 119 lines
Quoted 118 lines
Quoted 103 lines
lib/pmu/rte_pmu.c new file +79 -0
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu.event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = new_event(name);
+	if (event == NULL)
+		return -ENOMEM;
+
+	event->index = rte_pmu.num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu.event_list, event, next);
+
+	return event->index;
+}
+
+int
+rte_pmu_init(void)
+{
+	int ret;
+
+	/* Allow calling init from multiple contexts within a single thread. This simplifies
+	 * resource management a bit e.g in case fast-path tracepoint has already been enabled
+	 * via command line but application doesn't care enough and performs init/fini again.
+	 */
+	if (rte_pmu.initialized != 0) {
+		rte_pmu.initialized++;
+		return 0;
+	}
+
+	ret = scan_pmus();
+	if (ret)
+		goto out;
+
+	ret = pmu_arch_init();
+	if (ret)
+		goto out;
+
+	TAILQ_INIT(&rte_pmu.event_list);
+	TAILQ_INIT(&rte_pmu.event_group_list);
+	rte_spinlock_init(&rte_pmu.lock);
+	rte_pmu.initialized = 1;
+
+	return 0;
+out:
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+
+	return ret;
+}
+
+void
+rte_pmu_fini(void)
+{
+	struct rte_pmu_event_group *group, *tmp_group;
+	struct rte_pmu_event *event, *tmp_event;
+
+	/* cleanup once init count drops to zero */
+	if (rte_pmu.initialized == 0 || --rte_pmu.initialized != 0)
+		return;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) {
+		TAILQ_REMOVE(&rte_pmu.event_list, event, next);
+		free_event(event);
+	}
+
+	RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) {
+		TAILQ_REMOVE(&rte_pmu.event_group_list, group, next);
+		cleanup_events(group);
+	}
+
+	pmu_arch_fini();
+	free(rte_pmu.name);
+	rte_pmu.name = NULL;
+	rte_pmu.num_group_events = 0;
+}
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0 -0

100644 index 0000000000..6b664c3336

lib/pmu/rte_pmu.h new file +19 -0
@@ -0,0 +1,212 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU
+and
+ * read selected counters in runtime.
+ */
+
+#ifdef __cplusplus
+extern "C" {

lib/pmu/rte_pmu.h line 29

Quoted 10 lines
lib/pmu/rte_pmu.h new file +10 -0
+#endif
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h> #include <rte_common.h>
+#include <rte_compat.h> #include <rte_spinlock.h>
+
+/** Maximum number of events in a group */ #define
+MAX_NUM_GROUP_EVENTS 8

forgot RTE_ prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically by enable_group()?

Quoted 8 lines
Quoted 1 line

8 is reasonable number I think. X86/ARM have actually less that that (was that something like

4?).

Quoted 2 lines

Moreover events are scheduled as a group so there must be enough hw counters available for that to succeed. So this number should cover current needs.

If you think 8 will be enough to cover all possible future cases - I am ok either way. Still need RTE_ prefix for it.

Okay that can be added.

Quoted 19 lines
Quoted 6 lines
Quoted 6 lines

lib/pmu/rte_pmu.h

Quoted 6 lines
lib/pmu/rte_pmu.h new file +6 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages

*/

Quoted 11 lines
Quoted 8 lines

lib/pmu/rte_pmu.h line 41

Quoted 5 lines
lib/pmu/rte_pmu.h new file +5 -0
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	bool enabled; /**< true if group was enabled on particular lcore */
+	TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ }
+__rte_cache_aligned;
+

Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea),

We want to save as much cpu cycles as we possibly can and inlining does helps in that matter.

Quoted 7 lines
Quoted 3 lines

Ok, so asking same question from different thread: how many cycles it will save? What is the difference in terms of performance when you have this function inlined vs not inlined?

On x86 setup which is not under load, no cpusets configured, etc *just* not inlining rte_pmu_read() decreases performance by roughly 24% (44 vs 58 cpu cycles). At least that is reported by trace_perf_autotest.

From my perspective 14 cycles is not that much...
Considering that user will probably not call it very often, and by enabling measurements he probably already prepared to get some hit.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

why these two struct below (rte_pmu_event and rte_pmu) have to be public? I think both can be safely moved away from public headers.

struct rte_pmu_event can be hidden I guess.
struct rte_pmu is used in this header hence cannot be moved elsewhere.

Quoted 7 lines
Quoted 4 lines

Not sure why?
Is that because you use it inside rte_pmu_read()?
But that check I think can be safely moved into __rte_pmu_enable_group() or probably even into rte_pmu_add_event().

No, we should not do that. Otherwise we'll need to call function. Even though check will happen early on still function prologue/epilogue will happen. This takes cycles.

Not necessary. You can store this value in pmu_group itself, and by this value decide is pmu and group initialized, etc.

Quoted 100 lines
Quoted 100 lines
Quoted 99 lines
Quoted 56 lines
Quoted 56 lines
lib/pmu/rte_pmu.h new file +53 -0
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /**< name of an event */
+	unsigned int index; /**< event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ };
 
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */
+	rte_spinlock_t lock; /**< serialize access to event group list */
+	TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */
+	unsigned int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+	unsigned int initialized; /**< initialization counter */ };
+
+/** lcore event group */
+RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group);
+
+/** PMU state container */
+extern struct rte_pmu rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own
+version */ #ifndef rte_pmu_pmc_read #define
+rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read PMU counter.
+ *
+ * @warning This should be not called directly.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+static __rte_always_inline uint64_t __rte_pmu_read_userpage(struct
+perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
Quoted 9 lines

lib/pmu/rte_pmu.h line 102

Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+		width = pc->pmc_width;
+
+		/* index set to 0 means that particular counter cannot be used */
+		if (likely(pc->cap_user_rdpmc && index)) {

In mmap_events() you return EPERM if cap_user_rdpmc is not enabled. Do you need another check here? Or this capability can be disabled by kernel at run-time?

That extra check in mmap_event() may be removed actually. Some archs
allow disabling reading rdpmc (I think that on x86 one can do that) so this check needs to stay.

Quoted 17 lines
Quoted 17 lines
lib/pmu/rte_pmu.h new file +16 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}
+
+/**
Quoted 9 lines
Quoted 6 lines

+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.

__rte_internal ?

No this cannot be internal because that will make functions calling it internal as well hence apps won't be able to use that. This has already been brought up by one of the reviewers.

Quoted 5 lines
Quoted 2 lines

Ok, then we probably can mark it with ' @internal' tag in formal comments?

I added a warning not to call that directly. Since function is not internal (in DPDK parlance) per se I don’t think we should add more confusion that extra tag.

We doing it in other places, why not to add it here?

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

lib/pmu/rte_pmu.h new file +6 -0
+__rte_experimental
+int
+__rte_pmu_enable_group(void);
+
+/**
+ * @warning
Quoted 9 lines
Quoted 5 lines

+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.

Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?

This is inlined and has one caller i.e rte_pmu_read().

Quoted 8 lines
Quoted 3 lines

I thought we are talking here about rte_pmu_init().
I don't see where it is inlined and still not clear why it can't be called directly.

No this cannot be called by init because groups are configured in runtime. That is why __rte_pmu_enable_group() is called once in rte_pmu_read().

*Other* code should not call that directly. And yes, that is not inlined - my mistake.

Once again: we are discussing comments for rte_pmu_init() function. Why it can't be called directly?
In test_pmu_read() you do call it directly.

Quoted 19 lines
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */

Quoted 7 lines
Quoted 5 lines

Probably worth to mention that this function is not MT safe. Same for _fini_ and add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.

Yes they are meant to be called from main thread.

Ok, then please add that into formal API comments.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4 -0
+__rte_experimental
+int
+rte_pmu_init(void);
+