add support for self monitoring
Subsystems (2)
PATCH v8 1/4 lib: add generic support for reading PMU events
In reply to Tomasz DuszynskiAdd support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
--- MAINTAINERS | 5 + app/test/meson.build | 1 + app/test/test_pmu.c | 55 +++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 8 + doc/guides/rel_notes/release_23_03.rst | 9 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 29 ++ lib/pmu/rte_pmu.c | 464 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 205 +++++++++++ lib/pmu/version.map | 20 ++ 13 files changed, 813 insertions(+), 1 deletion(-) create mode 100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------
app/test/meson.build +1 -0
@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',
app/test/test_pmu.c new file +55 -0
55 lines not on this page · raw message
doc/api/doxy-api-index.md +2 -1
@@ -222,7 +222,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), + [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +8 -0
@@ -7,6 +7,14 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on different architectures. +Performance counter based profiling +----------------------------------- + +Majority of architectures support some sort hardware measurement unit which provides a set of +programmable counters that monitor specific events. There are different tools which can gather +that information, perf being an example here. Though in some scenarios, eg. when CPU cores are +isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can +read specific events directly from application via ``rte_pmu_read()``. Profiling on x86 ----------------
doc/guides/rel_notes/release_23_03.rst +9 -0
@@ -55,10 +55,19 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added PMU library.** + + Added a new PMU (performance measurement unit) library which allows applications + to perform self monitoring activities without depending on external utilities like perf. + After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters + can be stored in CTF format for further analysis. + * **Updated AMD axgbe driver.** * Added multi-process support. +* **Added multi-process support for axgbe PMD.** + * **Updated Corigine nfp driver.** * Added support for meter options.
lib/meson.build +1 -0
@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2023 Marvell International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. +*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +464 -0
464 lines not on this page · raw message
lib/pmu/rte_pmu.h new file +205 -0
205 lines not on this page · raw message
lib/pmu/version.map new file +20 -0
@@ -0,0 +1,20 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +}; + +INTERNAL { + global: + + rte_pmu_enable_group; +};
Signature · 1 line
2.34.1
RE: [PATCH v8 1/4] lib: add generic support for reading PMU events
Quoted 37 lines
-----Original Message-----
From: Tomasz Duszynski <[email protected]>
Sent: Thursday, February 2, 2023 5:44 PM
To: [email protected]; [email protected]; Tomasz Duszynski <[email protected]>
Cc: [email protected]; Ruifeng Wang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v8 1/4] lib: add generic support for reading PMU eventsAdd support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated (
nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>--- MAINTAINERS | 5 + app/test/meson.build | 1 + app/test/test_pmu.c | 55 +++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 8 + doc/guides/rel_notes/release_23_03.rst | 9 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 29 ++ lib/pmu/rte_pmu.c | 464 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 205 +++++++++++ lib/pmu/version.map | 20 ++ 13 files changed, 813 insertions(+), 1 deletion(-) create mode 100644
app/test/test_pmu.ccreate mode 100644lib/pmu/meson.buildcreate mode 100644lib/pmu/pmu_private.hcreate mode 100644lib/pmu/rte_pmu.ccreate mode 100644lib/pmu/rte_pmu.hcreate mode 100644lib/pmu/version.map
<snip>
doc/guides/rel_notes/release_23_03.rst New Features · line 60
Quoted 12 lines
doc/guides/rel_notes/release_23_03.rst +3
-0@@ -55,10 +55,19 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added PMU library.** + + Added a new PMU (performance measurement unit) library which allows
Overall looks good to me. Just a minor comment.
Should it be 'performance *monitoring* unit'?
I see the same terminology is used across architectures. It will be better if we align with that.
Thanks.
<snip>
PATCH v8 2/4 pmu: support reading ARM PMU events in runtime
In reply to Tomasz DuszynskiAdd support for reading ARM PMU events in runtime.
--- app/test/test_pmu.c | 4 ++ lib/pmu/meson.build | 7 +++ lib/pmu/pmu_arm64.c | 94 +++++++++++++++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 4 ++ lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++ 5 files changed, 139 insertions(+) create mode 100644 lib/pmu/pmu_arm64.c create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
app/test/test_pmu.c +4 -0
@@ -26,6 +26,10 @@ test_pmu_read(void)if (rte_pmu_init () <0 )return TEST_FAILED ; +#if defined(RTE_ARCH_ARM64 ) + event =rte_pmu_add_event ("cpu_cycles" ); +#endif +while (tries--) val +=rte_pmu_read (event);
lib/pmu/meson.build +7 -0
@@ -11,3 +11,10 @@ includes = [global_inc] sources = files('rte_pmu.c') headers = files('rte_pmu.h') +indirect_headers += files( + 'rte_pmu_pmc_arm64.h', +) + +if dpdk_conf.has('RTE_ARCH_ARM64') + sources += files('pmu_arm64.c') +endif
lib/pmu/pmu_arm64.c new file +94 -0
94 lines not on this page · raw message
lib/pmu/rte_pmu.h +4 -0
@@ -26,6 +26,10 @@ extern "C" {#include <rte_compat.h> #include <rte_spinlock.h> +#if defined(RTE_ARCH_ARM64 ) +#include "rte_pmu_pmc_arm64.h" +#endif +/** Maximum number of events in a group */ #define MAX_NUM_GROUP_EVENTS 8
lib/pmu/rte_pmu_pmc_arm64.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell. +*/ +#ifndef _RTE_PMU_PMC_ARM64_H_ +#define _RTE_PMU_PMC_ARM64_H_ + +#include <rte_common.h> + +static __rte_always_inline uint64_t +rte_pmu_pmc_read (int index) +{ +uint64_t val; + +if (index ==31 ) { +/* CPU Cycles (0x11) must be read via pmccntr_el0 */ + asmvolatile ("mrs %0, pmccntr_el0" :"=r" (val)); + }else { + asmvolatile ( +"msr pmselr_el0, %x0\n" +"mrs %0, pmxevcntr_el0\n" + :"=r" (val) + :"rZ" (index) + ); + } + +return val; +} +#define rte_pmu_pmc_read rte_pmu_pmc_read + +#endif /* _RTE_PMU_PMC_ARM64_H_ */
Signature · 1 line
2.34.1
PATCH v8 3/4 pmu: support reading Intel x86_64 PMU events in runtime
In reply to Tomasz DuszynskiAdd support for reading Intel x86_64 PMU events in runtime.
--- app/test/test_pmu.c | 2 ++ lib/pmu/meson.build | 1 + lib/pmu/rte_pmu.h | 2 ++ lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++++++++++++++++++++++++ 4 files changed, 29 insertions(+) create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
app/test/test_pmu.c +2 -0
@@ -28,6 +28,8 @@ test_pmu_read(void)#if defined(RTE_ARCH_ARM64 ) event =rte_pmu_add_event ("cpu_cycles" ); +#elif defined(RTE_ARCH_X86_64 ) + event =rte_pmu_add_event ("cpu-cycles" );#endif while (tries--)
lib/pmu/meson.build +1 -0
@@ -13,6 +13,7 @@ sources = files('rte_pmu.c') headers = files('rte_pmu.h') indirect_headers += files( 'rte_pmu_pmc_arm64.h', + 'rte_pmu_pmc_x86_64.h', ) if dpdk_conf.has('RTE_ARCH_ARM64')
lib/pmu/rte_pmu.h +2 -0
@@ -28,6 +28,8 @@ extern "C" {#if defined(RTE_ARCH_ARM64 )#include "rte_pmu_pmc_arm64.h" +#elif defined(RTE_ARCH_X86_64 ) +#include "rte_pmu_pmc_x86_64.h" #endif /** Maximum number of events in a group */
lib/pmu/rte_pmu_pmc_x86_64.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell. +*/ +#ifndef _RTE_PMU_PMC_X86_64_H_ +#define _RTE_PMU_PMC_X86_64_H_ + +#include <rte_common.h> + +static __rte_always_inline uint64_t +rte_pmu_pmc_read (int index) +{ +uint64_t low, high; + + asmvolatile ( +"rdpmc\n" + :"=a" (low),"=d" (high) + :"c" (index) + ); + +return low | (high <<32 ); +} +#define rte_pmu_pmc_read rte_pmu_pmc_read + +#endif /* _RTE_PMU_PMC_X86_64_H_ */
Signature · 1 line
2.34.1
PATCH v8 4/4 eal: add PMU support to tracing library
In reply to Tomasz DuszynskiIn order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.
--- app/test/test_trace_perf.c | 10 ++++ doc/guides/prog_guide/profile_app.rst | 5 ++ doc/guides/prog_guide/trace_lib.rst | 32 +++++++++++++ lib/eal/common/eal_common_trace.c | 13 ++++- lib/eal/common/eal_common_trace_points.c | 5 ++ lib/eal/include/rte_eal_trace.h | 13 +++++ lib/eal/meson.build | 3 ++ lib/eal/version.map | 1 + lib/pmu/rte_pmu.c | 61 ++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 14 ++++++ lib/pmu/version.map | 1 + 11 files changed, 157 insertions(+), 1 deletion(-)
app/test/test_trace_perf.c +10 -0
@@ -114,6 +114,10 @@ worker_fn_##func(void *arg) \#define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)#define GENERIC_STR rte_eal_trace_generic_str("hello world")#define VOID_FP app_dpdk_test_fp() +#ifdef RTE_EXEC_ENV_LINUX +/* 0 corresponds first event passed via --trace= */ +#define READ_PMU rte_eal_trace_pmu_read(0) +#endif WORKER_DEFINE (GENERIC_VOID)WORKER_DEFINE (GENERIC_U64)
@@ -122,6 +126,9 @@ WORKER_DEFINE(GENERIC_FLOAT) WORKER_DEFINE(GENERIC_DOUBLE) WORKER_DEFINE(GENERIC_STR) WORKER_DEFINE(VOID_FP) +#ifdef RTE_EXEC_ENV_LINUX +WORKER_DEFINE(READ_PMU) +#endif static void run_test (const char *str,lcore_function_t f,struct test_data *data,size_t sz)
@@ -174,6 +181,9 @@ test_trace_perf(void)run_test ("double" , worker_fn_GENERIC_DOUBLE, data, sz);run_test ("string" , worker_fn_GENERIC_STR, data, sz);run_test ("void_fp" , worker_fn_VOID_FP, data, sz); +#ifdef RTE_EXEC_ENV_LINUX +run_test ("read_pmu" , worker_fn_READ_PMU, data, sz); +#endif rte_free (data);return TEST_SUCCESS ;
doc/guides/prog_guide/profile_app.rst +5 -0
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can read specific events directly from application via ``rte_pmu_read()``. +Alternatively tracing library can be used which offers dedicated tracepoint +``rte_eal_trace_pmu_event()``. + +Refer to :doc:`../prog_guide/trace_lib` for more details. + Profiling on x86 ----------------
doc/guides/prog_guide/trace_lib.rst +32 -0
@@ -46,6 +46,7 @@ DPDK tracing library features trace format and is compatible with ``LTTng``. For detailed information, refer to `Common Trace Format <https://diamon.org/ctf/>`_. +- Support reading PMU events on ARM64 and x86-64 (Intel) How to add a tracepoint? ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``. ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using the ``enable_trace_fp`` option for meson build. +PMU tracepoint +-------------- + +Performance measurement unit (PMU) event values can be read from hardware +registers using predefined ``rte_pmu_read`` tracepoint. + +Tracing is enabled via ``--trace`` EAL option by passing both expression +matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression +``e=ev1[,ev2,...]`` matching particular events:: + + --trace='.*pmu.read\|e=cpu_cycles,l1d_cache' + +Event names are available under ``/sys/bus/event_source/devices/PMU/events`` +directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory +containing ``cpus``. + +In contrary to other tracepoints this does not need any extra variables +added to source files. Instead, caller passes index which follows the order of +events specified via ``--trace`` parameter. In the following example index ``0`` +corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``. + +.. code-block:: c + + ... + rte_eal_trace_pmu_read(0); + rte_eal_trace_pmu_read(1); + ... + +PMU tracing support must be explicitly enabled using the ``enable_trace_fp`` +option for meson build. + Event record mode -----------------
lib/eal/common/eal_common_trace.c +12 -1
@@ -11,6 +11,9 @@#include <rte_errno.h> #include <rte_lcore.h> #include <rte_per_lcore.h> +#ifdef RTE_EXEC_ENV_LINUX +#include <rte_pmu.h> +#endif #include <rte_string_fns.h> #include "eal_trace.h"
@@ -71,8 +74,13 @@ eal_trace_init(void) goto free_meta;/* Apply global configurations */ - STAILQ_FOREACH(arg, &trace.args, next) + STAILQ_FOREACH(arg, &trace.args, next) {trace_args_apply (arg->val); +#ifdef RTE_EXEC_ENV_LINUX +if (rte_pmu_init () ==0 ) +rte_pmu_add_events_by_pattern (arg->val); +#endif + }rte_trace_mode_set (trace.mode);
@@ -88,6 +96,9 @@ eal_trace_init(void)void eal_trace_fini (void ) { +#ifdef RTE_EXEC_ENV_LINUX +rte_pmu_fini (); +#endif trace_mem_free ();trace_metadata_destroy ();eal_trace_args_free ();
lib/eal/common/eal_common_trace_points.c +5 -0
@@ -75,3 +75,8 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable, lib.eal.intr.enable) RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable, lib.eal.intr.disable) + +#ifdef RTE_EXEC_ENV_LINUX +RTE_TRACE_POINT_REGISTER (rte_eal_trace_pmu_read, + lib.eal.pmu.read) +#endif
lib/eal/include/rte_eal_trace.h +13 -0
@@ -17,6 +17,9 @@ extern "C" {#include <rte_alarm.h> #include <rte_interrupts.h> +#ifdef RTE_EXEC_ENV_LINUX +#include <rte_pmu.h> +#endif #include <rte_trace_point.h> #include "eal_interrupts.h"
@@ -279,6 +282,16 @@ RTE_TRACE_POINT(rte_trace_point_emit_string (cpuset); ) +#ifdef RTE_EXEC_ENV_LINUX +RTE_TRACE_POINT_FP ( + rte_eal_trace_pmu_read, + RTE_TRACE_POINT_ARGS(unsigned int index), + uint64_t val; + val = rte_pmu_read(index); + rte_trace_point_emit_u64(val); +) +#endif + #ifdef __cplusplus }#endif
lib/eal/meson.build +3 -0
@@ -26,6 +26,9 @@ deps += ['kvargs'] if not is_windows deps += ['telemetry'] endif +if is_linux + deps += ['pmu'] +endif if dpdk_conf.has('RTE_USE_LIBBSD') ext_deps += libbsd endif
lib/eal/version.map +1 -0
@@ -441,6 +441,7 @@ EXPERIMENTAL { rte_thread_join; # added in 23.03 + __rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT rte_thread_set_name; };
lib/pmu/rte_pmu.c +61 -0
67 lines not on this page · raw message
lib/pmu/rte_pmu.h +14 -0
@@ -169,6 +169,20 @@ __rte_experimentalint rte_pmu_add_event (const char *name); +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add events matching pattern to the group of enabled events. +* +* @param pattern +* Pattern e=ev1[,ev2,...] matching events, where evX is a placeholder for an event listed under +* /sys/bus/event_source/devices/pmu/events. +*/ +__rte_experimental +int +rte_pmu_add_events_by_pattern (const char *pattern); +/** * @warning * @b EXPERIMENTAL: this API may change without prior notice
lib/pmu/version.map +1 -0
@@ -8,6 +8,7 @@ EXPERIMENTAL { per_lcore__event_group; rte_pmu; rte_pmu_add_event; + rte_pmu_add_events_by_pattern; rte_pmu_fini; rte_pmu_init; rte_pmu_read;
Signature · 1 line
2.34.1
v9 cover add support for self monitoring
In reply to Tomasz DuszynskiThis series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.
Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.
By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.
Events are enabled by names, which need to be read from standard location under sysfs i.e
/sys/bus/event_source/devices/PMU/events
where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.
v9:
- fix 'maybe-uninitialized' warning reported by CI
v8:
- just rebase series
v7:
- use per-lcore event group instead of global table index by lcore-id
- don't add pmu_autotest to fast tests because due to lack of suported on every arch
v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there
v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra
Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library
MAINTAINERS | 5 + app/test/meson.build | 1 + app/test/test_pmu.c | 61 +++ app/test/test_trace_perf.c | 10 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 13 + doc/guides/prog_guide/trace_lib.rst | 32 ++ doc/guides/rel_notes/release_23_03.rst | 9 + lib/eal/common/eal_common_trace.c | 13 +- lib/eal/common/eal_common_trace_points.c | 5 + lib/eal/include/rte_eal_trace.h | 13 + lib/eal/meson.build | 3 + lib/eal/version.map | 1 + lib/meson.build | 1 + lib/pmu/meson.build | 21 + lib/pmu/pmu_arm64.c | 94 ++++ lib/pmu/pmu_private.h | 29 ++ lib/pmu/rte_pmu.c | 525 +++++++++++++++++++++++ lib/pmu/rte_pmu.h | 225 ++++++++++ lib/pmu/rte_pmu_pmc_arm64.h | 30 ++ lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++ lib/pmu/version.map | 21 + 23 files changed, 1138 insertions(+), 2 deletions(-) create mode 100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_arm64.c create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h create mode 100644 lib/pmu/version.map
--
2.34.1
PATCH v9 1/4 lib: add generic support for reading PMU events
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
--- MAINTAINERS | 5 + app/test/meson.build | 1 + app/test/test_pmu.c | 55 +++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 8 + doc/guides/rel_notes/release_23_03.rst | 9 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 29 ++ lib/pmu/rte_pmu.c | 464 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 205 +++++++++++ lib/pmu/version.map | 20 ++ 13 files changed, 813 insertions(+), 1 deletion(-) create mode 100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------
app/test/meson.build +1 -0
@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',
app/test/test_pmu.c new file +55 -0
55 lines not on this page · raw message
doc/api/doxy-api-index.md +2 -1
@@ -222,7 +222,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), + [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +8 -0
@@ -7,6 +7,14 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on different architectures. +Performance counter based profiling +----------------------------------- + +Majority of architectures support some sort hardware measurement unit which provides a set of +programmable counters that monitor specific events. There are different tools which can gather +that information, perf being an example here. Though in some scenarios, eg. when CPU cores are +isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can +read specific events directly from application via ``rte_pmu_read()``. Profiling on x86 ----------------
doc/guides/rel_notes/release_23_03.rst +9 -0
@@ -55,10 +55,19 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added PMU library.** + + Added a new PMU (performance measurement unit) library which allows applications + to perform self monitoring activities without depending on external utilities like perf. + After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters + can be stored in CTF format for further analysis. + * **Updated AMD axgbe driver.** * Added multi-process support. +* **Added multi-process support for axgbe PMD.** + * **Updated Corigine nfp driver.** * Added support for meter options.
lib/meson.build +1 -0
@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2023 Marvell International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. +*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +464 -0
464 lines not on this page · raw message
lib/pmu/rte_pmu.h new file +205 -0
205 lines not on this page · raw message
lib/pmu/version.map new file +20 -0
@@ -0,0 +1,20 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +}; + +INTERNAL { + global: + + rte_pmu_enable_group; +};
Signature · 1 line
2.34.1
Re: [PATCH v9 1/4] lib: add generic support for reading PMU events
Hello,
Quoted 7 lines · Tomasz Duszynski
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated (
nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
For my understanding, what OS capability/permission are required to use this library?
Quoted 46 lines
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>--- MAINTAINERS | 5 + app/test/meson.build | 1 + app/test/test_pmu.c | 55 +++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 8 + doc/guides/rel_notes/release_23_03.rst | 9 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 29 ++ lib/pmu/rte_pmu.c | 464 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 205 +++++++++++ lib/pmu/version.map | 20 ++ 13 files changed, 813 insertions(+), 1 deletion(-) create mode 100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.mapMAINTAINERS +5
-0@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------app/test/meson.build +0
-0
app/test/meson.build test_sources = files( · line 117
Quoted 7 lines
app/test/meson.build +1
-0@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',
This code adds a new test.
This test should be added to an existing testsuite, like fast-tests etc...
Quoted 93 lines
app/test/test_pmu.c new file +55
-0@@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include "test.h" + +#ifndef RTE_EXEC_ENV_LINUX + +static int +test_pmu (void ) +{ +printf ("pmu_autotest only supported on Linux, skipping test\n" ); +return TEST_SKIPPED ; +} + +#else + +#include <rte_pmu.h> + +static int +test_pmu_read (void ) +{ +int tries =10 , event =-1 ; +uint64_t val =0 ; + +if (rte_pmu_init () <0 ) +return TEST_FAILED ; + +while (tries--) + val +=rte_pmu_read (event); + +rte_pmu_fini (); + +return val ?TEST_SUCCESS :TEST_FAILED ; +} + +static struct unit_test_suite pmu_tests = { + .suite_name ="pmu autotest" , + .setup =NULL , + .teardown =NULL , + .unit_test_cases = { +TEST_CASE (test_pmu_read), +TEST_CASES_END () + } +}; + +static int +test_pmu (void ) +{ +return unit_test_suite_runner (&pmu_tests); +} + +#endif /* RTE_EXEC_ENV_LINUX */ + +REGISTER_TEST_COMMAND (pmu_autotest, test_pmu);doc/api/doxy-api-index.md +2
-1@@ -222,7 +222,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), + [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),doc/api/doxy-api.conf.in +1
-0@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \doc/guides/prog_guide/profile_app.rst +0
-0@@ -7,6 +7,14 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on
doc/guides/prog_guide/profile_app.rst Profile Your Application · line 17
Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +8
-0different architectures. +Performance counter based profiling +----------------------------------- + +Majority of architectures support some sort hardware measurement unit which provides a set of +programmable counters that monitor specific events. There are different tools which can gather +that information, perf being an example here. Though in some scenarios, eg. when CPU cores are +isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can +read specific events directly from application via ``rte_pmu_read()``.
We need a common definition in the documentation of what PMU stands for and use it consistently.
I am not sure this documentation is the best place, but at least, I'd prefer we go with "Performance Monitoring Unit" and stick to it.
Plus, this block is a bit hard to read too, what do you think of:
"""
A majority of architectures support some performance monitoring unit (PMU). Such unit provides programmable counters that monitor specific events.
Different tools gather that information, like for example perf. However, in some scenarios when CPU cores are isolated (nohz_full) and run dedicated tasks, interrupting those tasks with perf may be undesirable.
In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``.
"""
And, a double newline is used between sections in this doc.
Quoted 8 lines
doc/guides/prog_guide/profile_app.rst +0
-0Profiling on x86 ----------------doc/guides/rel_notes/release_23_03.rst +0
-0
doc/guides/rel_notes/release_23_03.rst New Features · line 60
Quoted 6 lines
doc/guides/rel_notes/release_23_03.rst +3
-0@@ -55,10 +55,19 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added PMU library.** + + Added a new PMU (performance measurement unit) library which allows applications
Performance Monitoring Unit.
doc/guides/rel_notes/release_23_03.rst New Features · line 63
Quoted 4 lines
doc/guides/rel_notes/release_23_03.rst +3
-0+ to perform self monitoring activities without depending on external utilities like perf. + After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters + can be stored in CTF format for further analysis.
Afaiu, this integration comes later in the series. This part of the RN update should go with it.
doc/guides/rel_notes/release_23_03.rst New Features · line 72
Quoted 9 lines
doc/guides/rel_notes/release_23_03.rst +3
-0+ * **Updated AMD axgbe driver.** * Added multi-process support. +* **Added multi-process support for axgbe PMD.** + * **Updated Corigine nfp driver.**
Unrelated rebase damage.. please pay attention to such detail.
Quoted 52 lines
* Added support for meter options.
lib/meson.build +1
-0@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ringlib/pmu/meson.build new file +13
-0@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2023 Marvell International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')lib/pmu/pmu_private.h new file +14
-0@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int
lib/pmu/pmu_private.h line 24
Quoted 10 lines
lib/pmu/pmu_private.h new file +10
-0+pmu_arch_init (void); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall.
Please describe config[].
Quoted 16 lines
lib/pmu/pmu_private.h new file +5
-0+*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */ lib/pmu/rte_pmu.c new file +5
-0@@ -0,0 +1,464 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h>
lib/pmu/rte_pmu.c line 15
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h>
Asking to be sure because I did not check:
do we need all those includes, or is this just copy/pasted from somewhere else?
lib/pmu/rte_pmu.c line 27
Quoted 12 lines
lib/pmu/rte_pmu.c new file +12
-0+ +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> + +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices" + +#ifndef GENMASK_ULL
This macro is copy/pasted all over the dpdk tree...
This is worth a cleanup later, read: I am not asking for it as part of this series.
However, here, there is no need for protecting against its definition.
lib/pmu/rte_pmu.c line 33
Quoted 6 lines
lib/pmu/rte_pmu.c new file +6
-0+#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h))))) +#endif + +#ifndef FIELD_PREP +#define FIELD_PREP (m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m)) +#endif
Idem.
lib/pmu/rte_pmu.c line 41
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+ +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu; + +/* +* Following __rte_weak functions provide default no-op. Architectures should override them if +* necessary. +*/
Prefer using per architectures #ifdef.
It is easier to get a simple link error than use weak symbols that make it look like it could work on some arch.
lib/pmu/rte_pmu.c line 46
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ +int +__rte_weak pmu_arch_init(void) +{ +return 0 ;
Add a debug log message indicating that this arch does not support PMU.
Quoted 9 lines
lib/pmu/rte_pmu.c new file +9
-0+} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3])
lib/pmu/rte_pmu.c line 65
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+{ +} + +static int +get_term_format (const char *name,int *num,uint64_t *mask) +{ +char *config =NULL ; +char path[PATH_MAX ]; +int high, low, ret; +FILE *fp;
Reverse xmas tree when possible.
lib/pmu/rte_pmu.c line 67
Quoted 2 lines
lib/pmu/rte_pmu.c new file +2
-0+ +/* quiesce -Wmaybe-uninitialized warning */
This comment just seems to be a note for yourself. What was the issue exactly?
Quoted 190 lines
lib/pmu/rte_pmu.c new file +190
-0+ *num =0 ; + *mask =0 ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/format/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) +{ +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +} + +static int +get_event_config (const char *name,uint64_t config[3 ]) +{ +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) +{ +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); +} + +static int +open_events (struct rte_pmu_event_group *group) +{ +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) +{ +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) +{ +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE ,PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } +
lib/pmu/rte_pmu.c line 267
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int __rte_noinline
This symbol is exported out of this library, no need for noinline.
Quoted 137 lines
lib/pmu/rte_pmu.c new file +137
-0+rte_pmu_enable_group (void) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next); +rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; +} + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) +{ +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) +{ +struct rte_pmu_event *event; +char path[PATH_MAX ]; + +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); +if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} +
lib/pmu/rte_pmu.c line 414
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized) {
This is an integer so check against 0 explicitly (there may be other cases in this patch, I did not recheck the whole patch).
Quoted 86 lines
lib/pmu/rte_pmu.c new file +50
-0+ rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (!rte_pmu.initialized || --rte_pmu.initialized) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file +30
-0@@ -0,0 +1,205 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> +#include <rte_common.h> +#include <rte_compat.h> +#include <rte_spinlock.h> + +/** Maximum number of events in a group */ +#define MAX_NUM_GROUP_EVENTS 8
lib/pmu/rte_pmu.h line 40
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */ +int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ +} __rte_cache_aligned;
One problem for the future is that we have a fixed size fd array. Do we need to expose this whole structure to the application?
lib/pmu/rte_pmu.h line 47
Quoted 7 lines
lib/pmu/rte_pmu.h new file +7
-0+ +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */
This is an internal consideration.
Do we need to expose this to the application?
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ +}; + +/**
lib/pmu/rte_pmu.h line 61
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ +};
Idem, do we need to expose this to the application?
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53
-0+ +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own version */ +#ifndef rte_pmu_pmc_read +#define rte_pmu_pmc_read (index) ({ 0; }) +#endif + +/** +* @internal +* +* Read PMU counter. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +__rte_internal +static __rte_always_inline uint64_t +rte_pmu_read_userpage (struct perf_event_mmap_page *pc) +{ +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier (); + index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +}
lib/pmu/rte_pmu.h line 124
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+ +/** +* @internal +* +* Enable group of events on the calling lcore. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_internal
Unless I missed something, this symbol is called from rte_pmu_read() so this makes rte_pmu_read() itself internal.
So external applications won't be able to use the PMU API.
This can probably be confirmed by adding some call to the PMU API in an examples/.
Quoted 110 lines
lib/pmu/rte_pmu.h new file +81
-0+int +rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. +* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ +__rte_experimental +static __rte_always_inline uint64_t +rte_pmu_read (unsigned int index) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_PMU_H_ */ lib/pmu/version.map new file +20
-1@@ -0,0 +1,20 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +}; + +INTERNAL { + global: + + rte_pmu_enable_group; +}; --2.34.1
Signature · 1 line
David Marchand
RE: [EXT] Re: [PATCH v9 1/4] lib: add generic support for reading PMU events
Hi David,
Thanks for review. Comments inline.
Quoted 16 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, February 6, 2023 12:03 PM
To: Tomasz Duszynski <[email protected]>
Cc: [email protected]; Thomas Monjalon <[email protected]>; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v9 1/4] lib: add generic support for reading PMU eventsExternal Email
----------------------------------------------------------------------
Hello,Quoted 1 line · Tomasz Duszynski
Quoted 9 lines
Quoted 6 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated (
nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.For my understanding, what OS capability/permission are required to use this library?
On x86 it sufficient for self-monitoring to have kernel built with perf events enabled and /proc/sys/kernel/perf_event_paranoid knob should be set to 2, which should be a default value anyway, unless changed by some scripts.
On ARM64 you need to additionally set /proc/sys/kernel/perf_user_access to bypass kernel when accessing hw counters.
Quoted 160 lines
Quoted 45 lines
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>--- MAINTAINERS | 5 + app/test/meson.build | 1 + app/test/test_pmu.c | 55 +++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 8 + doc/guides/rel_notes/release_23_03.rst | 9 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 29 ++ lib/pmu/rte_pmu.c | 464 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 205 +++++++++++ lib/pmu/version.map | 20 ++ 13 files changed, 813 insertions(+), 1 deletion(-) create mode100644
app/test/test_pmu.ccreate mode 100644lib/pmu/meson.buildcreate mode 100644lib/pmu/pmu_private.hcreate mode 100644lib/pmu/rte_pmu.ccreate mode 100644lib/pmu/rte_pmu.hcreate mode 100644lib/pmu/version.mapMAINTAINERS index 9a0f416d2e..9f13eafd95 renamed from MAINTAINERS +0
-0100644
MAINTAINERS +5
-0@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------app/test/meson.build index renamed from app/test/meson.build +0
-0
f34d19e3c3..7b6b69dcf1100644app/test/meson.build test_sources = files( · line 117
Quoted 7 lines
app/test/meson.build +1
-0@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',This code adds a new test.
This test should be added to an existing testsuite, like fast-tests etc...Quoted 93 lines
app/test/test_pmu.c new file mode renamed from app/test/test_pmu.c +0
-0100644 index 0000000000..a9bfb1a427
app/test/test_pmu.c new file +54
-0@@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include "test.h" + +#ifndef RTE_EXEC_ENV_LINUX + +static int +test_pmu (void ) +{ +printf ("pmu_autotest only supported on Linux, skipping test\n" ); +return TEST_SKIPPED ; +} + +#else + +#include <rte_pmu.h> + +static int +test_pmu_read (void ) +{ +int tries =10 , event =-1 ; +uint64_t val =0 ; + +if (rte_pmu_init () <0 ) +return TEST_FAILED ; + +while (tries--) + val +=rte_pmu_read (event); + +rte_pmu_fini (); + +return val ?TEST_SUCCESS :TEST_FAILED ; } + +static struct unit_test_suite pmu_tests = { + .suite_name ="pmu autotest" , + .setup =NULL , + .teardown =NULL , + .unit_test_cases = { +TEST_CASE (test_pmu_read), +TEST_CASES_END () + } +}; + +static int +test_pmu (void ) +{ +return unit_test_suite_runner (&pmu_tests); +} + +#endif /* RTE_EXEC_ENV_LINUX */ + +REGISTER_TEST_COMMAND (pmu_autotest, test_pmu);doc/api/doxy-api-index.md +1
-1@@ -222,7 +222,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0
-0
f0886c3bd1..920e615996100644doc/api/doxy-api.conf.in +1
-0@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \ diff --git
a/doc/guides/prog_guide/profile_app.rstb/doc/guides/prog_guide/profile_app.rst
index14292d4c25..a8b501fe0c100644doc/guides/prog_guide/profile_app.rst +0
-0@@ -7,6 +7,14 @@ Profile Your Application The following sectionsdescribe methods of profiling DPDK applications on different architectures.
doc/guides/prog_guide/profile_app.rst
Quoted 8 lines
doc/guides/prog_guide/profile_app.rst +8
-0+Performance counter based profiling +----------------------------------- + +Majority of architectures support some sort hardware measurement unit +which provides a set of programmable counters that monitor specific +events. There are different tools which can gather that information, +perf being an example here. Though in some scenarios, eg. when CPU +cores are isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In suchcases one can read specific events directly from application via ``rte_pmu_read()``.
Quoted 3 lines
We need a common definition in the documentation of what PMU stands for and use it consistently. I am not sure this documentation is the best place, but at least, I'd prefer we go with "Performance Monitoring Unit" and stick to it.
For the time being I think it's good enough. Frankly I don’t have better idea where to put that.
Quoted 7 lines
Plus, this block is a bit hard to read too, what do you think of:
"""
A majority of architectures support some performance monitoring unit (PMU). Such unit provides programmable counters that monitor specific events.
Quoted 9 lines
Different tools gather that information, like for example perf.
However, in some scenarios when CPU cores are isolated (nohz_full) and run dedicated tasks, interrupting those tasks with perf may be undesirable.
In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``. """And, a double newline is used between sections in this doc.
No problem.
Quoted 31 lines
Quoted 9 lines
Profiling on x86
----------------doc/guides/rel_notes/release_23_03.rst +0
-0doc/guides/rel_notes/release_23_03.rst New Features · line 61
Quoted 7 lines
doc/guides/rel_notes/release_23_03.rst +4
-0@@ -55,10 +55,19 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added PMU library.** + + Added a new PMU (performance measurement unit) library which allows + applicationsPerformance Monitoring Unit.
doc/guides/rel_notes/release_23_03.rst New Features
Quoted 4 lines
doc/guides/rel_notes/release_23_03.rst +3
-0+ to perform self monitoring activities without depending on external utilities like perf. + After integration with :doc:`../prog_guide/trace_lib` data gathered + from hardware counters can be stored in CTF format for further analysis.Afaiu, this integration comes later in the series. This part of the RN update should go with it.
Quoted 3 lines
doc/guides/rel_notes/release_23_03.rst +1
-0+ * **Updated AMD axgbe driver.**
Quoted 9 lines
doc/guides/rel_notes/release_23_03.rst New Features · line 71
Quoted 6 lines
doc/guides/rel_notes/release_23_03.rst +2
-0* Added multi-process support. +* **Added multi-process support for axgbe PMD.** + * **Updated Corigine nfp driver.**Unrelated rebase damage.. please pay attention to such detail.
Thanks, that creeped in somehow.
Quoted 55 lines
Quoted 55 lines
doc/guides/rel_notes/release_23_03.rst +0
-0* Added support for meter options.lib/meson.build index renamed from lib/meson.build +0
-0
a90fee31b7..7132131b5c100644lib/meson.build +1
-0@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ringlib/pmu/meson.build new file mode renamed from lib/pmu/meson.build +0
-0100644 index 0000000000..a4160b494e
lib/pmu/meson.build new file +13
-0@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(C) 2023 Marvell +International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')lib/pmu/pmu_private.h new file renamed from lib/pmu/pmu_private.h +0
-0mode 100644 index 0000000000..849549b125
lib/pmu/pmu_private.h new file +18
-0@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback.
Quoted 9 lines
lib/pmu/pmu_private.h line 24
Quoted 6 lines
lib/pmu/pmu_private.h new file +6
-0+*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. Please describe config[].
Well, the problem here is that each and every arch may expect different config so anyone adding new stuff would need to consult kernel sources anyway.
What kind of documentation would you like to see here?
Quoted 21 lines
Quoted 21 lines
lib/pmu/pmu_private.h new file +5
-0+*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */ lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..4cf3161155
lib/pmu/rte_pmu.c new file +10
-0@@ -0,0 +1,464 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h>
Quoted 9 lines
lib/pmu/rte_pmu.c line 15
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> Asking to be sure because I did not check:
do we need all those includes, or is this just copy/pasted from somewhere else?
No, it was not copy/pasted. Each header exports something used in this file. I'll double check if all still required.
Quoted 17 lines
lib/pmu/rte_pmu.c line 27
Quoted 13 lines
lib/pmu/rte_pmu.c new file +12
-0+ +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> + +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices" + +#ifndef GENMASK_ULL This macro is copy/pasted all over the dpdk tree...
This is worth a cleanup later, read: I am not asking for it as part of this series.
Quoted 2 lines
However, here, there is no need for protecting against its definition.
Okay.
Quoted 11 lines
lib/pmu/rte_pmu.c line 33
Quoted 7 lines
lib/pmu/rte_pmu.c new file +6
-0+#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> +((64 - 1 - (h))))) #endif + +#ifndef FIELD_PREP +#define FIELD_PREP (m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) +& (m)) #endifIdem.
Quoted 14 lines
lib/pmu/rte_pmu.c line 41
Quoted 9 lines
lib/pmu/rte_pmu.c new file +9
-0+ +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu; + +/* +* Following __rte_weak functions provide default no-op. +Architectures should override them if +* necessary. +*/ Prefer using per architectures #ifdef.
It is easier to get a simple link error than use weak symbols that make it look like it could work on some arch.
Rationale for that was actually to not break anything. That means you can compile that code for every arch, except on unsupported ones you'll see stream of zeros inside a trace file.
Quoted 9 lines
lib/pmu/rte_pmu.c line 46
Quoted 6 lines
lib/pmu/rte_pmu.c new file +5
-0+ +int +__rte_weak pmu_arch_init(void) +{ +return 0 ;Add a debug log message indicating that this arch does not support PMU.
Prove me wrong but logs are part of eal and unless they are moved to a separate library this lib shouldn't call these APIs. Otherwise we'll introduce dependency on eal which we wanted to avoid in the first place.
Quoted 11 lines
Quoted 11 lines
lib/pmu/rte_pmu.c new file +10
-0+} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { } +
Quoted 9 lines
lib/pmu/rte_pmu.c line 65
Quoted 6 lines
lib/pmu/rte_pmu.c new file +6
-0+static int +get_term_format (const char *name,int *num,uint64_t *mask) { +char *config =NULL ; +char path[PATH_MAX ]; +int high, low, ret; +FILE *fp;Reverse xmas tree when possible.
Okay.
Quoted 7 lines
lib/pmu/rte_pmu.c line 65
Quoted 3 lines
lib/pmu/rte_pmu.c new file +2
-0+ +/* quiesce -Wmaybe-uninitialized warning */ This comment just seems to be a note for yourself. What was the issue exactly?
Generally speaking, compiler thinks that function calling this function may use 'num' even though callee returned an error.
[1/198] Compiling C object lib/librte_pmu.a.p/pmu_rte_pmu.c.o In function _parse_event_,
inlined from _get_event_config_ at ../lib/pmu/rte_pmu.c:157:9:
../lib/pmu/rte_pmu.c:129:23: warning: _num_ may be used uninitialized [-Wmaybe-uninitialized]
129 | config[num] |= FIELD_PREP(mask, val);
| ~~~~~~^~~~~
../lib/pmu/rte_pmu.c: In function _get_event_config_:
../lib/pmu/rte_pmu.c:107:13: note: _num_ was declared here
107 | int num, ret, val;
|Quoted 191 lines
lib/pmu/rte_pmu.c
Quoted 5 lines
lib/pmu/rte_pmu.c new file +4
-0+ *num =0 ; + *mask =0 ; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name,name);
Quoted 60 lines
lib/pmu/rte_pmu.c new file +60
-0+ fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) { +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } +lib/pmu/rte_pmu.c
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+return 0 ; +} + +static int +get_event_config (const char *name,uint64_t config[3 ]) { +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,name);
Quoted 114 lines
lib/pmu/rte_pmu.c new file +114
-0+ fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); } + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) { +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd, +0 ); } + +static int +open_events (struct rte_pmu_event_group *group) { +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) { +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) { +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE , +PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + }
Quoted 9 lines
lib/pmu/rte_pmu.c line 267
Quoted 6 lines
lib/pmu/rte_pmu.c new file +6
-0+ } + + group->enabled = false; +} + +int __rte_noinlineThis symbol is exported out of this library, no need for noinline.
I recall that it was added deliberately because this function was actually being inlined. But given code changed a bit through multiple revisions this may not be necessary anymore.
Quoted 142 lines
Quoted 107 lines
lib/pmu/rte_pmu.c new file +106
-0+rte_pmu_enable_group (void) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next); +rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; } + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) { +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) +{lib/pmu/rte_pmu.c
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+struct rte_pmu_event *event; +char path[PATH_MAX ]; + +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,name);
Quoted 24 lines
lib/pmu/rte_pmu.c new file +24
-0+if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; +
Quoted 9 lines
lib/pmu/rte_pmu.c line 414
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized) {This is an integer so check against 0 explicitly (there may be other cases in this patch, I did not recheck the whole patch).
Okay.
Quoted 93 lines
Quoted 82 lines
lib/pmu/rte_pmu.c new file +50
-0+ rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (!rte_pmu.initialized || --rte_pmu.initialized) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..e360375a0c
lib/pmu/rte_pmu.h new file +26
-0@@ -0,0 +1,205 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> +#include <rte_common.h> +#include <rte_compat.h>lib/pmu/rte_pmu.h line 36
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+#include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8 + +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */
Quoted 6 lines
lib/pmu/rte_pmu.h
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned;One problem for the future is that we have a fixed size fd array.
This number can be increased if needed but rationale for that was PMUs come with relatively small number of hw counters. Here events are grouped together (i.e scheduled together) which means there must be enough hw counters for all events for things to work.
Quoted 2 lines
Do we need to expose this whole structure to the application?
Probably some part of it could have been hidden but all those structures are so small that presumably such partitioning would not bring much to the table.
Quoted 95 lines
lib/pmu/rte_pmu.h line 47
Quoted 8 lines
lib/pmu/rte_pmu.h new file +7
-0+ +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ This is an internal consideration.
Do we need to expose this to the application?lib/pmu/rte_pmu.h
Quoted 12 lines
lib/pmu/rte_pmu.h new file +12
-0+TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ };Idem, do we need to expose this to the application?
Quoted 51 lines
lib/pmu/rte_pmu.h new file +51
-0+ +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own version +*/ #ifndef rte_pmu_pmc_read#define rte_pmu_pmc_read (index) ({ 0; }) +#endif + +/** +* @internal +* +* Read PMU counter. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +__rte_internal +static __rte_always_inline uint64_t +rte_pmu_read_userpage (struct perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier (); + index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +}lib/pmu/rte_pmu.h line 124
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+ +/** +* @internal +* +* Enable group of events on the calling lcore. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_internalUnless I missed something, this symbol is called from
rte_pmu_read()so this makesrte_pmu_read()itself internal.
So external applications won't be able to use the PMU API.
Quoted 2 lines
This can probably be confirmed by adding some call to the PMU API in an examples/.
Good point actually. This was not that obvious when I looked at the patch introducing that change. So in this case it needs to be exported but given app should not call that itself maybe I'll just make the intent clear by renaming it perhaps to __rte_pmu_enable_group() or something alike.
Quoted 114 lines
Quoted 108 lines
lib/pmu/rte_pmu.h new file +81
-0+int +rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. +* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ +__rte_experimental +static __rte_always_inline uint64_t +rte_pmu_read (unsigned int index) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_PMU_H_ */ lib/pmu/version.map new file mode renamed from lib/pmu/version.map +0
-0100644 index 0000000000..50fb0f354e
lib/pmu/version.map new file +20
-1@@ -0,0 +1,20 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +}; + +INTERNAL { + global: + + rte_pmu_enable_group; +}; --Quoted 2 lines
2.34.1
--
David Marchand
PATCH v9 2/4 pmu: support reading ARM PMU events in runtime
In reply to Tomasz DuszynskiAdd support for reading ARM PMU events in runtime.
--- app/test/test_pmu.c | 4 ++ lib/pmu/meson.build | 7 +++ lib/pmu/pmu_arm64.c | 94 +++++++++++++++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 4 ++ lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++ 5 files changed, 139 insertions(+) create mode 100644 lib/pmu/pmu_arm64.c create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
app/test/test_pmu.c +4 -0
@@ -26,6 +26,10 @@ test_pmu_read(void)if (rte_pmu_init () <0 )return TEST_FAILED ; +#if defined(RTE_ARCH_ARM64 ) + event =rte_pmu_add_event ("cpu_cycles" ); +#endif +while (tries--) val +=rte_pmu_read (event);
lib/pmu/meson.build +7 -0
@@ -11,3 +11,10 @@ includes = [global_inc] sources = files('rte_pmu.c') headers = files('rte_pmu.h') +indirect_headers += files( + 'rte_pmu_pmc_arm64.h', +) + +if dpdk_conf.has('RTE_ARCH_ARM64') + sources += files('pmu_arm64.c') +endif
lib/pmu/pmu_arm64.c new file +94 -0
94 lines not on this page · raw message
lib/pmu/rte_pmu.h +4 -0
@@ -26,6 +26,10 @@ extern "C" {#include <rte_compat.h> #include <rte_spinlock.h> +#if defined(RTE_ARCH_ARM64 ) +#include "rte_pmu_pmc_arm64.h" +#endif +/** Maximum number of events in a group */ #define MAX_NUM_GROUP_EVENTS 8
lib/pmu/rte_pmu_pmc_arm64.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell. +*/ +#ifndef _RTE_PMU_PMC_ARM64_H_ +#define _RTE_PMU_PMC_ARM64_H_ + +#include <rte_common.h> + +static __rte_always_inline uint64_t +rte_pmu_pmc_read (int index) +{ +uint64_t val; + +if (index ==31 ) { +/* CPU Cycles (0x11) must be read via pmccntr_el0 */ + asmvolatile ("mrs %0, pmccntr_el0" :"=r" (val)); + }else { + asmvolatile ( +"msr pmselr_el0, %x0\n" +"mrs %0, pmxevcntr_el0\n" + :"=r" (val) + :"rZ" (index) + ); + } + +return val; +} +#define rte_pmu_pmc_read rte_pmu_pmc_read + +#endif /* _RTE_PMU_PMC_ARM64_H_ */
Signature · 1 line
2.34.1
PATCH v9 3/4 pmu: support reading Intel x86_64 PMU events in runtime
In reply to Tomasz DuszynskiAdd support for reading Intel x86_64 PMU events in runtime.
--- app/test/test_pmu.c | 2 ++ lib/pmu/meson.build | 1 + lib/pmu/rte_pmu.h | 2 ++ lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++++++++++++++++++++++++ 4 files changed, 29 insertions(+) create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
app/test/test_pmu.c +2 -0
@@ -28,6 +28,8 @@ test_pmu_read(void)#if defined(RTE_ARCH_ARM64 ) event =rte_pmu_add_event ("cpu_cycles" ); +#elif defined(RTE_ARCH_X86_64 ) + event =rte_pmu_add_event ("cpu-cycles" );#endif while (tries--)
lib/pmu/meson.build +1 -0
@@ -13,6 +13,7 @@ sources = files('rte_pmu.c') headers = files('rte_pmu.h') indirect_headers += files( 'rte_pmu_pmc_arm64.h', + 'rte_pmu_pmc_x86_64.h', ) if dpdk_conf.has('RTE_ARCH_ARM64')
lib/pmu/rte_pmu.h +2 -0
@@ -28,6 +28,8 @@ extern "C" {#if defined(RTE_ARCH_ARM64 )#include "rte_pmu_pmc_arm64.h" +#elif defined(RTE_ARCH_X86_64 ) +#include "rte_pmu_pmc_x86_64.h" #endif /** Maximum number of events in a group */
lib/pmu/rte_pmu_pmc_x86_64.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell. +*/ +#ifndef _RTE_PMU_PMC_X86_64_H_ +#define _RTE_PMU_PMC_X86_64_H_ + +#include <rte_common.h> + +static __rte_always_inline uint64_t +rte_pmu_pmc_read (int index) +{ +uint64_t low, high; + + asmvolatile ( +"rdpmc\n" + :"=a" (low),"=d" (high) + :"c" (index) + ); + +return low | (high <<32 ); +} +#define rte_pmu_pmc_read rte_pmu_pmc_read + +#endif /* _RTE_PMU_PMC_X86_64_H_ */
Signature · 1 line
2.34.1
PATCH v9 4/4 eal: add PMU support to tracing library
In reply to Tomasz DuszynskiIn order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.
--- app/test/test_trace_perf.c | 10 ++++ doc/guides/prog_guide/profile_app.rst | 5 ++ doc/guides/prog_guide/trace_lib.rst | 32 +++++++++++++ lib/eal/common/eal_common_trace.c | 13 ++++- lib/eal/common/eal_common_trace_points.c | 5 ++ lib/eal/include/rte_eal_trace.h | 13 +++++ lib/eal/meson.build | 3 ++ lib/eal/version.map | 1 + lib/pmu/rte_pmu.c | 61 ++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 14 ++++++ lib/pmu/version.map | 1 + 11 files changed, 157 insertions(+), 1 deletion(-)
app/test/test_trace_perf.c +10 -0
@@ -114,6 +114,10 @@ worker_fn_##func(void *arg) \#define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)#define GENERIC_STR rte_eal_trace_generic_str("hello world")#define VOID_FP app_dpdk_test_fp() +#ifdef RTE_EXEC_ENV_LINUX +/* 0 corresponds first event passed via --trace= */ +#define READ_PMU rte_eal_trace_pmu_read(0) +#endif WORKER_DEFINE (GENERIC_VOID)WORKER_DEFINE (GENERIC_U64)
@@ -122,6 +126,9 @@ WORKER_DEFINE(GENERIC_FLOAT) WORKER_DEFINE(GENERIC_DOUBLE) WORKER_DEFINE(GENERIC_STR) WORKER_DEFINE(VOID_FP) +#ifdef RTE_EXEC_ENV_LINUX +WORKER_DEFINE(READ_PMU) +#endif static void run_test (const char *str,lcore_function_t f,struct test_data *data,size_t sz)
@@ -174,6 +181,9 @@ test_trace_perf(void)run_test ("double" , worker_fn_GENERIC_DOUBLE, data, sz);run_test ("string" , worker_fn_GENERIC_STR, data, sz);run_test ("void_fp" , worker_fn_VOID_FP, data, sz); +#ifdef RTE_EXEC_ENV_LINUX +run_test ("read_pmu" , worker_fn_READ_PMU, data, sz); +#endif rte_free (data);return TEST_SUCCESS ;
doc/guides/prog_guide/profile_app.rst +5 -0
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can read specific events directly from application via ``rte_pmu_read()``. +Alternatively tracing library can be used which offers dedicated tracepoint +``rte_eal_trace_pmu_event()``. + +Refer to :doc:`../prog_guide/trace_lib` for more details. + Profiling on x86 ----------------
doc/guides/prog_guide/trace_lib.rst +32 -0
@@ -46,6 +46,7 @@ DPDK tracing library features trace format and is compatible with ``LTTng``. For detailed information, refer to `Common Trace Format <https://diamon.org/ctf/>`_. +- Support reading PMU events on ARM64 and x86-64 (Intel) How to add a tracepoint? ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``. ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using the ``enable_trace_fp`` option for meson build. +PMU tracepoint +-------------- + +Performance measurement unit (PMU) event values can be read from hardware +registers using predefined ``rte_pmu_read`` tracepoint. + +Tracing is enabled via ``--trace`` EAL option by passing both expression +matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression +``e=ev1[,ev2,...]`` matching particular events:: + + --trace='.*pmu.read\|e=cpu_cycles,l1d_cache' + +Event names are available under ``/sys/bus/event_source/devices/PMU/events`` +directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory +containing ``cpus``. + +In contrary to other tracepoints this does not need any extra variables +added to source files. Instead, caller passes index which follows the order of +events specified via ``--trace`` parameter. In the following example index ``0`` +corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``. + +.. code-block:: c + + ... + rte_eal_trace_pmu_read(0); + rte_eal_trace_pmu_read(1); + ... + +PMU tracing support must be explicitly enabled using the ``enable_trace_fp`` +option for meson build. + Event record mode -----------------
lib/eal/common/eal_common_trace.c +12 -1
@@ -11,6 +11,9 @@#include <rte_errno.h> #include <rte_lcore.h> #include <rte_per_lcore.h> +#ifdef RTE_EXEC_ENV_LINUX +#include <rte_pmu.h> +#endif #include <rte_string_fns.h> #include "eal_trace.h"
@@ -71,8 +74,13 @@ eal_trace_init(void) goto free_meta;/* Apply global configurations */ - STAILQ_FOREACH(arg, &trace.args, next) + STAILQ_FOREACH(arg, &trace.args, next) {trace_args_apply (arg->val); +#ifdef RTE_EXEC_ENV_LINUX +if (rte_pmu_init () ==0 ) +rte_pmu_add_events_by_pattern (arg->val); +#endif + }rte_trace_mode_set (trace.mode);
@@ -88,6 +96,9 @@ eal_trace_init(void)void eal_trace_fini (void ) { +#ifdef RTE_EXEC_ENV_LINUX +rte_pmu_fini (); +#endif trace_mem_free ();trace_metadata_destroy ();eal_trace_args_free ();
lib/eal/common/eal_common_trace_points.c +5 -0
@@ -75,3 +75,8 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable, lib.eal.intr.enable) RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable, lib.eal.intr.disable) + +#ifdef RTE_EXEC_ENV_LINUX +RTE_TRACE_POINT_REGISTER (rte_eal_trace_pmu_read, + lib.eal.pmu.read) +#endif
lib/eal/include/rte_eal_trace.h +13 -0
@@ -17,6 +17,9 @@ extern "C" {#include <rte_alarm.h> #include <rte_interrupts.h> +#ifdef RTE_EXEC_ENV_LINUX +#include <rte_pmu.h> +#endif #include <rte_trace_point.h> #include "eal_interrupts.h"
@@ -279,6 +282,16 @@ RTE_TRACE_POINT(rte_trace_point_emit_string (cpuset); ) +#ifdef RTE_EXEC_ENV_LINUX +RTE_TRACE_POINT_FP ( + rte_eal_trace_pmu_read, + RTE_TRACE_POINT_ARGS(unsigned int index), + uint64_t val; + val = rte_pmu_read(index); + rte_trace_point_emit_u64(val); +) +#endif + #ifdef __cplusplus }#endif
lib/eal/meson.build +3 -0
@@ -26,6 +26,9 @@ deps += ['kvargs'] if not is_windows deps += ['telemetry'] endif +if is_linux + deps += ['pmu'] +endif if dpdk_conf.has('RTE_USE_LIBBSD') ext_deps += libbsd endif
lib/eal/version.map +1 -0
@@ -441,6 +441,7 @@ EXPERIMENTAL { rte_thread_join; # added in 23.03 + __rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT rte_thread_set_name; };
lib/pmu/rte_pmu.c +61 -0
67 lines not on this page · raw message
lib/pmu/rte_pmu.h +14 -0
@@ -169,6 +169,20 @@ __rte_experimentalint rte_pmu_add_event (const char *name); +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add events matching pattern to the group of enabled events. +* +* @param pattern +* Pattern e=ev1[,ev2,...] matching events, where evX is a placeholder for an event listed under +* /sys/bus/event_source/devices/pmu/events. +*/ +__rte_experimental +int +rte_pmu_add_events_by_pattern (const char *pattern); +/** * @warning * @b EXPERIMENTAL: this API may change without prior notice
lib/pmu/version.map +1 -0
@@ -8,6 +8,7 @@ EXPERIMENTAL { per_lcore__event_group; rte_pmu; rte_pmu_add_event; + rte_pmu_add_events_by_pattern; rte_pmu_fini; rte_pmu_init; rte_pmu_read;
Signature · 1 line
2.34.1
v10 cover add support for self monitoring
In reply to Tomasz DuszynskiThis series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain advantages when application runs on isolated cores running dedicated tasks.
Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.
By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.
Events are enabled by names, which need to be read from standard location under sysfs i.e
/sys/bus/event_source/devices/PMU/events
where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.
v10:
- check permissions before using counters
- do not use internal symbols in exported functions
- address review comments
v9:
- fix 'maybe-uninitialized' warning reported by CI
v8:
- just rebase series
v7:
- use per-lcore event group instead of global table index by lcore-id
- don't add pmu_autotest to fast tests because due to lack of suported on every arch
v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there
v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra
Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library
MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 68 +++ app/test/test_trace_perf.c | 10 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 17 + doc/guides/prog_guide/trace_lib.rst | 32 ++ doc/guides/rel_notes/release_23_03.rst | 7 + lib/eal/common/eal_common_trace.c | 13 +- lib/eal/common/eal_common_trace_points.c | 5 + lib/eal/include/rte_eal_trace.h | 13 + lib/eal/meson.build | 3 + lib/eal/version.map | 1 + lib/meson.build | 1 + lib/pmu/meson.build | 21 + lib/pmu/pmu_arm64.c | 94 ++++ lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 521 +++++++++++++++++++++++ lib/pmu/rte_pmu.h | 232 ++++++++++ lib/pmu/rte_pmu_pmc_arm64.h | 30 ++ lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++ lib/pmu/version.map | 16 + 23 files changed, 1149 insertions(+), 2 deletions(-) create mode 100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_arm64.c create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h create mode 100644 lib/pmu/version.map
--
2.34.1
PATCH v10 1/4 lib: add generic support for reading PMU events
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
--- MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 62 ++++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 12 + doc/guides/rel_notes/release_23_03.rst | 7 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 460 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 212 ++++++++++++ lib/pmu/version.map | 15 + 13 files changed, 824 insertions(+), 1 deletion(-) create mode 100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------
app/test/meson.build +2 -0
@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [ ['kni_autotest', false, true], ['kvargs_autotest', true, true], ['member_autotest', true, true], + ['pmu_autotest', true, true], ['power_cpufreq_autotest', false, true], ['power_autotest', true, true], ['power_kvm_vm_autotest', false, true],
app/test/test_pmu.c new file +62 -0
62 lines not on this page · raw message
doc/api/doxy-api-index.md +2 -1
@@ -223,7 +223,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), + [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +12 -0
@@ -7,6 +7,18 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on different architectures. +Performance counter based profiling +----------------------------------- + +Majority of architectures support some performance monitoring unit (PMU). +Such unit provides programmable counters that monitor specific events. + +Different tools gather that information, like for example perf. +However, in some scenarios when CPU cores are isolated and run +dedicated tasks interrupting those tasks with perf may be undesirable. + +In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``. + Profiling on x86 ----------------
doc/guides/rel_notes/release_23_03.rst +7 -0
@@ -147,6 +147,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added PMU library.** + + Added a new performance monitoring unit (PMU) library which allows applications + to perform self monitoring activities without depending on external utilities like perf. + After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters + can be stored in CTF format for further analysis. + Removed Items -------------
lib/meson.build +1 -0
@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2023 Marvell International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. +* +* @param config +* Architecture specific event configuration. Consult kernel sources for available options. +*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +460 -0
460 lines not on this page · raw message
lib/pmu/rte_pmu.h new file +212 -0
212 lines not on this page · raw message
lib/pmu/version.map new file +15 -0
@@ -0,0 +1,15 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + __rte_pmu_enable_group; + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +};
--
2.34.1
RE: [PATCH v10 1/4] lib: add generic support for reading PMU events
Quoted 100 lines
-----Original Message-----
From: Tomasz Duszynski <[email protected]>
Sent: Monday, February 13, 2023 7:32 PM
To: [email protected]; [email protected]; Tomasz Duszynski <[email protected]>
Cc: [email protected]; Ruifeng Wang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v10 1/4] lib: add generic support for reading PMU eventsAdd support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>--- MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 62 ++++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 12 + doc/guides/rel_notes/release_23_03.rst | 7 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 460 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 212 ++++++++++++ lib/pmu/version.map | 15 + 13 files changed, 824 insertions(+), 1 deletion(-) create mode 100644
app/test/test_pmu.ccreate mode 100644lib/pmu/meson.buildcreate mode 100644lib/pmu/pmu_private.hcreate mode 100644lib/pmu/rte_pmu.ccreate mode 100644lib/pmu/rte_pmu.hcreate mode 100644lib/pmu/version.mapMAINTAINERS +5
-0@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------app/test/meson.build index f34d19e3c3..6b61b7fc32 renamed from app/test/meson.build +0
-0100644
app/test/meson.build +2
-0@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',@@ -239,6 +240,7 @@ fast_tests = [ ['kni_autotest', false, true], ['kvargs_autotest', true, true], ['member_autotest', true, true], + ['pmu_autotest', true, true], ['power_cpufreq_autotest', false, true], ['power_autotest', true, true], ['power_kvm_vm_autotest', false, true], diff --git a/app/test/test_pmu.c
b/app/test/test_pmu.cnew file mode 100644 index 0000000000..a64564b5f5app/test/test_pmu.c new file +23
-0@@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include "test.h" + +#ifndef RTE_EXEC_ENV_LINUX + +static int +test_pmu (void ) +{ +printf ("pmu_autotest only supported on Linux, skipping test\n" ); +return TEST_SKIPPED ; +} + +#else + +#include <rte_pmu.h> + +static int +test_pmu_read (void ) +{ +const char *name =NULL ;
app/test/test_pmu.c line 33
Quoted 10 lines
app/test/test_pmu.c new file +10
-0+int tries =10 , event; +uint64_t val =0 ; + +if (name ==NULL ) { +printf ("PMU not supported on this arch\n" ); +return TEST_SKIPPED ; + } + +if (rte_pmu_init () <0 ) +return TEST_FAILED ;
Can we return TEST_SKIPPED here?
On aarch64, this feature requires kernel version >= 5.17. CI setups doesn't meet this requirement will start to report failure when running fast_tests.
Quoted 19 lines
app/test/test_pmu.c new file +19
-0+ + event =rte_pmu_add_event (name); +while (tries--) + val +=rte_pmu_read (event); + +rte_pmu_fini (); + +return val ?TEST_SUCCESS :TEST_FAILED ; } + +static struct unit_test_suite pmu_tests = { + .suite_name ="pmu autotest" , + .setup =NULL , + .teardown =NULL , + .unit_test_cases = { +TEST_CASE (test_pmu_read), +TEST_CASES_END () + } +}; +
<snip>
RE: [PATCH v10 1/4] lib: add generic support for reading PMU events
[...]
Quoted 8 lines
app/test/test_pmu.c line 33
Quoted 3 lines
app/test/test_pmu.c new file +3
-0+ +if (rte_pmu_init () <0 ) +return TEST_FAILED ;Can we return
TEST_SKIPPEDhere?
On aarch64, this feature requires kernel version >= 5.17. CI setups doesn't meet this requirement will start to report failure when runningfast_tests.
Okay. I think that's good enough for CI.
Quoted 20 lines
Quoted 19 lines
app/test/test_pmu.c new file +19
-0+ + event =rte_pmu_add_event (name); +while (tries--) + val +=rte_pmu_read (event); + +rte_pmu_fini (); + +return val ?TEST_SUCCESS :TEST_FAILED ; } + +static struct unit_test_suite pmu_tests = { + .suite_name ="pmu autotest" , + .setup =NULL , + .teardown =NULL , + .unit_test_cases = { +TEST_CASE (test_pmu_read), +TEST_CASES_END () + } +}; +<snip>
PATCH v10 2/4 pmu: support reading ARM PMU events in runtime
In reply to Tomasz DuszynskiAdd support for reading ARM PMU events in runtime.
--- app/test/test_pmu.c | 4 ++ lib/pmu/meson.build | 7 +++ lib/pmu/pmu_arm64.c | 94 +++++++++++++++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 4 ++ lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++ 5 files changed, 139 insertions(+) create mode 100644 lib/pmu/pmu_arm64.c create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
app/test/test_pmu.c +4 -0
@@ -24,6 +24,10 @@ test_pmu_read(void)int tries =10 , event;uint64_t val =0 ; +#if defined(RTE_ARCH_ARM64 ) + name ="cpu_cycles" ; +#endif +if (name ==NULL ) {printf ("PMU not supported on this arch\n" );return TEST_SKIPPED ;
lib/pmu/meson.build +7 -0
@@ -11,3 +11,10 @@ includes = [global_inc] sources = files('rte_pmu.c') headers = files('rte_pmu.h') +indirect_headers += files( + 'rte_pmu_pmc_arm64.h', +) + +if dpdk_conf.has('RTE_ARCH_ARM64') + sources += files('pmu_arm64.c') +endif
lib/pmu/pmu_arm64.c new file +94 -0
94 lines not on this page · raw message
lib/pmu/rte_pmu.h +4 -0
@@ -26,6 +26,10 @@ extern "C" {#include <rte_compat.h> #include <rte_spinlock.h> +#if defined(RTE_ARCH_ARM64 ) +#include "rte_pmu_pmc_arm64.h" +#endif +/** Maximum number of events in a group */ #define MAX_NUM_GROUP_EVENTS 8
lib/pmu/rte_pmu_pmc_arm64.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell. +*/ +#ifndef _RTE_PMU_PMC_ARM64_H_ +#define _RTE_PMU_PMC_ARM64_H_ + +#include <rte_common.h> + +static __rte_always_inline uint64_t +rte_pmu_pmc_read (int index) +{ +uint64_t val; + +if (index ==31 ) { +/* CPU Cycles (0x11) must be read via pmccntr_el0 */ + asmvolatile ("mrs %0, pmccntr_el0" :"=r" (val)); + }else { + asmvolatile ( +"msr pmselr_el0, %x0\n" +"mrs %0, pmxevcntr_el0\n" + :"=r" (val) + :"rZ" (index) + ); + } + +return val; +} +#define rte_pmu_pmc_read rte_pmu_pmc_read + +#endif /* _RTE_PMU_PMC_ARM64_H_ */
Signature · 1 line
2.34.1
RE: [PATCH v10 2/4] pmu: support reading ARM PMU events in runtime
Quoted 15 lines
-----Original Message-----
From: Tomasz Duszynski <[email protected]>
Sent: Monday, February 13, 2023 7:32 PM
To: [email protected]; Tomasz Duszynski <[email protected]>; Ruifeng Wang <[email protected]>
Cc: [email protected]; Ruifeng Wang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v10 2/4] pmu: support reading ARM PMU events in runtimeAdd support for reading ARM PMU events in runtime.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>
Quoted 10 lines
--- app/test/test_pmu.c | 4 ++ lib/pmu/meson.build | 7 +++ lib/pmu/pmu_arm64.c | 94 +++++++++++++++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 4 ++ lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++ 5 files changed, 139 insertions(+) create mode 100644 lib/pmu/pmu_arm64.c create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
Thanks.
PATCH v10 3/4 pmu: support reading Intel x86_64 PMU events in runtime
In reply to Tomasz DuszynskiAdd support for reading Intel x86_64 PMU events in runtime.
--- app/test/test_pmu.c | 2 ++ lib/pmu/meson.build | 1 + lib/pmu/rte_pmu.h | 2 ++ lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++++++++++++++++++++++++ 4 files changed, 29 insertions(+) create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
app/test/test_pmu.c +2 -0
@@ -26,6 +26,8 @@ test_pmu_read(void)#if defined(RTE_ARCH_ARM64 ) name ="cpu_cycles" ; +#elif defined(RTE_ARCH_X86_64 ) + name ="cpu-cycles" ;#endif if (name ==NULL ) {
lib/pmu/meson.build +1 -0
@@ -13,6 +13,7 @@ sources = files('rte_pmu.c') headers = files('rte_pmu.h') indirect_headers += files( 'rte_pmu_pmc_arm64.h', + 'rte_pmu_pmc_x86_64.h', ) if dpdk_conf.has('RTE_ARCH_ARM64')
lib/pmu/rte_pmu.h +2 -0
@@ -28,6 +28,8 @@ extern "C" {#if defined(RTE_ARCH_ARM64 )#include "rte_pmu_pmc_arm64.h" +#elif defined(RTE_ARCH_X86_64 ) +#include "rte_pmu_pmc_x86_64.h" #endif /** Maximum number of events in a group */
lib/pmu/rte_pmu_pmc_x86_64.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell. +*/ +#ifndef _RTE_PMU_PMC_X86_64_H_ +#define _RTE_PMU_PMC_X86_64_H_ + +#include <rte_common.h> + +static __rte_always_inline uint64_t +rte_pmu_pmc_read (int index) +{ +uint64_t low, high; + + asmvolatile ( +"rdpmc\n" + :"=a" (low),"=d" (high) + :"c" (index) + ); + +return low | (high <<32 ); +} +#define rte_pmu_pmc_read rte_pmu_pmc_read + +#endif /* _RTE_PMU_PMC_X86_64_H_ */
Signature · 1 line
2.34.1
PATCH v10 4/4 eal: add PMU support to tracing library
In reply to Tomasz DuszynskiIn order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.
--- app/test/test_trace_perf.c | 10 ++++ doc/guides/prog_guide/profile_app.rst | 5 ++ doc/guides/prog_guide/trace_lib.rst | 32 +++++++++++++ lib/eal/common/eal_common_trace.c | 13 ++++- lib/eal/common/eal_common_trace_points.c | 5 ++ lib/eal/include/rte_eal_trace.h | 13 +++++ lib/eal/meson.build | 3 ++ lib/eal/version.map | 1 + lib/pmu/rte_pmu.c | 61 ++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 14 ++++++ lib/pmu/version.map | 1 + 11 files changed, 157 insertions(+), 1 deletion(-)
app/test/test_trace_perf.c +10 -0
@@ -114,6 +114,10 @@ worker_fn_##func(void *arg) \#define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)#define GENERIC_STR rte_eal_trace_generic_str("hello world")#define VOID_FP app_dpdk_test_fp() +#ifdef RTE_EXEC_ENV_LINUX +/* 0 corresponds first event passed via --trace= */ +#define READ_PMU rte_eal_trace_pmu_read(0) +#endif WORKER_DEFINE (GENERIC_VOID)WORKER_DEFINE (GENERIC_U64)
@@ -122,6 +126,9 @@ WORKER_DEFINE(GENERIC_FLOAT) WORKER_DEFINE(GENERIC_DOUBLE) WORKER_DEFINE(GENERIC_STR) WORKER_DEFINE(VOID_FP) +#ifdef RTE_EXEC_ENV_LINUX +WORKER_DEFINE(READ_PMU) +#endif static void run_test (const char *str,lcore_function_t f,struct test_data *data,size_t sz)
@@ -174,6 +181,9 @@ test_trace_perf(void)run_test ("double" , worker_fn_GENERIC_DOUBLE, data, sz);run_test ("string" , worker_fn_GENERIC_STR, data, sz);run_test ("void_fp" , worker_fn_VOID_FP, data, sz); +#ifdef RTE_EXEC_ENV_LINUX +run_test ("read_pmu" , worker_fn_READ_PMU, data, sz); +#endif rte_free (data);return TEST_SUCCESS ;
doc/guides/prog_guide/profile_app.rst +5 -0
@@ -19,6 +19,11 @@ dedicated tasks interrupting those tasks with perf may be undesirable. In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``. +Alternatively tracing library can be used which offers dedicated tracepoint +``rte_eal_trace_pmu_event()``. + +Refer to :doc:`../prog_guide/trace_lib` for more details. + Profiling on x86 ----------------
doc/guides/prog_guide/trace_lib.rst +32 -0
@@ -46,6 +46,7 @@ DPDK tracing library features trace format and is compatible with ``LTTng``. For detailed information, refer to `Common Trace Format <https://diamon.org/ctf/>`_. +- Support reading PMU events on ARM64 and x86-64 (Intel) How to add a tracepoint? ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``. ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using the ``enable_trace_fp`` option for meson build. +PMU tracepoint +-------------- + +Performance monitoring unit (PMU) event values can be read from hardware +registers using predefined ``rte_pmu_read`` tracepoint. + +Tracing is enabled via ``--trace`` EAL option by passing both expression +matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression +``e=ev1[,ev2,...]`` matching particular events:: + + --trace='.*pmu.read\|e=cpu_cycles,l1d_cache' + +Event names are available under ``/sys/bus/event_source/devices/PMU/events`` +directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory +containing ``cpus``. + +In contrary to other tracepoints this does not need any extra variables +added to source files. Instead, caller passes index which follows the order of +events specified via ``--trace`` parameter. In the following example index ``0`` +corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``. + +.. code-block:: c + + ... + rte_eal_trace_pmu_read(0); + rte_eal_trace_pmu_read(1); + ... + +PMU tracing support must be explicitly enabled using the ``enable_trace_fp`` +option for meson build. + Event record mode -----------------
lib/eal/common/eal_common_trace.c +12 -1
@@ -11,6 +11,9 @@#include <rte_errno.h> #include <rte_lcore.h> #include <rte_per_lcore.h> +#ifdef RTE_EXEC_ENV_LINUX +#include <rte_pmu.h> +#endif #include <rte_string_fns.h> #include "eal_trace.h"
@@ -71,8 +74,13 @@ eal_trace_init(void) goto free_meta;/* Apply global configurations */ - STAILQ_FOREACH(arg, &trace.args, next) + STAILQ_FOREACH(arg, &trace.args, next) {trace_args_apply (arg->val); +#ifdef RTE_EXEC_ENV_LINUX +if (rte_pmu_init () ==0 ) +rte_pmu_add_events_by_pattern (arg->val); +#endif + }rte_trace_mode_set (trace.mode);
@@ -88,6 +96,9 @@ eal_trace_init(void)void eal_trace_fini (void ) { +#ifdef RTE_EXEC_ENV_LINUX +rte_pmu_fini (); +#endif trace_mem_free ();trace_metadata_destroy ();eal_trace_args_free ();
lib/eal/common/eal_common_trace_points.c +5 -0
@@ -77,3 +77,8 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable, lib.eal.intr.enable) RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable, lib.eal.intr.disable) + +#ifdef RTE_EXEC_ENV_LINUX +RTE_TRACE_POINT_REGISTER (rte_eal_trace_pmu_read, + lib.eal.pmu.read) +#endif
lib/eal/include/rte_eal_trace.h +13 -0
@@ -17,6 +17,9 @@ extern "C" {#include <rte_alarm.h> #include <rte_interrupts.h> +#ifdef RTE_EXEC_ENV_LINUX +#include <rte_pmu.h> +#endif #include <rte_trace_point.h> #include "eal_interrupts.h"
@@ -285,6 +288,16 @@ RTE_TRACE_POINT(rte_trace_point_emit_string (cpuset); ) +#ifdef RTE_EXEC_ENV_LINUX +RTE_TRACE_POINT_FP ( + rte_eal_trace_pmu_read, + RTE_TRACE_POINT_ARGS(unsigned int index), + uint64_t val; + val = rte_pmu_read(index); + rte_trace_point_emit_u64(val); +) +#endif + #ifdef __cplusplus }#endif
lib/eal/meson.build +3 -0
@@ -26,6 +26,9 @@ deps += ['kvargs'] if not is_windows deps += ['telemetry'] endif +if is_linux + deps += ['pmu'] +endif if dpdk_conf.has('RTE_USE_LIBBSD') ext_deps += libbsd endif
lib/eal/version.map +1 -0
@@ -441,6 +441,7 @@ EXPERIMENTAL { rte_thread_join; # added in 23.03 + __rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT rte_lcore_register_usage_cb; rte_thread_create_control; rte_thread_set_name;
lib/pmu/rte_pmu.c +61 -0
67 lines not on this page · raw message
lib/pmu/rte_pmu.h +14 -0
@@ -176,6 +176,20 @@ __rte_experimentalint rte_pmu_add_event (const char *name); +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add events matching pattern to the group of enabled events. +* +* @param pattern +* Pattern e=ev1[,ev2,...] matching events, where evX is a placeholder for an event listed under +* /sys/bus/event_source/devices/pmu/events. +*/ +__rte_experimental +int +rte_pmu_add_events_by_pattern (const char *pattern); +/** * @warning * @b EXPERIMENTAL: this API may change without prior notice
lib/pmu/version.map +1 -0
@@ -9,6 +9,7 @@ EXPERIMENTAL { per_lcore__event_group; rte_pmu; rte_pmu_add_event; + rte_pmu_add_events_by_pattern; rte_pmu_fini; rte_pmu_init; rte_pmu_read;
Signature · 1 line
2.34.1
v11 cover add support for self monitoring
In reply to Tomasz DuszynskiThis series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain advantages when application runs on isolated cores running dedicated tasks.
Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.
By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.
Events are enabled by names, which need to be read from standard location under sysfs i.e
/sys/bus/event_source/devices/PMU/events
where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.
v11:
- skip fast test in case init fails
v10:
- check permissions before using counters
- do not use internal symbols in exported functions
- address review comments
v9:
- fix 'maybe-uninitialized' warning reported by CI
v8:
- just rebase series
v7:
- use per-lcore event group instead of global table index by lcore-id
- don't add pmu_autotest to fast tests because due to lack of suported on every arch
v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there
v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra
Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library
MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 68 +++ app/test/test_trace_perf.c | 10 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 17 + doc/guides/prog_guide/trace_lib.rst | 32 ++ doc/guides/rel_notes/release_23_03.rst | 7 + lib/eal/common/eal_common_trace.c | 13 +- lib/eal/common/eal_common_trace_points.c | 5 + lib/eal/include/rte_eal_trace.h | 13 + lib/eal/meson.build | 3 + lib/eal/version.map | 1 + lib/meson.build | 1 + lib/pmu/meson.build | 21 + lib/pmu/pmu_arm64.c | 94 ++++ lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 521 +++++++++++++++++++++++ lib/pmu/rte_pmu.h | 232 ++++++++++ lib/pmu/rte_pmu_pmc_arm64.h | 30 ++ lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++ lib/pmu/version.map | 16 + 23 files changed, 1149 insertions(+), 2 deletions(-) create mode 100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_arm64.c create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h create mode 100644 lib/pmu/version.map
--
2.34.1
PATCH v11 1/4 lib: add generic support for reading PMU events
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
--- MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 62 ++++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 12 + doc/guides/rel_notes/release_23_03.rst | 7 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 460 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 212 ++++++++++++ lib/pmu/version.map | 15 + 13 files changed, 824 insertions(+), 1 deletion(-) create mode 100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------
app/test/meson.build +2 -0
@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',
@@ -239,6 +240,7 @@ fast_tests = [ ['kni_autotest', false, true], ['kvargs_autotest', true, true], ['member_autotest', true, true], + ['pmu_autotest', true, true], ['power_cpufreq_autotest', false, true], ['power_autotest', true, true], ['power_kvm_vm_autotest', false, true],
app/test/test_pmu.c new file +62 -0
62 lines not on this page · raw message
doc/api/doxy-api-index.md +2 -1
@@ -223,7 +223,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), + [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +12 -0
@@ -7,6 +7,18 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on different architectures. +Performance counter based profiling +----------------------------------- + +Majority of architectures support some performance monitoring unit (PMU). +Such unit provides programmable counters that monitor specific events. + +Different tools gather that information, like for example perf. +However, in some scenarios when CPU cores are isolated and run +dedicated tasks interrupting those tasks with perf may be undesirable. + +In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``. + Profiling on x86 ----------------
doc/guides/rel_notes/release_23_03.rst +7 -0
@@ -147,6 +147,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added PMU library.** + + Added a new performance monitoring unit (PMU) library which allows applications + to perform self monitoring activities without depending on external utilities like perf. + After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters + can be stored in CTF format for further analysis. + Removed Items -------------
lib/meson.build +1 -0
@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2023 Marvell International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. +* +* @param config +* Architecture specific event configuration. Consult kernel sources for available options. +*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +460 -0
460 lines not on this page · raw message
lib/pmu/rte_pmu.h new file +212 -0
212 lines not on this page · raw message
lib/pmu/version.map new file +15 -0
@@ -0,0 +1,15 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + __rte_pmu_enable_group; + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +};
Signature · 1 line
2.34.1
Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
16/02/2023 17:54, Tomasz Duszynski пишет:
Quoted 292 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>--- MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 62 ++++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 12 + doc/guides/rel_notes/release_23_03.rst | 7 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 460 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 212 ++++++++++++ lib/pmu/version.map | 15 + 13 files changed, 824 insertions(+), 1 deletion(-)create mode 100644
app/test/test_pmu.c
create mode 100644lib/pmu/meson.build
create mode 100644lib/pmu/pmu_private.h
create mode 100644lib/pmu/rte_pmu.c
create mode 100644lib/pmu/rte_pmu.h
create mode 100644lib/pmu/version.mapMAINTAINERS +5
-0@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------app/test/meson.build +2
-0@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',@@ -239,6 +240,7 @@ fast_tests = [ ['kni_autotest', false, true], ['kvargs_autotest', true, true], ['member_autotest', true, true], + ['pmu_autotest', true, true], ['power_cpufreq_autotest', false, true], ['power_autotest', true, true], ['power_kvm_vm_autotest', false, true],app/test/test_pmu.c new file +62
-0@@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include "test.h" + +#ifndef RTE_EXEC_ENV_LINUX + +static int +test_pmu (void ) +{ +printf ("pmu_autotest only supported on Linux, skipping test\n" ); +return TEST_SKIPPED ; +} + +#else + +#include <rte_pmu.h> + +static int +test_pmu_read (void ) +{ +const char *name =NULL ; +int tries =10 , event; +uint64_t val =0 ; + +if (name ==NULL ) { +printf ("PMU not supported on this arch\n" ); +return TEST_SKIPPED ; + } + +if (rte_pmu_init () <0 ) +return TEST_SKIPPED ; + + event =rte_pmu_add_event (name); +while (tries--) + val +=rte_pmu_read (event); + +rte_pmu_fini (); + +return val ?TEST_SUCCESS :TEST_FAILED ; +} + +static struct unit_test_suite pmu_tests = { + .suite_name ="pmu autotest" , + .setup =NULL , + .teardown =NULL , + .unit_test_cases = { +TEST_CASE (test_pmu_read), +TEST_CASES_END () + } +}; + +static int +test_pmu (void ) +{ +return unit_test_suite_runner (&pmu_tests); +} + +#endif /* RTE_EXEC_ENV_LINUX */ + +REGISTER_TEST_COMMAND (pmu_autotest, test_pmu);doc/api/doxy-api-index.md +2
-1@@ -223,7 +223,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), + [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),doc/api/doxy-api.conf.in +1
-0@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \doc/guides/prog_guide/profile_app.rst +12
-0@@ -7,6 +7,18 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on different architectures. +Performance counter based profiling +----------------------------------- + +Majority of architectures support some performance monitoring unit (PMU). +Such unit provides programmable counters that monitor specific events. + +Different tools gather that information, like for example perf. +However, in some scenarios when CPU cores are isolated and run +dedicated tasks interrupting those tasks with perf may be undesirable. + +In such cases, an application can use the PMU library to read such events via ``rte_pmu_read()``. + Profiling on x86 ----------------doc/guides/rel_notes/release_23_03.rst +7
-0@@ -147,6 +147,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added PMU library.** + + Added a new performance monitoring unit (PMU) library which allows applications + to perform self monitoring activities without depending on external utilities like perf. + After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters + can be stored in CTF format for further analysis. + Removed Items -------------lib/meson.build +1
-0@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ringlib/pmu/meson.build new file +13
-0@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2023 Marvell International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')lib/pmu/pmu_private.h new file +32
-0@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. +* +* @param config +* Architecture specific event configuration. Consult kernel sources for available options. +*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */ lib/pmu/rte_pmu.c new file +15
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h>
lib/pmu/rte_pmu.c line 25
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+ +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> + +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?
lib/pmu/rte_pmu.c line 31
Quoted 6 lines
lib/pmu/rte_pmu.c new file +6
-0+ +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h))))) +#define FIELD_PREP (m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu;
Do we really need struct declaration here?
Quoted 520 lines
lib/pmu/rte_pmu.c new file +428
-0+/* +* Following __rte_weak functions provide default no-op. Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) +{ +} + +static int +get_term_format (const char *name,int *num,uint64_t *mask) +{ +char path[PATH_MAX ]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/format/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) +{ +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +} + +static int +get_event_config (const char *name,uint64_t config[3 ]) +{ +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) +{ +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); +} + +static int +open_events (struct rte_pmu_event_group *group) +{ +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) +{ +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) +{ +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE ,PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next); +rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; +} + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) +{ +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) +{ +struct rte_pmu_event *event; +char path[PATH_MAX ]; + +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); +if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file +86
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> +#include <rte_common.h> +#include <rte_compat.h> +#include <rte_spinlock.h> + +/** Maximum number of events in a group */ +#define MAX_NUM_GROUP_EVENTS 8 + +/** + * A structure describing a group of events. + */ +struct rte_pmu_event_group { + struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */ + int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */ + bool enabled; /**< true if group was enabled on particular lcore */ + TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ +} __rte_cache_aligned; + +/** + * A structure describing an event. + */ +struct rte_pmu_event { + char *name; /**< name of an event */ + unsigned int index; /**< event index into fds/mmap_pages */ + TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ +}; + +/** + * A PMU state container. + */ +struct rte_pmu { + char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */ + rte_spinlock_t lock; /**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ + unsigned int num_group_events; /**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ + unsigned int initialized; /**< initialization counter */ +}; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own version */ +#ifndef rte_pmu_pmc_read +#define rte_pmu_pmc_read(index) ({ 0; }) +#endif + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Read PMU counter. + * + * @warning This should be not called directly. + * + * @param pc + * Pointer to the mmapped user page. + * @return + * Counter value read from hardware. + */
lib/pmu/rte_pmu.h line 96
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) +{ +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();
Are you sure that compiler_barrier() is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?
Quoted 137 lines
lib/pmu/rte_pmu.h new file +116
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) + return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. +* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ +__rte_experimental +static __rte_always_inline uint64_t +rte_pmu_read (unsigned int index) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_PMU_H_ */ lib/pmu/version.map new file +15
-0@@ -0,0 +1,15 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + __rte_pmu_enable_group; + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +};
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 308 lines
-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Friday, February 17, 2023 12:51 AM
To: [email protected]
Subject: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU eventsExternal Email
----------------------------------------------------------------------
16/02/2023 17:54, Tomasz Duszynski пишет:Quoted 168 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>--- MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 62 ++++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 12 + doc/guides/rel_notes/release_23_03.rst | 7 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 460 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 212 ++++++++++++ lib/pmu/version.map | 15 + 13 files changed, 824 insertions(+), 1 deletion(-)create mode 100644
app/test/test_pmu.c
create mode 100644lib/pmu/meson.build
create mode 100644lib/pmu/pmu_private.h
create mode 100644lib/pmu/rte_pmu.c
create mode 100644lib/pmu/rte_pmu.h
create mode 100644lib/pmu/version.mapMAINTAINERS index 3495946d0f..d37f242120 renamed from MAINTAINERS +0
-0100644
MAINTAINERS +5
-0@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------app/test/meson.build index renamed from app/test/meson.build +0
-0
f34d19e3c3..6b61b7fc32100644app/test/meson.build +2
-0@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',@@ -239,6 +240,7 @@ fast_tests = [ ['kni_autotest', false, true], ['kvargs_autotest', true, true], ['member_autotest', true, true], + ['pmu_autotest', true, true], ['power_cpufreq_autotest', false, true], ['power_autotest', true, true], ['power_kvm_vm_autotest', false, true], diff --git
a/app/test/test_pmu.cb/app/test/test_pmu.cnew file mode 100644 index 0000000000..c257638e8bapp/test/test_pmu.c new file +61
-0@@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include "test.h" + +#ifndef RTE_EXEC_ENV_LINUX + +static int +test_pmu (void ) +{ +printf ("pmu_autotest only supported on Linux, skipping test\n" ); +return TEST_SKIPPED ; +} + +#else + +#include <rte_pmu.h> + +static int +test_pmu_read (void ) +{ +const char *name =NULL ; +int tries =10 , event; +uint64_t val =0 ; + +if (name ==NULL ) { +printf ("PMU not supported on this arch\n" ); +return TEST_SKIPPED ; + } + +if (rte_pmu_init () <0 ) +return TEST_SKIPPED ; + + event =rte_pmu_add_event (name); +while (tries--) + val +=rte_pmu_read (event); + +rte_pmu_fini (); + +return val ?TEST_SUCCESS :TEST_FAILED ; } + +static struct unit_test_suite pmu_tests = { + .suite_name ="pmu autotest" , + .setup =NULL , + .teardown =NULL , + .unit_test_cases = { +TEST_CASE (test_pmu_read), +TEST_CASES_END () + } +}; + +static int +test_pmu (void ) +{ +return unit_test_suite_runner (&pmu_tests); +} + +#endif /* RTE_EXEC_ENV_LINUX */ + +REGISTER_TEST_COMMAND (pmu_autotest, test_pmu);doc/api/doxy-api-index.md +1
-1@@ -223,7 +223,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0
-0
e859426099..350b5a8c94100644doc/api/doxy-api.conf.in +1
-0@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \ diff --git
a/doc/guides/prog_guide/profile_app.rstb/doc/guides/prog_guide/profile_app.rst
index14292d4c25..89e38cd301100644doc/guides/prog_guide/profile_app.rst +1
-0@@ -7,6 +7,18 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on different architectures. +Performance counter based profilingdoc/guides/prog_guide/profile_app.rst Profile Your Application · line 20
Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +10
-0+----------------------------------- + +Majority of architectures support some performance monitoring unit (PMU). +Such unit provides programmable counters that monitor specific events. + +Different tools gather that information, like for example perf. +However, in some scenarios when CPU cores are isolated and run +dedicated tasks interrupting those tasks with perf may be undesirable. + +In such cases, an application can use the PMU library to read such events via``rte_pmu_read()``.
Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +1
-0+ Profiling on x86 ----------------doc/guides/rel_notes/release_23_03.rst +0
-0doc/guides/rel_notes/release_23_03.rst New Features · line 153
Quoted 7 lines
doc/guides/rel_notes/release_23_03.rst +4
-0@@ -147,6 +147,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added PMU library.** + + Added a new performance monitoring unit (PMU) library which allows + applications to perform self monitoring activities without depending on external utilitieslike perf.
Quoted 99 lines
doc/guides/rel_notes/release_23_03.rst +3
-0+ After integration with :doc:`../prog_guide/trace_lib` data gathered + from hardware counters can be stored in CTF format for further analysis. + Removed Items -------------lib/meson.build index renamed from lib/meson.build +0
-0
450c061d2b..8a42d45d20100644lib/meson.build +1
-0@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ring diff --gita/lib/pmu/meson.build b/lib/pmu/meson.build new file mode 100644 index 0000000000..a4160b494e
lib/pmu/meson.build new file +13
-0@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(C) 2023 Marvell +International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')lib/pmu/pmu_private.h new file renamed from lib/pmu/pmu_private.h +0
-0mode 100644 index 0000000000..b9f8c1ddc8
lib/pmu/pmu_private.h new file +32
-0@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. +* +* @param config +* Architecture specific event configuration. Consult kernel sources for available options. +*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */ lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..950f999cb7
lib/pmu/rte_pmu.c new file +21
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h>
Quoted 9 lines
lib/pmu/rte_pmu.c line 25
Quoted 4 lines
lib/pmu/rte_pmu.c new file +4
-0+ +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?
There are not any plans to support that elsewhere currently so flat directory structure is good enough.
Quoted 10 lines
lib/pmu/rte_pmu.c line 31
Quoted 7 lines
lib/pmu/rte_pmu.c new file +7
-0+ +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> +((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) << +(__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu;Do we really need struct declaration here?
What’s the problem with this placement precisely?
Quoted 512 lines
Quoted 512 lines
lib/pmu/rte_pmu.c new file +418
-0+/* +* Following __rte_weak functions provide default no-op. +Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { } + +static int +get_term_format (const char *name,int *num,uint64_t *mask) { +char path[PATH_MAX ]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/format/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) { +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +} + +static int +get_event_config (const char *name,uint64_t config[3 ]) { +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) { +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); } + +static int +open_events (struct rte_pmu_event_group *group) { +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) { +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) { +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE ,PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next); +rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; +} + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) { +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) +{ +struct rte_pmu_event *event; +char path[PATH_MAX ]; + +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); +if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..6b664c3336
lib/pmu/rte_pmu.h new file +88
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> +#include <rte_common.h> +#include <rte_compat.h> +#include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8 + +/** + * A structure describing a group of events. + */ +struct rte_pmu_event_group { + struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */ + int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */ + bool enabled; /**< true if group was enabled on particular lcore */ + TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ } +__rte_cache_aligned; + +/** + * A structure describing an event. + */ +struct rte_pmu_event { + char *name; /**< name of an event */ + unsigned int index; /**< event index into fds/mmap_pages */ + TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ }; + +/** + * A PMU state container. + */ +struct rte_pmu { + char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */ + rte_spinlock_t lock; /**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ + unsigned int num_group_events; /**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ + unsigned int initialized; /**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own version +*/ #ifndef rte_pmu_pmc_read #define rte_pmu_pmc_read(index) ({ 0; }) +#endif + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Read PMU counter. + * + * @warning This should be not called directly. + * + * @param pc + * Pointer to the mmapped user page. + * @return + * Counter value read from hardware. + */ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) { + uint64_t width, offset; + uint32_t seq, index; + int64_t pmc;
Quoted 9 lines
lib/pmu/rte_pmu.h line 96
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?
It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Quoted 136 lines
Quoted 136 lines
lib/pmu/rte_pmu.h new file +116
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) + return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. +* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ +__rte_experimental +static __rte_always_inline uint64_t +rte_pmu_read (unsigned int index) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_PMU_H_ */ lib/pmu/version.map new file mode renamed from lib/pmu/version.map +0
-0100644 index 0000000000..39a4f279c1
lib/pmu/version.map new file +15
-0@@ -0,0 +1,15 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + __rte_pmu_enable_group; + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +};
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 320 lines
Quoted 296 lines
Quoted 168 lines
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>--- MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 62 ++++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 12 + doc/guides/rel_notes/release_23_03.rst | 7 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 460 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 212 ++++++++++++ lib/pmu/version.map | 15 + 13 files changed, 824 insertions(+), 1 deletion(-)create mode 100644
app/test/test_pmu.c
create mode 100644lib/pmu/meson.build
create mode 100644lib/pmu/pmu_private.h
create mode 100644lib/pmu/rte_pmu.c
create mode 100644lib/pmu/rte_pmu.h
create mode 100644lib/pmu/version.mapMAINTAINERS index 3495946d0f..d37f242120 renamed from MAINTAINERS +0
-0100644
MAINTAINERS +5
-0@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------app/test/meson.build index renamed from app/test/meson.build +0
-0
f34d19e3c3..6b61b7fc32100644app/test/meson.build +2
-0@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',@@ -239,6 +240,7 @@ fast_tests = [ ['kni_autotest', false, true], ['kvargs_autotest', true, true], ['member_autotest', true, true], + ['pmu_autotest', true, true], ['power_cpufreq_autotest', false, true], ['power_autotest', true, true], ['power_kvm_vm_autotest', false, true], diff --git
a/app/test/test_pmu.cb/app/test/test_pmu.cnew file mode 100644 index 0000000000..c257638e8bapp/test/test_pmu.c new file +61
-0@@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include "test.h" + +#ifndef RTE_EXEC_ENV_LINUX + +static int +test_pmu (void ) +{ +printf ("pmu_autotest only supported on Linux, skipping test\n" ); +return TEST_SKIPPED ; +} + +#else + +#include <rte_pmu.h> + +static int +test_pmu_read (void ) +{ +const char *name =NULL ; +int tries =10 , event; +uint64_t val =0 ; + +if (name ==NULL ) { +printf ("PMU not supported on this arch\n" ); +return TEST_SKIPPED ; + } + +if (rte_pmu_init () <0 ) +return TEST_SKIPPED ; + + event =rte_pmu_add_event (name); +while (tries--) + val +=rte_pmu_read (event); + +rte_pmu_fini (); + +return val ?TEST_SUCCESS :TEST_FAILED ; } + +static struct unit_test_suite pmu_tests = { + .suite_name ="pmu autotest" , + .setup =NULL , + .teardown =NULL , + .unit_test_cases = { +TEST_CASE (test_pmu_read), +TEST_CASES_END () + } +}; + +static int +test_pmu (void ) +{ +return unit_test_suite_runner (&pmu_tests); +} + +#endif /* RTE_EXEC_ENV_LINUX */ + +REGISTER_TEST_COMMAND (pmu_autotest, test_pmu);doc/api/doxy-api-index.md +1
-1@@ -223,7 +223,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0
-0
e859426099..350b5a8c94100644doc/api/doxy-api.conf.in +1
-0@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \ diff --git
a/doc/guides/prog_guide/profile_app.rstb/doc/guides/prog_guide/profile_app.rst
index14292d4c25..89e38cd301100644doc/guides/prog_guide/profile_app.rst +1
-0@@ -7,6 +7,18 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on different architectures. +Performance counter based profilingdoc/guides/prog_guide/profile_app.rst Profile Your Application · line 20
Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +10
-0+----------------------------------- + +Majority of architectures support some performance monitoring unit (PMU). +Such unit provides programmable counters that monitor specific events. + +Different tools gather that information, like for example perf. +However, in some scenarios when CPU cores are isolated and run +dedicated tasks interrupting those tasks with perf may be undesirable. + +In such cases, an application can use the PMU library to read such events via``rte_pmu_read()``.
Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +1
-0+ Profiling on x86 ----------------doc/guides/rel_notes/release_23_03.rst +0
-0doc/guides/rel_notes/release_23_03.rst New Features · line 153
Quoted 7 lines
doc/guides/rel_notes/release_23_03.rst +4
-0@@ -147,6 +147,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added PMU library.** + + Added a new performance monitoring unit (PMU) library which allows + applications to perform self monitoring activities without depending on external utilitieslike perf.
Quoted 99 lines
doc/guides/rel_notes/release_23_03.rst +3
-0+ After integration with :doc:`../prog_guide/trace_lib` data gathered + from hardware counters can be stored in CTF format for further analysis. + Removed Items -------------lib/meson.build index renamed from lib/meson.build +0
-0
450c061d2b..8a42d45d20100644lib/meson.build +1
-0@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ring diff --gita/lib/pmu/meson.build b/lib/pmu/meson.build new file mode 100644 index 0000000000..a4160b494e
lib/pmu/meson.build new file +13
-0@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(C) 2023 Marvell +International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')lib/pmu/pmu_private.h new file renamed from lib/pmu/pmu_private.h +0
-0mode 100644 index 0000000000..b9f8c1ddc8
lib/pmu/pmu_private.h new file +32
-0@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. +* +* @param config +* Architecture specific event configuration. Consult kernel sources for available options. +*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */ lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..950f999cb7
lib/pmu/rte_pmu.c new file +21
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> Quoted 9 lines
lib/pmu/rte_pmu.c line 25
Quoted 4 lines
lib/pmu/rte_pmu.c new file +4
-0+ +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?
There are not any plans to support that elsewhere currently so flat directory structure is good enough.
Quoted 10 lines
lib/pmu/rte_pmu.c line 31
Quoted 7 lines
lib/pmu/rte_pmu.c new file +7
-0+ +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> +((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) << +(__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu;Do we really need struct declaration here?
Quoted 1 line
What’s the problem with this placement precisely?
Not a big deal, but It seems excessive for me.
As I understand you do have include just above for the whole .h that contains definition of that struct anyway.
Quoted 523 lines
Quoted 513 lines
Quoted 513 lines
lib/pmu/rte_pmu.c new file +418
-0+/* +* Following __rte_weak functions provide default no-op. +Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { } + +static int +get_term_format (const char *name,int *num,uint64_t *mask) { +char path[PATH_MAX ]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/format/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) { +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +} + +static int +get_event_config (const char *name,uint64_t config[3 ]) { +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) { +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); } + +static int +open_events (struct rte_pmu_event_group *group) { +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) { +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) { +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE ,PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next); +rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; +} + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) { +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) +{ +struct rte_pmu_event *event; +char path[PATH_MAX ]; + +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); +if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..6b664c3336
lib/pmu/rte_pmu.h new file +88
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> +#include <rte_common.h> +#include <rte_compat.h> +#include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8 + +/** + * A structure describing a group of events. + */ +struct rte_pmu_event_group { + struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */ + int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */ + bool enabled; /**< true if group was enabled on particular lcore */ + TAILQ_ENTRY(rte_pmu_event_group) next; /**< list entry */ } +__rte_cache_aligned; + +/** + * A structure describing an event. + */ +struct rte_pmu_event { + char *name; /**< name of an event */ + unsigned int index; /**< event index into fds/mmap_pages */ + TAILQ_ENTRY(rte_pmu_event) next; /**< list entry */ }; + +/** + * A PMU state container. + */ +struct rte_pmu { + char *name; /**< name of core PMU listed under /sys/bus/event_source/devices */ + rte_spinlock_t lock; /**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ + unsigned int num_group_events; /**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ + unsigned int initialized; /**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own version +*/ #ifndef rte_pmu_pmc_read #define rte_pmu_pmc_read(index) ({ 0; }) +#endif + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Read PMU counter. + * + * @warning This should be not called directly. + * + * @param pc + * Pointer to the mmapped user page. + * @return + * Counter value read from hardware. + */ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage(struct perf_event_mmap_page *pc) { + uint64_t width, offset; + uint32_t seq, index; + int64_t pmc;Quoted 9 lines
lib/pmu/rte_pmu.h line 96
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?
Quoted 3 lines
It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?
Quoted 137 lines
Quoted 137 lines
Quoted 137 lines
lib/pmu/rte_pmu.h new file +116
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) + return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. +* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ +__rte_experimental +static __rte_always_inline uint64_t +rte_pmu_read (unsigned int index) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_PMU_H_ */ lib/pmu/version.map new file mode renamed from lib/pmu/version.map +0
-0100644 index 0000000000..39a4f279c1
lib/pmu/version.map new file +15
-0@@ -0,0 +1,15 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + __rte_pmu_enable_group; + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +};
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 864 lines
-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Friday, February 17, 2023 11:15 AM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU eventsQuoted 321 lines
Quoted 297 lines
Quoted 168 lines
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>--- MAINTAINERS | 5 + app/test/meson.build | 2 + app/test/test_pmu.c | 62 ++++ doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/profile_app.rst | 12 + doc/guides/rel_notes/release_23_03.rst | 7 + lib/meson.build | 1 + lib/pmu/meson.build | 13 + lib/pmu/pmu_private.h | 32 ++ lib/pmu/rte_pmu.c | 460 +++++++++++++++++++++++++ lib/pmu/rte_pmu.h | 212 ++++++++++++ lib/pmu/version.map | 15 + 13 files changed, 824 insertions(+), 1 deletion(-)create mode 100644
app/test/test_pmu.c
create mode 100644lib/pmu/meson.build
create mode 100644lib/pmu/pmu_private.h
create mode 100644lib/pmu/rte_pmu.c
create mode 100644lib/pmu/rte_pmu.h
create mode 100644lib/pmu/version.mapMAINTAINERS index 3495946d0f..d37f242120 renamed from MAINTAINERS +0
-0100644
MAINTAINERS +5
-0@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]> M: Pavan Nikhilesh <[email protected]> F: lib/node/ +PMU - EXPERIMENTAL +M: Tomasz Duszynski <[email protected]> +F: lib/pmu/ +F: app/test/test_pmu* + Test Applications -----------------app/test/meson.build index renamed from app/test/meson.build +0
-0
f34d19e3c3..6b61b7fc32100644app/test/meson.build +2
-0@@ -111,6 +111,7 @@ test_sources = files( 'test_reciprocal_division_perf.c', 'test_red.c', 'test_pie.c', + 'test_pmu.c', 'test_reorder.c', 'test_rib.c', 'test_rib6.c',@@ -239,6 +240,7 @@ fast_tests = [ ['kni_autotest', false, true], ['kvargs_autotest', true, true], ['member_autotest', true, true], + ['pmu_autotest', true, true], ['power_cpufreq_autotest', false, true], ['power_autotest', true, true], ['power_kvm_vm_autotest', false, true], diff --git
a/app/test/test_pmu.cb/app/test/test_pmu.cnew file mode 100644 index 0000000000..c257638e8bapp/test/test_pmu.c new file +61
-0@@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include "test.h" + +#ifndef RTE_EXEC_ENV_LINUX + +static int +test_pmu (void ) +{ +printf ("pmu_autotest only supported on Linux, skipping test\n" ); +return TEST_SKIPPED ; +} + +#else + +#include <rte_pmu.h> + +static int +test_pmu_read (void ) +{ +const char *name =NULL ; +int tries =10 , event; +uint64_t val =0 ; + +if (name ==NULL ) { +printf ("PMU not supported on this arch\n" ); +return TEST_SKIPPED ; + } + +if (rte_pmu_init () <0 ) +return TEST_SKIPPED ; + + event =rte_pmu_add_event (name); +while (tries--) + val +=rte_pmu_read (event); + +rte_pmu_fini (); + +return val ?TEST_SUCCESS :TEST_FAILED ; } + +static struct unit_test_suite pmu_tests = { + .suite_name ="pmu autotest" , + .setup =NULL , + .teardown =NULL , + .unit_test_cases = { +TEST_CASE (test_pmu_read), +TEST_CASES_END () + } +}; + +static int +test_pmu (void ) +{ +return unit_test_suite_runner (&pmu_tests); +} + +#endif /* RTE_EXEC_ENV_LINUX */ + +REGISTER_TEST_COMMAND (pmu_autotest, test_pmu);doc/api/doxy-api-index.md +1
-1@@ -223,7 +223,8 @@ The public API headers are grouped by topics: [log](@ref rte_log.h), [errno](@ref rte_errno.h), [trace](@ref rte_trace.h), - [trace_point](@ref rte_trace_point.h) + [trace_point](@ref rte_trace_point.h), [pmu](@ref rte_pmu.h) - **misc**: [EAL config](@ref rte_eal.h),doc/api/doxy-api.conf.in +0
-0index
e859426099..350b5a8c94100644doc/api/doxy-api.conf.in +1
-0@@ -63,6 +63,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ + @TOPDIR@/lib/pmu \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ @TOPDIR@/lib/rawdev \ diff --git
a/doc/guides/prog_guide/profile_app.rstb/doc/guides/prog_guide/profile_app.rst
index14292d4c25..89e38cd301100644doc/guides/prog_guide/profile_app.rst +2
-0@@ -7,6 +7,18 @@ Profile Your Application The following sections describe methods of profiling DPDK applications on different architectures. +Performance counter based profiling +-----------------------------------doc/guides/prog_guide/profile_app.rst Profile Your Application · line 21
Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +10
-0+ +Majority of architectures support some performance monitoring unit (PMU). +Such unit provides programmable counters that monitor specific events. + +Different tools gather that information, like for example perf. +However, in some scenarios when CPU cores are isolated and run +dedicated tasks interrupting those tasks with perf may be undesirable. + +In such cases, an application can use the PMU library to read such +events via``rte_pmu_read()``.
Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +1
-0+ Profiling on x86 ----------------doc/guides/rel_notes/release_23_03.rst +0
-0doc/guides/rel_notes/release_23_03.rst New Features · line 154
Quoted 8 lines
doc/guides/rel_notes/release_23_03.rst +5
-0@@ -147,6 +147,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added PMU library.** + + Added a new performance monitoring unit (PMU) library which + allows applications to perform self monitoring activities + without depending on external utilitieslike perf.
Quoted 99 lines
doc/guides/rel_notes/release_23_03.rst +3
-0+ After integration with :doc:`../prog_guide/trace_lib` data + gathered from hardware counters can be stored in CTF format for further analysis. + Removed Items -------------lib/meson.build index renamed from lib/meson.build +0
-0
450c061d2b..8a42d45d20100644lib/meson.build +1
-0@@ -11,6 +11,7 @@ libraries = [ 'kvargs', # eal depends on kvargs 'telemetry', # basic info querying + 'pmu', 'eal', # everything depends on eal 'ring', 'rcu', # rcu depends on ring diff --gita/lib/pmu/meson.build b/lib/pmu/meson.build new file mode 100644 index 0000000000..a4160b494e
lib/pmu/meson.build new file +13
-0@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(C) 2023 +Marvell International Ltd. + +if not is_linux + build = false + reason = 'only supported on Linux' + subdir_done() +endif + +includes = [global_inc] + +sources = files('rte_pmu.c') +headers = files('rte_pmu.h')lib/pmu/pmu_private.h new file renamed from lib/pmu/pmu_private.h +0
-0mode 100644 index 0000000000..b9f8c1ddc8
lib/pmu/pmu_private.h new file +32
-0@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _PMU_PRIVATE_H_ +#define _PMU_PRIVATE_H_ + +/** +* Architecture specific PMU init callback. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +int +pmu_arch_init (void ); + +/** +* Architecture specific PMU cleanup callback. +*/ +void +pmu_arch_fini (void ); + +/** +* Apply architecture specific settings to config before passing it to syscall. +* +* @param config +* Architecture specific event configuration. Consult kernel sources for available options. +*/ +void +pmu_arch_fixup_config (uint64_t config[3 ]); + +#endif /* _PMU_PRIVATE_H_ */ lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..950f999cb7
lib/pmu/rte_pmu.c new file +21
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> Quoted 9 lines
lib/pmu/rte_pmu.c line 25
Quoted 4 lines
lib/pmu/rte_pmu.c new file +4
-0+ +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?
There are not any plans to support that elsewhere currently so flat directory structure is good enough.
Quoted 10 lines
lib/pmu/rte_pmu.c line 31
Quoted 7 lines
lib/pmu/rte_pmu.c new file +7
-0+ +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> +((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) << +(__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu;Do we really need struct declaration here?
Quoted 1 line
What’s the problem with this placement precisely?
Not a big deal, but It seems excessive for me.
As I understand you do have include just above for the whole .h that contains definition of that struct anyway.Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
lib/pmu/rte_pmu.c new file +22
-0+/* +* Following __rte_weak functions provide default no-op. +Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) +{ } + +static intQuoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+get_term_format(const char *name, int *num, uint64_t *mask) { + char path[PATH_MAX]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name,name);
Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
lib/pmu/rte_pmu.c new file +62
-0+ fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) { +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +}Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+ +static int +get_event_config (const char *name,uint64_t config[3 ]) { +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,name);
Quoted 231 lines
Quoted 231 lines
Quoted 231 lines
lib/pmu/rte_pmu.c new file +231
-0+ fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) { +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); } + +static int +open_events (struct rte_pmu_event_group *group) { +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) { +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) { +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE , +PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next); +rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; } + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) { +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) { +struct rte_pmu_event *event; +char path[PATH_MAX ];Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+ +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,name);
Quoted 111 lines
Quoted 111 lines
Quoted 111 lines
lib/pmu/rte_pmu.c new file +79
-0+if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..6b664c3336
lib/pmu/rte_pmu.h new file +27
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU +and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> +Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8 + +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */
Quoted 62 lines
Quoted 61 lines
Quoted 46 lines
lib/pmu/rte_pmu.h new file +46
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; + +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own +version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. lib/pmu/rte_pmu.h line 96
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+*/ +static __rte_always_inline uint64_t __rte_pmu_read_userpage(struct +perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?
Quoted 8 lines
Quoted 3 lines
It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?
This number is just an indicator whether kernel did change something or not.
If cpu reordering will come into play then this will not change anything from pov of this loop. All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.
Quoted 137 lines
Quoted 137 lines
Quoted 137 lines
Quoted 137 lines
lib/pmu/rte_pmu.h new file +115
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) + return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. +* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ +__rte_experimental +static __rte_always_inline uint64_t rte_pmu_read(unsigned int +index) { +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_PMU_H_ */ lib/pmu/version.map new file renamed from lib/pmu/version.map +0
-0mode
100644 index 0000000000..39a4f279c1lib/pmu/version.map new file +15
-0@@ -0,0 +1,15 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + __rte_pmu_enable_group; + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +};
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 107 lines
Quoted 107 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..6b664c3336
lib/pmu/rte_pmu.h new file +27
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU +and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> +Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8 + +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */
Quoted 65 lines
Quoted 52 lines
Quoted 52 lines
lib/pmu/rte_pmu.h new file +52
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; + +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own +version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t __rte_pmu_read_userpage(struct +perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc;Quoted 9 lines
lib/pmu/rte_pmu.h line 96
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Quoted 6 lines
Quoted 4 lines
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?This number is just an indicator whether kernel did change something or not.
You are talking about pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual for perf_event_open() also has a code snippet with compiler barrier only...
Quoted 3 lines
If cpu reordering will come into play then this will not change anything from pov of this loop. All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.
Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:
cpu #1 kernel vent_update_userpage:
pc->lock++; // pc->lock==1
pc->index=I2;
pc->offset=O2;
...
pc->lock++; //pc->lock==2
T3:
cpu #0 continue with read pmu:
seq=pc->lock; //seq == 2
offset=pc->offset; // offset == O2
....
pmc = rte_pmu_pmc_read(index - 1); // Note that we read at I1, not I2
offset += pmc; //offset == O2 + pmcread(I1-1);
if (pc->lock == seq) // they are equal, return
return offset;Or, it can happen, but by some reason we don't care much?
Quoted 82 lines
Quoted 82 lines
Quoted 82 lines
Quoted 82 lines
Quoted 82 lines
lib/pmu/rte_pmu.h new file +81
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 185
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/
Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header? As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25
-0+__rte_experimental +static __rte_always_inline uint64_t rte_pmu_read(unsigned int +index) { +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif +
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 157 lines
-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Monday, February 20, 2023 3:31 PM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU eventsQuoted 109 lines
Quoted 109 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..6b664c3336
lib/pmu/rte_pmu.h new file +28
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup +PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> + +/** Maximum number of events in a group */ #defineQuoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+MAX_NUM_GROUP_EVENTS 8 + +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ]; +/**< array of user pages */
Quoted 66 lines
Quoted 53 lines
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; + +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, +_event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own +version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc;Quoted 9 lines
lib/pmu/rte_pmu.h line 96
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Quoted 6 lines
Quoted 4 lines
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?This number is just an indicator whether kernel did change something or not.
You are talking about
pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual forperf_event_open()also has a code snippet with compiler barrier only...Quoted 3 lines
If cpu reordering will come into play then this will not change anything from pov of this loop. All we want is fresh data when needed and no involvement of compiler
when it comes to reordering code.Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:cpu #1 kernel vent_update_userpage: pc->lock++; // pc->lock==1 pc->index=I2; pc->offset=O2; ... pc->lock++; //pc->lock==2 T3: cpu #0 continue with read pmu: seq=pc->lock; //seq == 2 offset=pc->offset; // offset == O2 .... pmc = rte_pmu_pmc_read(index - 1); // Note that we read at I1, not I2 offset += pmc; //offset == O2 + pmcread(I1-1); if (pc->lock == seq) // they are equal, return return offset;
Quoted 2 lines
Or, it can happen, but by some reason we don't care much?
This code does self-monitoring and user page (whole group actually) is per thread running on current cpu. Hence I am not sure what are you trying to prove with that example.
Quoted 122 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
lib/pmu/rte_pmu.h new file +48
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimentalQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25
-0+*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 185
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ Another question - do we really need to have
__rte_pmu_read_userpage()andrte_pmu_read()as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25
-0+__rte_experimental +static __rte_always_inline uint64_t rte_pmu_read(unsigned int +index) { +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif +
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 149 lines
Quoted 149 lines
Quoted 110 lines
Quoted 110 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..6b664c3336
lib/pmu/rte_pmu.h new file +28
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup +PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> + +/** Maximum number of events in a group */ #defineQuoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 36
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+MAX_NUM_GROUP_EVENTS 8 + +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ]; +/**< array of user pages */
Quoted 66 lines
Quoted 53 lines
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; + +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, +_event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own +version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc;Quoted 9 lines
lib/pmu/rte_pmu.h line 93
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Quoted 6 lines
Quoted 4 lines
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?This number is just an indicator whether kernel did change something or not.
You are talking about
pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual forperf_event_open()also has a code snippet with compiler barrier only...Quoted 3 lines
If cpu reordering will come into play then this will not change anything from pov of this loop. All we want is fresh data when needed and no involvement of compiler
when it comes to reordering code.Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:cpu #1 kernel vent_update_userpage: pc->lock++; // pc->lock==1 pc->index=I2; pc->offset=O2; ... pc->lock++; //pc->lock==2 T3: cpu #0 continue with read pmu: seq=pc->lock; //seq == 2 offset=pc->offset; // offset == O2 .... pmc = rte_pmu_pmc_read(index - 1); // Note that we read at I1, not I2 offset += pmc; //offset == O2 + pmcread(I1-1); if (pc->lock == seq) // they are equal, return return offset;
Quoted 5 lines
Quoted 2 lines
Or, it can happen, but by some reason we don't care much?
This code does self-monitoring and user page (whole group actually) is per thread running on current cpu. Hence I am not sure what are you trying to prove with that example.
I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why?
My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them.
So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?
Also there was another question below, which you probably missed, so I copied it here:
Another question - do we really need to have __rte_pmu_read_userpage() and rte_pmu_read() as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.
Quoted 123 lines
Quoted 123 lines
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
lib/pmu/rte_pmu.h new file +48
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimentalQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 149
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25
-0+*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 182
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ Another question - do we really need to have
__rte_pmu_read_userpage()andrte_pmu_read()as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25
-0+__rte_experimental +static __rte_always_inline uint64_t rte_pmu_read(unsigned int +index) { +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif +
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 167 lines
-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Monday, February 20, 2023 6:21 PM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU eventsQuoted 115 lines
Quoted 115 lines
Quoted 115 lines
Quoted 114 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
lib/pmu/rte_pmu.h new file renamed from lib/pmu/rte_pmu.h +0
-0mode
100644 index 0000000000..6b664c3336lib/pmu/rte_pmu.h new file +28
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell */ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to +setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 36
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page +*mmap_pages[MAX_NUM_GROUP_EVENTS ]; +/**< array of user pages */
Quoted 63 lines
Quoted 53 lines
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; + +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, +_event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its +own version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc;Quoted 9 lines
lib/pmu/rte_pmu.h line 93
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?Quoted 3 lines
It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?Quoted 8 lines
Quoted 8 lines
Quoted 1 line
This number is just an indicator whether kernel did change something or not.
You are talking about
pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual forperf_event_open()also has a code snippet with compiler barrier only...Quoted 1 line
If cpu reordering will come into play then this will not change anything from pov of this
loop.
Quoted 26 lines
Quoted 26 lines
Quoted 2 lines
All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.
Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:cpu #1 kernel vent_update_userpage: pc->lock++; // pc->lock==1 pc->index=I2; pc->offset=O2; ... pc->lock++; //pc->lock==2 T3: cpu #0 continue with read pmu: seq=pc->lock; //seq == 2 offset=pc->offset; // offset == O2 .... pmc = rte_pmu_pmc_read(index - 1); // Note that we read at I1, not I2 offset += pmc; //offset == O2 + pmcread(I1-1); if (pc->lock == seq) // they are equal, return return offset;Quoted 5 lines
Quoted 2 lines
Or, it can happen, but by some reason we don't care much?
This code does self-monitoring and user page (whole group actually) is
per thread running on current cpu. Hence I am not sure what are you trying to prove with thatexample.
Quoted 8 lines
I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why?
My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them.
So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?
Look. I'll reiterate that.
1. That user page/group/PMU config is per process. Other processes do not access that.
All this happens on the very same CPU where current thread is running.
2. Suppose you've already read seq. Now for some reason kernel updates data in page seq was read from.
3. Kernel will enter critical section during update. seq changes along with other data without app knowing about it. If you want nitty gritty details consult kernel sources.
4. app resumes and has some stale data but *WILL* read new seq. Code loops again because values do not match.
5. Otherwise seq values match and data is valid.
Quoted 6 lines
Also there was another question below, which you probably missed, so I copied it here:
Another question - do we really need to have__rte_pmu_read_userpage()andrte_pmu_read()as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.
These functions need to be inlined otherwise performance takes a hit.
Quoted 125 lines
Quoted 125 lines
Quoted 125 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
lib/pmu/rte_pmu.h new file +49
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +intQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 150
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+rte_pmu_init (void); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU +counters are no longer being read.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25
-0+*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 183
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ Another question - do we really need to have
__rte_pmu_read_userpage()andrte_pmu_read()as static inline functions in public header? As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
lib/pmu/rte_pmu.h new file +26
-0+__rte_experimental +static __rte_always_inline uint64_t rte_pmu_read(unsigned +int +index) { +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif +
Re: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 167 lines
Quoted 158 lines
Quoted 116 lines
Quoted 116 lines
Quoted 116 lines
Quoted 115 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
lib/pmu/rte_pmu.h new file renamed from lib/pmu/rte_pmu.h +0
-0mode
100644 index 0000000000..6b664c3336lib/pmu/rte_pmu.h new file +28
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell */ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to +setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 36
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page +*mmap_pages[MAX_NUM_GROUP_EVENTS ]; +/**< array of user pages */
Quoted 63 lines
Quoted 53 lines
Quoted 53 lines
lib/pmu/rte_pmu.h new file +53
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; + +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, +_event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its +own version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc;Quoted 9 lines
lib/pmu/rte_pmu.h line 93
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?Quoted 3 lines
It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?Quoted 8 lines
Quoted 8 lines
Quoted 1 line
This number is just an indicator whether kernel did change something or not.
You are talking about
pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual forperf_event_open()also has a code snippet with compiler barrier only...Quoted 1 line
If cpu reordering will come into play then this will not change anything from pov of this
loop.
Quoted 26 lines
Quoted 26 lines
Quoted 2 lines
All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.
Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:cpu #1 kernel vent_update_userpage: pc->lock++; // pc->lock==1 pc->index=I2; pc->offset=O2; ... pc->lock++; //pc->lock==2 T3: cpu #0 continue with read pmu: seq=pc->lock; //seq == 2 offset=pc->offset; // offset == O2 .... pmc = rte_pmu_pmc_read(index - 1); // Note that we read at I1, not I2 offset += pmc; //offset == O2 + pmcread(I1-1); if (pc->lock == seq) // they are equal, return return offset;Quoted 5 lines
Quoted 2 lines
Or, it can happen, but by some reason we don't care much?
This code does self-monitoring and user page (whole group actually) is
per thread running on current cpu. Hence I am not sure what are you trying to prove with thatexample.
Quoted 8 lines
I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why?
My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them.
So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?
Quoted 3 lines
Look. I'll reiterate that.
1. That user page/group/PMU config is per process. Other processes do not access that.
Ok, that's clear.
Quoted 1 line
All this happens on the very same CPU where current thread is running.
Ok... but can't this page be updated by kernel thread running simultaneously on different CPU?
Quoted 3 lines
2. Suppose you've already read seq. Now for some reason kernel updates data in page seq was read from.
3. Kernel will enter critical section during update. seq changes along with other data without app knowing about it. If you want nitty gritty details consult kernel sources.
Look, I don't have to beg you to answer these questions.
In fact, I expect library author to document all such narrow things clearly either in in PG, or in source code comments (ideally in both). If not, then from my perspective the patch is not ready stage and shouldn't be accepted.
I don't know is compiler-barrier is enough here or not, but I think it is definitely worth a clear explanation in the docs. I suppose it wouldn't be only me who will get confused here.
So please take an effort and document it clearly why you believe there is no race-condition.
Quoted 1 line
4. app resumes and has some stale data but *WILL* read new seq. Code loops again because values do not match.
If the kernel will always execute update for this page in the same thread context, then yes, - user code will always note the difference after resume.
But why it can't happen that your user-thread reads this page on one CPU, while some kernel code on other CPU updates it simultaneously?
Quoted 10 lines
5. Otherwise seq values match and data is valid.
Quoted 6 lines
Also there was another question below, which you probably missed, so I copied it here:
Another question - do we really need to have__rte_pmu_read_userpage()andrte_pmu_read()as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.These functions need to be inlined otherwise performance takes a hit.
I understand that perfomance might be affected, but how big is hit?
I expect actual PMU read will not be free anyway, right?
If the diff is small, might be it is worth to go for such change,
removing unneeded structures from public headers would help a lot in future in terms of ABI/API stability.
Quoted 126 lines
Quoted 125 lines
Quoted 125 lines
Quoted 125 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
lib/pmu/rte_pmu.h new file +49
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +intQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 150
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+rte_pmu_init (void); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU +counters are no longer being read.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25
-0+*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 183
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ Another question - do we really need to have
__rte_pmu_read_userpage()andrte_pmu_read()as static inline functions in public header? As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
lib/pmu/rte_pmu.h new file +26
-0+__rte_experimental +static __rte_always_inline uint64_t rte_pmu_read(unsigned +int +index) { +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif +
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 180 lines
-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Tuesday, February 21, 2023 1:48 AM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: Re: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU eventsQuoted 170 lines
Quoted 159 lines
Quoted 117 lines
Quoted 117 lines
Quoted 116 lines
Quoted 106 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
lib/pmu/rte_pmu.h new file renamed from lib/pmu/rte_pmu.h +0
-0mode
100644 index 0000000000..6b664c3336lib/pmu/rte_pmu.h new file +28
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell */ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup +PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 36
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page +*mmap_pages[MAX_NUM_GROUP_EVENTS ]; +/**< array of user pages */
Quoted 63 lines
Quoted 62 lines
Quoted 47 lines
lib/pmu/rte_pmu.h new file +47
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; + +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, +_event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own +version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. lib/pmu/rte_pmu.h line 93
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?Quoted 8 lines
Quoted 3 lines
It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?This number is just an indicator whether kernel did change something or not.
Quoted 7 lines
Quoted 7 lines
You are talking about
pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual forperf_event_open()also has a code snippet with compiler barrier only...Quoted 2 lines
If cpu reordering will come into play then this will not change anything from pov of this
loop.
Quoted 26 lines
Quoted 26 lines
Quoted 2 lines
All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.
Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:cpu #1 kernel vent_update_userpage: pc->lock++; // pc->lock==1 pc->index=I2; pc->offset=O2; ... pc->lock++; //pc->lock==2 T3: cpu #0 continue with read pmu: seq=pc->lock; //seq == 2 offset=pc->offset; // offset == O2 .... pmc = rte_pmu_pmc_read(index - 1); // Note that we read at I1, not I2 offset += pmc; //offset == O2 + pmcread(I1-1); if (pc->lock == seq) // they are equal, return return offset;Quoted 6 lines
Quoted 2 lines
Or, it can happen, but by some reason we don't care much?
This code does self-monitoring and user page (whole group actually) is per thread running on current cpu. Hence I am not sure what are you trying to prove with that
example.
Quoted 8 lines
I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why? My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them. So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?Look. I'll reiterate that.
Quoted 9 lines
Quoted 1 line
1. That user page/group/PMU config is per process. Other processes do not access that.
Ok, that's clear.
Quoted 1 line
All this happens on the very same CPU where current thread is running.
Ok... but can't this page be updated by kernel thread running simultaneously on different CPU?
I already pointed out that event/counter configuration is bound to current cpu. How can possibly other cpu update that configuration? This cannot work.
If you think that there's some problem with the code (or is simply broken on your setup) and logic has obvious flaw and you can provide meaningful evidence of that then I'd be more than happy to apply that fix. Otherwise that discussion will get us nowhere.
Quoted 173 lines
Quoted 2 lines
2. Suppose you've already read seq. Now for some reason kernel updates data in page seq was read
from.
Quoted 1 line
3. Kernel will enter critical section during update. seq changes along with other data without
app knowing about it.
Quoted 1 line
If you want nitty gritty details consult kernel sources.
Look, I don't have to beg you to answer these questions.
In fact, I expect library author to document all such narrow things clearly either in in PG, or in source code comments (ideally in both). If not, then from my perspective the patch is not ready stage and shouldn't be accepted.
I don't know is compiler-barrier is enough here or not, but I think it is definitely worth a clear explanation in the docs.
I suppose it wouldn't be only me who will get confused here. So please take an effort and document it clearly why you believe there is no race-condition.Quoted 1 line
4. app resumes and has some stale data but *WILL* read new seq. Code loops again because values
do not match.
If the kernel will always execute update for this page in the same thread context, then yes, - user code will always note the difference after resume.
But why it can't happen that your user-thread reads this page on one CPU, while some kernel code on other CPU updates it simultaneously?Quoted 10 lines
5. Otherwise seq values match and data is valid.
Quoted 6 lines
Also there was another question below, which you probably missed, so I copied it here:
Another question - do we really need to have__rte_pmu_read_userpage()andrte_pmu_read()as static inline functions in public header?
As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.These functions need to be inlined otherwise performance takes a hit.
I understand that perfomance might be affected, but how big is hit? I expect actual PMU read will not be free anyway, right? If the diff is small, might be it is worth to go for such change, removing unneeded structures from public headers would help a lot in future in terms of ABI/API stability.
Quoted 126 lines
Quoted 125 lines
Quoted 125 lines
Quoted 125 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
lib/pmu/rte_pmu.h new file +49
-0+ index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) { + pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. +* +* @return +* 0 in case of success, negative value otherwise. +*/ +__rte_experimental +intQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 150
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+rte_pmu_init (void); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU +counters are no longer being read.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/pmu/rte_pmu.h new file +25
-0+*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 183
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ Another question - do we really need to have
__rte_pmu_read_userpage()andrte_pmu_read()as static inline functions in public header? As I understand, because of that we also have to make 'struct rte_pmu_*' definitions also public.Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
lib/pmu/rte_pmu.h new file +26
-0+__rte_experimental +static __rte_always_inline uint64_t rte_pmu_read(unsigned +int +index) { +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif +
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 177 lines
Quoted 177 lines
Quoted 170 lines
Quoted 161 lines
Quoted 119 lines
Quoted 119 lines
Quoted 118 lines
Quoted 108 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
lib/pmu/rte_pmu.h new file renamed from lib/pmu/rte_pmu.h +0
-0mode
100644 index 0000000000..6b664c3336lib/pmu/rte_pmu.h new file +28
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell */ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup +PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 36
Quoted 8 lines
lib/pmu/rte_pmu.h new file +8
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page +*mmap_pages[MAX_NUM_GROUP_EVENTS ]; +/**< array of user pages */
Quoted 63 lines
Quoted 62 lines
Quoted 47 lines
lib/pmu/rte_pmu.h new file +47
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; + +/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, +_event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own +version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. lib/pmu/rte_pmu.h line 93
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier ();Are you sure that
compiler_barrier()is enough here? On some archs CPU itself has freedom to re-order reads. Or I am missing something obvious here?Quoted 8 lines
Quoted 3 lines
It's a matter of not keeping old stuff cached in registers and making sure that we have two reads of lock. CPU reordering won't do any harm here.
Sorry, I didn't get you here:
Suppose CPU will re-order reads and will read lock *after* index or offset value. Wouldn't it mean that in that case index and/or offset can contain old/invalid values?This number is just an indicator whether kernel did change something or not.
Quoted 7 lines
Quoted 7 lines
You are talking about
pc->lock, right?
Yes, I do understand that it is sort of seqlock.
That's why I am puzzled why we do not care about possible cpu read-reordering. Manual forperf_event_open()also has a code snippet with compiler barrier only...Quoted 2 lines
If cpu reordering will come into play then this will not change anything from pov of this
loop.
Quoted 26 lines
Quoted 26 lines
Quoted 2 lines
All we want is fresh data when needed and no involvement of compiler when it comes to reordering code.
Ok, can you probably explain to me why the following could not happen:
T0:
pc->seqlock==0; pc->index==I1; pc->offset==O1;
T1:
cpu #0 read pmu (due to cpu read reorder, we get index value before seqlock):
index=pc->index; //index==I1;
T2:cpu #1 kernel vent_update_userpage: pc->lock++; // pc->lock==1 pc->index=I2; pc->offset=O2; ... pc->lock++; //pc->lock==2 T3: cpu #0 continue with read pmu: seq=pc->lock; //seq == 2 offset=pc->offset; // offset == O2 .... pmc = rte_pmu_pmc_read(index - 1); // Note that we read at I1, not I2 offset += pmc; //offset == O2 + pmcread(I1-1); if (pc->lock == seq) // they are equal, return return offset;Quoted 6 lines
Quoted 2 lines
Or, it can happen, but by some reason we don't care much?
This code does self-monitoring and user page (whole group actually) is per thread running on current cpu. Hence I am not sure what are you trying to prove with that
example.
Quoted 8 lines
I am not trying to prove anything so far.
I am asking is such situation possible or not, and if not, why? My current understanding (possibly wrong) is that after you mmaped these pages, kernel still can asynchronously update them. So, when reading the data from these pages you have to check 'lock' value before and after accessing other data.
If so, why possible cpu read-reordering doesn't matter?Quoted 3 lines
Look. I'll reiterate that.
1. That user page/group/PMU config is per process. Other processes do not access that.
Ok, that's clear.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
All this happens on the very same CPU where current thread is running.
Ok... but can't this page be updated by kernel thread running simultaneously on different CPU?
I already pointed out that event/counter configuration is bound to current cpu. How can possibly other cpu update that configuration? This cannot work.
Can you elaborate a bit what you mean with 'event/counter configuration is bound to current cpu'? If that means it could be updated only by code running on given, CPU - yes it is clear. But can this page be read by user-space from different CPU?
Or you just assume that your user-space thread will *always* be bounded just to one particular CPU and would never switch?
Quoted 6 lines
If you think that there's some problem with the code (or is simply broken on your setup) and logic[] a bit has obvious flaw and you can provide meaningful evidence of that then I'd be more than happy to apply that fix. Otherwise that discussion will get us nowhere.
Yes, we are going in cycles here.
I keep asking you same questions about library function internals, you keep refusing to explain things to me insisting that it is 'way too obvious'.
Well, sorry but it is not obvious to me.
So I still insist that a clearly documented internal design and expected usage is required for that patch before it can be accepted.
Konstantin
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
In reply to Tomasz DuszynskiQuoted 37 lines
Quoted 27 lines
Quoted 27 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..950f999cb7
lib/pmu/rte_pmu.c new file +21
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> Quoted 9 lines
lib/pmu/rte_pmu.c line 25
Quoted 4 lines
lib/pmu/rte_pmu.c new file +4
-0+ +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"I suppose that pass (as the whole implementation) is linux specific? If so, wouldn't it make sense to have it under linux subdir?
Quoted 2 lines
There are not any plans to support that elsewhere currently so flat directory structure is good enough.
Ok, I suppose then best choice is to ask freebsd and windows maintainers. Guys, any thoughts here?
Thanks
Konstantin
Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
In reply to Tomasz DuszynskiQuoted 10 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>
Few more comments/questions below.
Quoted 287 lines
lib/pmu/rte_pmu.c new file +281
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> + +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices" + +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h))))) +#define FIELD_PREP (m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu; + +/* +* Following __rte_weak functions provide default no-op. Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) +{ +} + +static int +get_term_format (const char *name,int *num,uint64_t *mask) +{ +char path[PATH_MAX ]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/format/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) +{ +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +} + +static int +get_event_config (const char *name,uint64_t config[3 ]) +{ +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) +{ +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); +} + +static int +open_events (struct rte_pmu_event_group *group) +{ +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) +{ +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) +{ +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE ,PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno;
lib/pmu/rte_pmu.c line 291
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+ goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);
Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Can memory from its TLS block get re-used (by other thread or for other purposes)?
Quoted 195 lines
lib/pmu/rte_pmu.c new file +169
-0+rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; +} + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) +{ +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) +{ +struct rte_pmu_event *event; +char path[PATH_MAX ]; + +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); +if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file +20
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif +
lib/pmu/rte_pmu.h line 30
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> +#include <rte_common.h> +#include <rte_compat.h> +#include <rte_spinlock.h> + +/** Maximum number of events in a group */ +#define MAX_NUM_GROUP_EVENTS 8
forgot RTE_ prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically by enable_group()?
lib/pmu/rte_pmu.h line 41
Quoted 11 lines
lib/pmu/rte_pmu.h new file +11
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */ +int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ +} __rte_cache_aligned; +
Even if we'd decide to keep rte_pmu_read() as static inline (still not sure it is a good idea), why these two struct below (rte_pmu_event and rte_pmu) have to be public?
I think both can be safely moved away from public headers.
Quoted 52 lines
lib/pmu/rte_pmu.h new file +51
-0+/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ +}; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ +}; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own version */ +#ifndef rte_pmu_pmc_read +#define rte_pmu_pmc_read (index) ({ 0; }) +#endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) +{ +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc;
lib/pmu/rte_pmu.h line 102
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+ +for (;;) { + seq = pc->lock; +rte_compiler_barrier (); + index = pc->index; + offset = pc->offset; + width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) {
In mmap_events() you return EPERM if cap_user_rdpmc is not enabled.
Do you need another check here? Or this capability can be disabled by kernel at run-time?
Quoted 12 lines
lib/pmu/rte_pmu.h new file +12
-0+ pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } +
lib/pmu/rte_pmu.h line 124
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly.
__rte_internal ?
Quoted 5 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */lib/pmu/rte_pmu.h new file +1
-0+__rte_experimental
lib/pmu/rte_pmu.h line 139
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly.
Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?
Quoted 4 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */
Probably worth to mention that this function is not MT safe.
Same for _fini_ and add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.
Quoted 90 lines
lib/pmu/rte_pmu.h new file +69
-0+__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. +* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ +__rte_experimental +static __rte_always_inline uint64_t +rte_pmu_read (unsigned int index) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_PMU_H_ */ lib/pmu/version.map new file +15
-0@@ -0,0 +1,15 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + __rte_pmu_enable_group; + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +};
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 308 lines
-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Tuesday, February 21, 2023 3:17 AM
To: [email protected]
Subject: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU eventsExternal Email
----------------------------------------------------------------------
Quoted 9 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>Few more comments/questions below.
Quoted 283 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..950f999cb7
lib/pmu/rte_pmu.c new file +278
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> + +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices" + +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> +((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) << +(__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu; + +/* +* Following __rte_weak functions provide default no-op. +Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { } + +static int +get_term_format (const char *name,int *num,uint64_t *mask) { +char path[PATH_MAX ]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/format/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) { +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +} + +static int +get_event_config (const char *name,uint64_t config[3 ]) { +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) { +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); } + +static int +open_events (struct rte_pmu_event_group *group) { +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) { +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) { +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE ,PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno;
Quoted 8 lines
lib/pmu/rte_pmu.c line 291
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Nothing special. Any pointers to that thread-local in that thread are invalided.
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for other purposes)?
Why would any other thread reuse that? Eventually main thread will need that data to do the cleanup.
Quoted 202 lines
Quoted 202 lines
lib/pmu/rte_pmu.c new file +169
-0+rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; +} + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) +{ +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) +{ +struct rte_pmu_event *event; +char path[PATH_MAX ]; + +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); +if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file +26
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> +#include <rte_common.h> +#include <rte_compat.h>
Quoted 9 lines
lib/pmu/rte_pmu.h line 30
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+#include <rte_spinlock.h> + +/** Maximum number of events in a group */ +#define MAX_NUM_GROUP_EVENTS 8forgot
RTE_prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically byenable_group()?
8 is reasonable number I think. X86/ARM have actually less that that (was that something like 4?). Moreover events are scheduled as a group so there must be enough hw counters available for that to succeed. So this number should cover current needs.
Quoted 6 lines
Quoted 6 lines
lib/pmu/rte_pmu.h new file +6
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */
Quoted 8 lines
lib/pmu/rte_pmu.h line 41
Quoted 5 lines
lib/pmu/rte_pmu.h new file +5
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ +} __rte_cache_aligned; +Even if we'd decide to keep
rte_pmu_read()as static inline (still not sure it is a good idea),
We want to save as much cpu cycles as we possibly can and inlining does helps in that matter.
Quoted 3 lines
why these two struct below (
rte_pmu_eventandrte_pmu) have to be public? I think both can be safely moved away from public headers.
struct rte_pmu_event can be hidden I guess.
struct rte_pmu is used in this header hence cannot be moved elsewhere.
Quoted 59 lines
Quoted 59 lines
lib/pmu/rte_pmu.h new file +57
-0+/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ +}; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ +}; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own version */ +#ifndef rte_pmu_pmc_read +#define rte_pmu_pmc_read (index) ({ 0; }) +#endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) +{ +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier (); + index = pc->index; + offset = pc->offset;
Quoted 9 lines
lib/pmu/rte_pmu.h line 102
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) {In
mmap_events()you return EPERM ifcap_user_rdpmcis not enabled. Do you need another check here? Or this capability can be disabled by kernel at run-time?
That extra check in mmap_event() may be removed actually. Some archs allow disabling reading rdpmc (I think that on x86 one can do that) so this check needs to stay.
Quoted 17 lines
Quoted 17 lines
lib/pmu/rte_pmu.h new file +16
-0+ pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/**
Quoted 9 lines
Quoted 6 lines
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
__rte_internal?
No this cannot be internal because that will make functions calling it internal as well hence apps won't be able to use that. This has already been brought up by one of the reviewers.
Quoted 10 lines
Quoted 10 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */lib/pmu/rte_pmu.h new file +6
-0+__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning
Quoted 9 lines
Quoted 5 lines
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?
This is inlined and has one caller i.e rte_pmu_read().
Quoted 10 lines
Quoted 4 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */Probably worth to mention that this function is not MT safe. Same for _fini_ and
add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.
Yes they are meant to be called from main thread.
Quoted 90 lines
Quoted 90 lines
lib/pmu/rte_pmu.h new file +69
-0+__rte_experimental +int +rte_pmu_init (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Finalize PMU library. This should be called after PMU counters are no longer being read. +*/ +__rte_experimental +void +rte_pmu_fini (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Add event to the group of enabled events. +* +* @param name +* Name of an event listed under /sys/bus/event_source/devices/pmu/events. +* @return +* Event index in case of success, negative value otherwise. +*/ +__rte_experimental +int +rte_pmu_add_event (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read hardware counter configured to count occurrences of an event. +* +* @param index +* Index of an event to be read. +* @return +* Event value read from register. In case of errors or lack of support +* 0 is returned. In other words, stream of zeros in a trace file +* indicates problem with reading particular PMU event register. +*/ +__rte_experimental +static __rte_always_inline uint64_t +rte_pmu_read (unsigned int index) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (unlikely (!rte_pmu.initialized)) +return 0 ; + +if (unlikely (!group->enabled)) { + ret =__rte_pmu_enable_group (); +if (ret) +return 0 ; + } + +if (unlikely (index >= rte_pmu.num_group_events)) +return 0 ; + +return __rte_pmu_read_userpage (group->mmap_pages[index]); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_PMU_H_ */ lib/pmu/version.map new file +15
-0@@ -0,0 +1,15 @@ +DPDK_23 { + local: *; +}; + +EXPERIMENTAL { + global: + + __rte_pmu_enable_group; + per_lcore__event_group; + rte_pmu; + rte_pmu_add_event; + rte_pmu_fini; + rte_pmu_init; + rte_pmu_read; +};
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 307 lines
Quoted 298 lines
Quoted 11 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>Few more comments/questions below.
Quoted 283 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..950f999cb7
lib/pmu/rte_pmu.c new file +278
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> + +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices" + +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> +((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) << +(__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu; + +/* +* Following __rte_weak functions provide default no-op. +Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) { } + +static int +get_term_format (const char *name,int *num,uint64_t *mask) { +char path[PATH_MAX ]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/format/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) { +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +} + +static int +get_event_config (const char *name,uint64_t config[3 ]) { +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); + fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) { +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); } + +static int +open_events (struct rte_pmu_event_group *group) { +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) { +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) { +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE ,PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno;Quoted 8 lines
lib/pmu/rte_pmu.c line 291
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Quoted 7 lines
Nothing special. Any pointers to that thread-local in that thread are invalided.
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for other purposes)?
Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.
I understand that main thread would need to access that data.
I am not sure that it would be able to.
Imagine thread calls rte_pmu_read(...) and then terminates, while program continues to run. As I understand address of its RTE_PER_LCORE(_event_group) will still remain in rte_pmu.event_group_list, even if it is probably not valid any more.
Quoted 212 lines
Quoted 202 lines
Quoted 202 lines
lib/pmu/rte_pmu.c new file +169
-0+rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; +} + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) +{ +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) +{ +struct rte_pmu_event *event; +char path[PATH_MAX ]; + +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/events/%s" , rte_pmu.name, name); +if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file +26
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> +#include <rte_common.h> +#include <rte_compat.h>Quoted 9 lines
lib/pmu/rte_pmu.h line 30
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+#include <rte_spinlock.h> + +/** Maximum number of events in a group */ +#define MAX_NUM_GROUP_EVENTS 8forgot
RTE_prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically byenable_group()?
Quoted 3 lines
8 is reasonable number I think. X86/ARM have actually less that that (was that something like 4?). Moreover events are scheduled as a group so there must be enough hw counters available for that to succeed. So this number should cover current needs.
If you think 8 will be enough to cover all possible future cases - I am ok either way. Still need RTE_ prefix for it.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
lib/pmu/rte_pmu.h new file +6
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */ Quoted 8 lines
lib/pmu/rte_pmu.h line 41
Quoted 5 lines
lib/pmu/rte_pmu.h new file +5
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ +} __rte_cache_aligned; +Even if we'd decide to keep
rte_pmu_read()as static inline (still not sure it is a good idea),
Quoted 2 lines
We want to save as much cpu cycles as we possibly can and inlining does helps in that matter.
Ok, so asking same question from different thread: how many cycles it will save? What is the difference in terms of performance when you have this function inlined vs not inlined?
Quoted 6 lines
Quoted 3 lines
why these two struct below (
rte_pmu_eventandrte_pmu) have to be public? I think both can be safely moved away from public headers.struct
rte_pmu_eventcan be hidden I guess.
structrte_pmuis used in this header hence cannot be moved elsewhere.
Not sure why?
Is that because you use it inside rte_pmu_read()?
But that check I think can be safely moved into __rte_pmu_enable_group() or probably even into rte_pmu_add_event().
Quoted 99 lines
Quoted 59 lines
Quoted 59 lines
lib/pmu/rte_pmu.h new file +57
-0+/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ +}; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ +}; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own version */ +#ifndef rte_pmu_pmc_read +#define rte_pmu_pmc_read (index) ({ 0; }) +#endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t +__rte_pmu_read_userpage (struct perf_event_mmap_page *pc) +{ +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier (); + index = pc->index; + offset = pc->offset;Quoted 9 lines
lib/pmu/rte_pmu.h line 102
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) {In
mmap_events()you return EPERM ifcap_user_rdpmcis not enabled. Do you need another check here? Or this capability can be disabled by kernel at run-time?That extra check in
mmap_event()may be removed actually. Some archs allow disabling reading rdpmc (I think that on x86 one can do that) so this check needs to stay.Quoted 17 lines
Quoted 17 lines
lib/pmu/rte_pmu.h new file +16
-0+ pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/** Quoted 9 lines
Quoted 6 lines
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
__rte_internal?
Quoted 3 lines
No this cannot be internal because that will make functions calling it internal as well hence apps won't be able to use that. This has already been brought up by one of the reviewers.
Ok, then we probably can mark it with ' @internal' tag in formal comments?
Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */lib/pmu/rte_pmu.h new file +6
-0+__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning Quoted 9 lines
Quoted 5 lines
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?
Quoted 1 line
This is inlined and has one caller i.e
rte_pmu_read().
I thought we are talking here about rte_pmu_init().
I don't see where it is inlined and still not clear why it can't be called directly.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */
Quoted 7 lines
Quoted 5 lines
Probably worth to mention that this function is not MT safe. Same for _fini_ and
add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.Yes they are meant to be called from main thread.
Ok, then please add that into formal API comments.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+__rte_experimental +int +rte_pmu_init (void ); +
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 13 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 27 February 2023 21.53Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>
[...]
Quoted 42 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group);Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + + if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
Quoted 5 lines
Quoted 5 lines
lib/pmu/rte_pmu.c
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ ret = -errno; + goto out; + } + + if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) ==-1) {
Quoted 11 lines
Quoted 9 lines
lib/pmu/rte_pmu.c line 291
Quoted 6 lines
lib/pmu/rte_pmu.c new file +6
-0+ ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Nothing special. Any pointers to that thread-local in that thread are
invalided.
Quoted 2 lines
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for other
purposes)?
Quoted 2 lines
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread callsrte_pmu_read(...)and then terminates, while program continues to run.
Is the example you describe here (i.e. a thread terminating in the middle of doing something) really a scenario DPDK is supposed to support?
Quoted 3 lines
As I understand address of its
RTE_PER_LCORE(_event_group)will still remain in rte_pmu.event_group_list,
even if it is probably not valid any more.
There should be a "destructor/done/finish" function available to remove this from the list.
[...]
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
Even if we'd decide to keep
rte_pmu_read()as static inline (still not sure it is a good idea),We want to save as much cpu cycles as we possibly can and inlining does
helps
Quoted 1 line
in that matter.
Ok, so asking same question from different thread: how many cycles it will save?
What is the difference in terms of performance when you have this function inlined vs not inlined?
We expect to use this in our in-house profiler library. For this reason, I have a very strong preference for absolute maximum performance.
Reading PMU events is for performance profiling, so I expect other potential users of the PMU library to share my opinion on this.
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 59 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>[...]
Quoted 45 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group);Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + + if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) == -1) {
Quoted 5 lines
Quoted 5 lines
lib/pmu/rte_pmu.c
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ ret = -errno; + goto out; + } + + if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) ==-1) {
Quoted 11 lines
Quoted 9 lines
lib/pmu/rte_pmu.c line 291
Quoted 6 lines
lib/pmu/rte_pmu.c new file +6
-0+ ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Nothing special. Any pointers to that thread-local in that thread are
invalided.
Quoted 2 lines
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for other
purposes)?
Quoted 4 lines
Quoted 1 line
Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.
Quoted 7 lines
Quoted 4 lines
I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread callsrte_pmu_read(...)and then terminates, while program continues to run.Is the example you describe here (i.e. a thread terminating in the middle of doing something) really a scenario DPDK is supposed to support?
I am not talking about some abnormal termination.
We do have ability to spawn control threads, user can spawn his own thread, all these threads can have limited life-time.
Not to mention about rte_thread_register()/rte_thread_unregister().
Quoted 20 lines
Quoted 3 lines
As I understand address of its
RTE_PER_LCORE(_event_group)will still remain in rte_pmu.event_group_list,
even if it is probably not valid any more.There should be a "destructor/done/finish" function available to remove this from the list.
[...]
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
Even if we'd decide to keep
rte_pmu_read()as static inline (still not sure it is a good idea),We want to save as much cpu cycles as we possibly can and inlining does
helps
Quoted 1 line
in that matter.
Ok, so asking same question from different thread: how many cycles it will save?
What is the difference in terms of performance when you have this function inlined vs not inlined?
Quoted 4 lines
We expect to use this in our in-house profiler library. For this reason, I have a very strong preference for absolute maximum performance.
Reading PMU events is for performance profiling, so I expect other potential users of the PMU library to share my opinion on this.
Well, from my perspective 14 cycles are not that much... Though yes, it would be good to hear more opinions here.
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 68 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Tuesday, 28 February 2023 13.05Quoted 23 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>[...]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + + if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET,
PERF_IOC_FLAG_GROUP) == -Quoted 6 lines
Quoted 6 lines
1) {
Quoted 5 lines
Quoted 5 lines
lib/pmu/rte_pmu.c
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ ret = -errno; + goto out; + } + + if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE,
PERF_IOC_FLAG_GROUP) ==Quoted 26 lines
Quoted 25 lines
-1) {
Quoted 11 lines
Quoted 9 lines
lib/pmu/rte_pmu.c line 291
Quoted 6 lines
lib/pmu/rte_pmu.c new file +6
-0+ ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Nothing special. Any pointers to that thread-local in that thread are
invalided.
Quoted 2 lines
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for other
purposes)?
Quoted 4 lines
Quoted 1 line
Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread callsrte_pmu_read(...)and then terminates, while program continues to run.
Quoted 5 lines
Quoted 1 line
Is the example you describe here (i.e. a thread terminating in the middle of
doing something) really a scenario DPDK is supposed to
Quoted 1 line
support?
I am not talking about some abnormal termination.
Then I misunderstood your example; I thought you meant the tread was terminated while inside the rte_pmu_read() function.
Quoted 5 lines
We do have ability to spawn control threads, user can spawn his own thread, all these
threads can have limited life-time.
Not to mention about rte_thread_register()/rte_thread_unregister().
I agree that normal thread termination should be supported.
Quoted 22 lines
Quoted 1 line
Quoted 1 line
As I understand address of its
RTE_PER_LCORE(_event_group)will stillremain
Quoted 4 lines
Quoted 2 lines
in rte_pmu.event_group_list,
even if it is probably not valid any more.There should be a "destructor/done/finish" function available to remove this
from the list.
Quoted 15 lines
[...]
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
Even if we'd decide to keep
rte_pmu_read()as static inline (still not sure it is a good idea),We want to save as much cpu cycles as we possibly can and inlining does
helps
Quoted 1 line
in that matter.
Ok, so asking same question from different thread: how many cycles it will save?
What is the difference in terms of performance when you have this function inlined vs not inlined?
Quoted 8 lines
Quoted 1 line
We expect to use this in our in-house profiler library. For this reason, I
have a very strong preference for absolute maximum
Quoted 3 lines
performance.
Reading PMU events is for performance profiling, so I expect other potential
users of the PMU library to share my opinion on this.
Well, from my perspective 14 cycles are not that much...
For reference, the i40e testpmd per-core performance report shows that it uses 36 cycles per packet.
This is a total of 1152 cycles per burst of 32 packets. 14 cycles overhead per burst / 1152 cycles per burst = 1.2 % overhead.
But that is not all: If the application's pipeline has three stages, where the PMU counters are read for each stage, the per-invocation overhead of 14 cycles adds up, and the overhead per burst is now 3 * 14 / 1152 = 3.6 %.
Generalizing...
In my example here, the same function with 14 wasted cycles is called three times. It might as well be three individual libraries each wasting 14 cycles in its individual fast path processing function, due to a similarly relaxed attitude regarding wasting 14 cycles.
My point is:
Real applications do much more work than testpmd, so all this "insignificant" extra overhead in the libraries adds up!
Generally, I would like the DPDK Project to remain loyal to its original philosophy, where performance is considered a Key Performance Indicator, and overhead in the fast path is kept at an absolute minimum.
Quoted 1 line
Though yes, it would be good to hear more opinions here.
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
In reply to Konstantin AnanyevQuoted 132 lines
From: Morten Brørup
Sent: Tuesday, 28 February 2023 14.16Quoted 4 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Tuesday, 28 February 2023 13.05Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Add support for programming PMU counters and reading their values
in
Quoted 31 lines
Quoted 22 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>[...]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/pmu/rte_pmu.c new file +10
-0+int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + + if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET,
PERF_IOC_FLAG_GROUP) == -Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
1) {
Quoted 5 lines
Quoted 5 lines
lib/pmu/rte_pmu.c
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ ret = -errno; + goto out; + } + + if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE,
PERF_IOC_FLAG_GROUP) ==Quoted 15 lines
Quoted 15 lines
-1) {
Quoted 11 lines
Quoted 9 lines
lib/pmu/rte_pmu.c line 291
Quoted 6 lines
lib/pmu/rte_pmu.c new file +6
-0+ ret = -errno; + goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Nothing special. Any pointers to that thread-local in that thread are
invalided.
Quoted 2 lines
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for
other
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
purposes)?
Quoted 4 lines
Quoted 1 line
Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread callsrte_pmu_read(...)and then terminates, whileprogram
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
continues to run.
Is the example you describe here (i.e. a thread terminating in the middle
of
Quoted 4 lines
doing something) really a scenario DPDK is supposed to
Quoted 1 line
support?
I am not talking about some abnormal termination.
Then I misunderstood your example; I thought you meant the tread was terminated while inside the
rte_pmu_read()function.Quoted 5 lines
We do have ability to spawn control threads, user can spawn his own thread, all these
threads can have limited life-time.
Not to mention about rte_thread_register()/rte_thread_unregister().I agree that normal thread termination should be supported.
Quoted 6 lines
Quoted 1 line
Quoted 1 line
As I understand address of its
RTE_PER_LCORE(_event_group)will stillremain
Quoted 4 lines
Quoted 2 lines
in rte_pmu.event_group_list,
even if it is probably not valid any more.There should be a "destructor/done/finish" function available to remove
this
Quoted 5 lines
from the list.
Quoted 4 lines
[...]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Even if we'd decide to keep
rte_pmu_read()as static inline (stillnot
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
sure it is a good idea),
We want to save as much cpu cycles as we possibly can and inlining
does
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
helps
Quoted 1 line
in that matter.
Ok, so asking same question from different thread: how many cycles it
will
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
save?
What is the difference in terms of performance when you have thisfunction
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
inlined vs not inlined?
We expect to use this in our in-house profiler library. For this reason, I
have a very strong preference for absolute maximum
Quoted 3 lines
performance.
Reading PMU events is for performance profiling, so I expect other
potential
Quoted 3 lines
users of the PMU library to share my opinion on this.
Well, from my perspective 14 cycles are not that much...
For reference, the i40e testpmd per-core performance report shows that it uses 36 cycles per packet.
Quoted 6 lines
This is a total of 1152 cycles per burst of 32 packets. 14 cycles overhead per burst / 1152 cycles per burst = 1.2 % overhead.
But that is not all: If the application's pipeline has three stages, where the PMU counters are read for each stage, the per-invocation overhead of 14 cycles adds up, and the overhead per burst is now 3 * 14 / 1152 = 3.6 %.
I was too fast on the keyboard here... If the application does more work than testpmd, it certainly also uses more than 1152 cycles to do that work. So please ignore the 3.6 % as a wild exaggeration from an invalid example, and just stick with the 1.2 % overhead - which I still consider significant, and thus worth avoiding.
Quoted 18 lines
Generalizing...
In my example here, the same function with 14 wasted cycles is called three times. It might as well be three individual libraries each wasting 14 cycles in its individual fast path processing function, due to a similarly relaxed attitude regarding wasting 14 cycles.
My point is:
Real applications do much more work than testpmd, so all this "insignificant" extra overhead in the libraries adds up!
Generally, I would like the DPDK Project to remain loyal to its original philosophy, where performance is considered a Key Performance Indicator, and overhead in the fast path is kept at an absolute minimum.
Quoted 1 line
Though yes, it would be good to hear more opinions here.
Re: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 127 lines
Quoted 127 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Add support for programming PMU counters and reading their values
in
Quoted 31 lines
Quoted 22 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>[...]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
(quoted diff) +10
-0+int +__rte_pmu_enable_group(void) +{ + struct rte_pmu_event_group *group = &RTE_PER_LCORE(_event_group); + int ret; + + if (rte_pmu.num_group_events == 0) + return -ENODEV; + + ret = open_events(group);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+ if (ret) + goto out; + + ret = mmap_events(group); + if (ret) + goto out; + + if (ioctl(group->fds[0], PERF_EVENT_IOC_RESET,
PERF_IOC_FLAG_GROUP) == -Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
1) {
Quoted 5 lines
Quoted 5 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ ret = -errno; + goto out; + } + + if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE,
PERF_IOC_FLAG_GROUP) ==Quoted 15 lines
Quoted 15 lines
-1) {
Quoted 11 lines
Quoted 9 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ ret = -errno; + goto out; + } + + rte_spinlock_lock(&rte_pmu.lock); + TAILQ_INSERT_TAIL(&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Nothing special. Any pointers to that thread-local in that thread are
invalided.
Quoted 2 lines
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for
other
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
purposes)?
Quoted 4 lines
Quoted 1 line
Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.I understand that main thread would need to access that data. I am not sure that it would be able to.
Imagine thread callsrte_pmu_read(...)and then terminates, whileprogram
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
continues to run.
Is the example you describe here (i.e. a thread terminating in the middle
of
Quoted 4 lines
doing something) really a scenario DPDK is supposed to
Quoted 1 line
support?
I am not talking about some abnormal termination.
Then I misunderstood your example; I thought you meant the tread was terminated while inside the
rte_pmu_read()function.Quoted 5 lines
We do have ability to spawn control threads, user can spawn his own thread, all these
threads can have limited life-time.
Not to mention about rte_thread_register()/rte_thread_unregister().I agree that normal thread termination should be supported.
Quoted 6 lines
Quoted 1 line
Quoted 1 line
As I understand address of its
RTE_PER_LCORE(_event_group)will stillremain
Quoted 4 lines
Quoted 2 lines
in rte_pmu.event_group_list,
even if it is probably not valid any more.There should be a "destructor/done/finish" function available to remove
this
Quoted 5 lines
from the list.
Quoted 4 lines
[...]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Even if we'd decide to keep
rte_pmu_read()as static inline (stillnot
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
sure it is a good idea),
We want to save as much cpu cycles as we possibly can and inlining
does
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
helps
Quoted 1 line
in that matter.
Ok, so asking same question from different thread: how many cycles it
will
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
save?
What is the difference in terms of performance when you have thisfunction
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
inlined vs not inlined?
We expect to use this in our in-house profiler library. For this reason, I
have a very strong preference for absolute maximum
Quoted 3 lines
performance.
Reading PMU events is for performance profiling, so I expect other
potential
Quoted 3 lines
users of the PMU library to share my opinion on this.
Well, from my perspective 14 cycles are not that much...
For reference, the i40e testpmd per-core performance report shows that it uses 36 cycles per packet.
Quoted 8 lines
Quoted 6 lines
This is a total of 1152 cycles per burst of 32 packets. 14 cycles overhead per burst / 1152 cycles per burst = 1.2 % overhead.
But that is not all: If the application's pipeline has three stages, where the PMU counters are read for each stage, the per-invocation overhead of 14 cycles adds up, and the overhead per burst is now 3 * 14 / 1152 = 3.6 %.
I was too fast on the keyboard here... If the application does more work than testpmd, it certainly also uses more than 1152 cycles to do that work. So please ignore the 3.6 % as a wild exaggeration from an invalid example, and just stick with the 1.2 % overhead - which I still consider significant, and thus worth avoiding.
Wonder can we do both - hide struct rte_pmu_event_group from public API and have inline function to read pmu stats? if we can add a separate function that will allow user to get
struct perf_event_mmap_page * for given event index (or event name),
then later user can use__rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
directly.
Quoted 19 lines
Quoted 19 lines
Generalizing...
In my example here, the same function with 14 wasted cycles is called three times. It might as well be three individual libraries each wasting 14 cycles in its individual fast path processing function, due to a similarly relaxed attitude regarding wasting 14 cycles.
My point is:
Real applications do much more work than testpmd, so all this "insignificant" extra overhead in the libraries adds up!
Generally, I would like the DPDK Project to remain loyal to its original philosophy, where performance is considered a Key Performance Indicator, and overhead in the fast path is kept at an absolute minimum.
Quoted 1 line
Though yes, it would be good to hear more opinions here.
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
In reply to Konstantin AnanyevQuoted 325 lines
-----Original Message-----
From: Konstantin Ananyev <[email protected]>
Sent: Monday, February 27, 2023 9:53 PM
To: Tomasz Duszynski <[email protected]>; Konstantin Ananyev <[email protected]>; [email protected]
Subject: RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU eventsQuoted 73 lines
Quoted 73 lines
Quoted 9 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>Few more comments/questions below.
Quoted 60 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..950f999cb7
lib/pmu/rte_pmu.c new file +55
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> + +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices" + +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> +((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) << +(__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu; + +/* +* Following __rte_weak functions provide default no-op. +Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) +{ } + +static intQuoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+get_term_format(const char *name, int *num, uint64_t *mask) { + char path[PATH_MAX]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name,name);
Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
lib/pmu/rte_pmu.c new file +62
-0+ fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) { +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +}Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+ +static int +get_event_config (const char *name,uint64_t config[3 ]) { +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,name);
Quoted 161 lines
Quoted 147 lines
Quoted 147 lines
lib/pmu/rte_pmu.c new file +147
-0+ fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) { +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); } + +static int +open_events (struct rte_pmu_event_group *group) { +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) { +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) { +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE , +PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno;Quoted 8 lines
lib/pmu/rte_pmu.c line 291
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Nothing special. Any pointers to that thread-local in that thread are invalided.
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for other purposes)?
Quoted 9 lines
Quoted 2 lines
Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.I understand that main thread would need to access that data.
I am not sure that it would be able to.
Imagine thread callsrte_pmu_read(...)and then terminates, while program continues to run. As I understand address of itsRTE_PER_LCORE(_event_group)will still remain in rte_pmu.event_group_list, even if it is probably not valid any more.
Okay got your point. In DPDK that will not happen. We do not spawn/kill lcores in runtime. In other scenarios such approach will not work because once thread terminates it's per-thread-data becomes invalid.
Quoted 208 lines
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
lib/pmu/rte_pmu.c new file +79
-0+rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; } + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) { +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) { +struct rte_pmu_event *event; +char path[PATH_MAX ];Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+ +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,name);
Quoted 119 lines
Quoted 118 lines
Quoted 103 lines
lib/pmu/rte_pmu.c new file +79
-0+if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..6b664c3336
lib/pmu/rte_pmu.h new file +19
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU +and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" {lib/pmu/rte_pmu.h line 29
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8forgot
RTE_prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically byenable_group()?
Quoted 8 lines
Quoted 1 line
8 is reasonable number I think. X86/ARM have actually less that that (was that something like
4?).
Quoted 2 lines
Moreover events are scheduled as a group so there must be enough hw counters available for that to succeed. So this number should cover current needs.
If you think 8 will be enough to cover all possible future cases - I am ok either way. Still need
RTE_prefix for it.
Okay that can be added.
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
lib/pmu/rte_pmu.h
Quoted 6 lines
lib/pmu/rte_pmu.h new file +6
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */
Quoted 9 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 41
Quoted 5 lines
lib/pmu/rte_pmu.h new file +5
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; +Even if we'd decide to keep
rte_pmu_read()as static inline (still not sure it is a good idea),
Quoted 6 lines
Quoted 2 lines
We want to save as much cpu cycles as we possibly can and inlining does helps in that matter.
Ok, so asking same question from different thread: how many cycles it will save? What is the difference in terms of performance when you have this function inlined vs not inlined?
On x86 setup which is not under load, no cpusets configured, etc *just* not inlining rte_pmu_read() decreases performance by roughly 24% (44 vs 58 cpu cycles). At least that is reported by trace_perf_autotest.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
why these two struct below (
rte_pmu_eventandrte_pmu) have to be public? I think both can be safely moved away from public headers.struct
rte_pmu_eventcan be hidden I guess.
structrte_pmuis used in this header hence cannot be moved elsewhere.Not sure why?
Is that because you use it insiderte_pmu_read()?
But that check I think can be safely moved into__rte_pmu_enable_group()or probably even intorte_pmu_add_event().
No, we should not do that. Otherwise we'll need to call function. Even though check will happen early on still function prologue/epilogue will happen. This takes cycles.
Quoted 96 lines
Quoted 96 lines
Quoted 56 lines
Quoted 56 lines
lib/pmu/rte_pmu.h new file +53
-0+/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own +version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t __rte_pmu_read_userpage(struct +perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier (); + index = pc->index; + offset = pc->offset;Quoted 9 lines
lib/pmu/rte_pmu.h line 102
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) {In
mmap_events()you return EPERM ifcap_user_rdpmcis not enabled. Do you need another check here? Or this capability can be disabled by kernel at run-time?That extra check in
mmap_event()may be removed actually. Some archs
allow disabling reading rdpmc (I think that on x86 one can do that) so this check needs to stay.Quoted 26 lines
Quoted 13 lines
lib/pmu/rte_pmu.h new file +12
-0+ pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } +lib/pmu/rte_pmu.h line 124
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+return 0 ; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Enable group of events on the calling lcore. +* +* @warning This should be not called directly.
__rte_internal?
Quoted 6 lines
Quoted 3 lines
No this cannot be internal because that will make functions calling it internal as well hence apps won't be able to use that. This has already been brought up by one of the reviewers.
Ok, then we probably can mark it with ' @internal' tag in formal comments?
I added a warning not to call that directly. Since function is not internal (in DPDK parlance) per se I don’t think we should add more confusion that extra tag.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 6 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */lib/pmu/rte_pmu.h new file +1
-0+__rte_experimentallib/pmu/rte_pmu.h line 139
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+int +__rte_pmu_enable_group (void ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Initialize PMU library. +* +* @warning This should be not called directly. Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?
Quoted 5 lines
Quoted 1 line
This is inlined and has one caller i.e
rte_pmu_read().I thought we are talking here about
rte_pmu_init().
I don't see where it is inlined and still not clear why it can't be called directly.
No this cannot be called by init because groups are configured in runtime. That is why __rte_pmu_enable_group() is called once in rte_pmu_read().
*Other* code should not call that directly. And yes, that is not inlined - my mistake.
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */Quoted 7 lines
Quoted 5 lines
Probably worth to mention that this function is not MT safe. Same for _fini_ and
add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.Yes they are meant to be called from main thread.
Ok, then please add that into formal API comments.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+__rte_experimental +int +rte_pmu_init (void ); +
RE: [EXT] Re: [PATCH v11 1/4] lib: add generic support for reading PMU events
Quoted 319 lines
Quoted 319 lines
Quoted 75 lines
Quoted 75 lines
Quoted 11 lines
Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.
This is especially useful in cases where CPU cores are isolated i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.
Signed-off-by Tomasz Duszynski <[email protected]>Acked-by Morten Brørup <[email protected]>Few more comments/questions below.
Quoted 60 lines
lib/pmu/rte_pmu.c new file mode renamed from lib/pmu/rte_pmu.c +0
-0100644 index 0000000000..950f999cb7
lib/pmu/rte_pmu.c new file +55
-0@@ -0,0 +1,460 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2023 Marvell International Ltd. +*/ + +#include <ctype.h> +#include <dirent.h> +#include <errno.h> +#include <regex.h> +#include <stdlib.h> +#include <string.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/queue.h> +#include <sys/syscall.h> +#include <unistd.h> + +#include <rte_atomic.h> +#include <rte_per_lcore.h> +#include <rte_pmu.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> + +#include "pmu_private.h" + +#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices" + +#define GENMASK_ULL (h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> +((64 - 1 - (h))))) #define FIELD_PREP(m, v) (((uint64_t)(v) << +(__builtin_ffsll(m) - 1)) & (m)) + +RTE_DEFINE_PER_LCORE(struct rte_pmu_event_group, _event_group); +struct rte_pmu rte_pmu; + +/* +* Following __rte_weak functions provide default no-op. +Architectures should override them if +* necessary. +*/ + +int +__rte_weak pmu_arch_init(void) +{ +return 0 ; +} + +void +__rte_weak pmu_arch_fini(void) +{ +} + +void +__rte_weak pmu_arch_fixup_config(uint64_t __rte_unused config[3]) +{ } + +static intQuoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+get_term_format(const char *name, int *num, uint64_t *mask) { + char path[PATH_MAX]; +char *config =NULL ; +int high, low, ret; +FILE *fp; + + *num = *mask =0 ; + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu.name,name);
Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
lib/pmu/rte_pmu.c new file +62
-0+ fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + errno =0 ; + ret =fscanf (fp,"%m[^:]:%d-%d" , &config, &low, &high); +if (ret <2 ) { + ret = -ENODATA ; + goto out; + } +if (errno) { + ret = -errno; + goto out; + } + +if (ret ==2 ) + high = low; + + *mask =GENMASK_ULL (high, low); +/* Last digit should be [012]. If last digit is missing 0 is implied. */ + *num = config[strlen (config) -1 ]; + *num =isdigit (*num) ? *num -'0' :0 ; + + ret =0 ; +out: +free (config); +fclose (fp); + +return ret; +} + +static int +parse_event (char *buf,uint64_t config[3 ]) { +char *token, *term; +int num, ret, val; +uint64_t mask; + + config[0 ] = config[1 ] = config[2 ] =0 ; + + token =strtok (buf,"," ); +while (token) { + errno =0 ; +/* <term>=<value> */ + ret =sscanf (token,"%m[^=]=%i" , &term, &val); +if (ret <1 ) +return -ENODATA ; +if (errno) +return -errno; +if (ret ==1 ) + val =1 ; + + ret =get_term_format (term, &num, &mask); +free (term); +if (ret) +return ret; + + config[num] |=FIELD_PREP (mask, val); + token =strtok (NULL ,"," ); + } + +return 0 ; +}Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+ +static int +get_event_config (const char *name,uint64_t config[3 ]) { +char path[PATH_MAX ], buf[BUFSIZ ]; +FILE *fp; +int ret; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,name);
Quoted 163 lines
Quoted 147 lines
Quoted 147 lines
lib/pmu/rte_pmu.c new file +147
-0+ fp =fopen (path,"r" ); +if (fp ==NULL ) +return -errno; + + ret =fread (buf,1 ,sizeof (buf), fp); +if (ret ==0 ) { +fclose (fp); + +return -EINVAL ; + } +fclose (fp); + buf[ret] ='\0' ; + +return parse_event (buf, config); +} + +static int +do_perf_event_open (uint64_t config[3 ],int group_fd) { +struct perf_event_attr attr = { + .size =sizeof (struct perf_event_attr ), + .type =PERF_TYPE_RAW , + .exclude_kernel =1 , + .exclude_hv =1 , + .disabled =1 , + }; + +pmu_arch_fixup_config (config); + + attr.config = config[0 ]; + attr.config1 = config[1 ]; + attr.config2 = config[2 ]; + +return syscall (SYS_perf_event_open, &attr,0 ,-1 , group_fd,0 ); } + +static int +open_events (struct rte_pmu_event_group *group) { +struct rte_pmu_event *event; +uint64_t config[3 ]; +int num =0 , ret; + +/* group leader gets created first, with fd = -1 */ + group->fds[0 ] =-1 ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { + ret =get_event_config (event->name, config); +if (ret) +continue ; + + ret =do_perf_event_open (config, group->fds[0 ]); +if (ret ==-1 ) { + ret = -errno; + goto out; + } + + group->fds[event->index] = ret; + num++; + } + +return 0 ; +out: +for (--num; num >=0 ; num--) { +close (group->fds[num]); + group->fds[num] =-1 ; + } + + +return ret; +} + +static int +mmap_events (struct rte_pmu_event_group *group) { +long page_size =sysconf (_SC_PAGE_SIZE); +unsigned int i; +void *addr; +int ret; + +for (i =0 ; i < rte_pmu.num_group_events; i++) { + addr =mmap (0 , page_size,PROT_READ ,MAP_SHARED , group->fds[i],0 ); +if (addr ==MAP_FAILED ) { + ret = -errno; + goto out; + } + + group->mmap_pages[i] = addr; +if (!group->mmap_pages[i]->cap_user_rdpmc) { + ret = -EPERM ; + goto out; + } + } + +return 0 ; +out: +for (; i; i--) { +munmap (group->mmap_pages[i -1 ], page_size); + group->mmap_pages[i -1 ] =NULL ; + } + +return ret; +} + +static void +cleanup_events (struct rte_pmu_event_group *group) { +unsigned int i; + +if (group->fds[0 ] !=-1 ) +ioctl (group->fds[0 ],PERF_EVENT_IOC_DISABLE , +PERF_IOC_FLAG_GROUP ); + +for (i =0 ; i < rte_pmu.num_group_events; i++) { +if (group->mmap_pages[i]) { +munmap (group->mmap_pages[i],sysconf (_SC_PAGE_SIZE)); + group->mmap_pages[i] =NULL ; + } + +if (group->fds[i] !=-1 ) { +close (group->fds[i]); + group->fds[i] =-1 ; + } + } + + group->enabled = false; +} + +int +__rte_pmu_enable_group (void ) +{ +struct rte_pmu_event_group *group = &RTE_PER_LCORE (_event_group); +int ret; + +if (rte_pmu.num_group_events ==0 ) +return -ENODEV ; + + ret =open_events (group); +if (ret) + goto out; + + ret =mmap_events (group); +if (ret) + goto out; + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_RESET ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno; + goto out; + } + +if (ioctl (group->fds[0 ],PERF_EVENT_IOC_ENABLE ,PERF_IOC_FLAG_GROUP ) ==-1 ) { + ret = -errno;Quoted 8 lines
lib/pmu/rte_pmu.c line 291
Quoted 5 lines
lib/pmu/rte_pmu.c new file +5
-0+ goto out; + } + +rte_spinlock_lock (&rte_pmu.lock); +TAILQ_INSERT_TAIL (&rte_pmu.event_group_list, group, next);Hmm.. so we insert pointer to TLS variable into the global list? Wonder what would happen if that thread get terminated?
Nothing special. Any pointers to that thread-local in that thread are invalided.
Quoted 2 lines
Can memory from its TLS block get re-used (by other thread or for other purposes)?
Why would any other thread reuse that?
Eventually main thread will need that data to do the cleanup.
Quoted 8 lines
Quoted 6 lines
I understand that main thread would need to access that data.
I am not sure that it would be able to.
Imagine thread callsrte_pmu_read(...)and then terminates, while program continues to run. As I understand address of itsRTE_PER_LCORE(_event_group)will still remain in rte_pmu.event_group_list, even if it is probably not valid any more.Okay got your point. In DPDK that will not happen. We do not spawn/kill lcores in runtime.
Well, yes usually DPDK app doesn't do that, but in theory there is an API to register/unregister non-eal threads as lcores: rte_thread_register()/rte_thread_unregister().
Also besides of lcores there are control threads, some house-keeping threads, plus user is free to spawn/kill his own threads.
Are you saying that this library doesn't support none of them?
If so, then at least that should be very clearly documented.
Though I think a proper way is handle this situation somehow -
either return error at __rte_pmu_enable_group(), or change the code to allow it to work properly from any thread. I don't think it is that hard.
Quoted 241 lines
In other scenarios such approach will not work because once thread terminates it's per-thread-data becomes invalid.
Quoted 208 lines
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
lib/pmu/rte_pmu.c new file +79
-0+rte_spinlock_unlock (&rte_pmu.lock); + group->enabled = true; + +return 0 ; + +out: +cleanup_events (group); + +return ret; +} + +static int +scan_pmus (void ) +{ +char path[PATH_MAX ]; +struct dirent *dent; +const char *name; +DIR *dirp; + + dirp =opendir (EVENT_SOURCE_DEVICES_PATH ); +if (dirp ==NULL ) +return -errno; + +while ((dent =readdir (dirp))) { + name = dent->d_name; +if (name[0 ] =='.' ) +continue ; + +/* sysfs entry should either contain cpus or be a cpu */ +if (!strcmp (name,"cpu" )) +break ; + +snprintf (path,sizeof (path),EVENT_SOURCE_DEVICES_PATH "/%s/cpus" , name); +if (access (path,F_OK ) ==0 ) +break ; + } + +if (dent) { + rte_pmu.name =strdup (name); +if (rte_pmu.name ==NULL ) { +closedir (dirp); + +return -ENOMEM ; + } + } + +closedir (dirp); + +return rte_pmu.name ?0 : -ENODEV ; } + +static struct rte_pmu_event * +new_event (const char *name) +{ +struct rte_pmu_event *event; + + event =calloc (1 ,sizeof (*event)); +if (event ==NULL ) + goto out; + + event->name =strdup (name); +if (event->name ==NULL ) { +free (event); + event =NULL ; + } + +out: +return event; +} + +static void +free_event (struct rte_pmu_event *event) { +free (event->name); +free (event); +} + +int +rte_pmu_add_event (const char *name) { +struct rte_pmu_event *event; +char path[PATH_MAX ];Quoted 8 lines
Quoted 8 lines
lib/pmu/rte_pmu.c
Quoted 8 lines
lib/pmu/rte_pmu.c new file +8
-0+ +if (rte_pmu.name ==NULL ) +return -ENODEV ; + +if (rte_pmu.num_group_events +1 >=MAX_NUM_GROUP_EVENTS ) +return -ENOSPC ; + + snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu.name,name);
Quoted 119 lines
Quoted 118 lines
Quoted 103 lines
lib/pmu/rte_pmu.c new file +79
-0+if (access (path,R_OK )) +return -ENODEV ; + + TAILQ_FOREACH(event, &rte_pmu.event_list, next) { +if (!strcmp (event->name, name)) +return event->index; +continue ; + } + + event =new_event (name); +if (event ==NULL ) +return -ENOMEM ; + + event->index = rte_pmu.num_group_events++; +TAILQ_INSERT_TAIL (&rte_pmu.event_list, event, next); + +return event->index; +} + +int +rte_pmu_init (void ) +{ +int ret; + +/* Allow calling init from multiple contexts within a single thread. This simplifies +* resource management a bit e.g in case fast-path tracepoint has already been enabled +* via command line but application doesn't care enough and performs init/fini again. +*/ +if (rte_pmu.initialized !=0 ) { + rte_pmu.initialized++; +return 0 ; + } + + ret =scan_pmus (); +if (ret) + goto out; + + ret =pmu_arch_init (); +if (ret) + goto out; + +TAILQ_INIT (&rte_pmu.event_list); +TAILQ_INIT (&rte_pmu.event_group_list); +rte_spinlock_init (&rte_pmu.lock); + rte_pmu.initialized =1 ; + +return 0 ; +out: +free (rte_pmu.name); + rte_pmu.name =NULL ; + +return ret; +} + +void +rte_pmu_fini (void ) +{ +struct rte_pmu_event_group *group, *tmp_group; +struct rte_pmu_event *event, *tmp_event; + +/* cleanup once init count drops to zero */ +if (rte_pmu.initialized ==0 || --rte_pmu.initialized !=0 ) +return ; + + RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu.event_list, next, tmp_event) { +TAILQ_REMOVE (&rte_pmu.event_list, event, next); +free_event (event); + } + + RTE_TAILQ_FOREACH_SAFE(group, &rte_pmu.event_group_list, next, tmp_group) { +TAILQ_REMOVE (&rte_pmu.event_group_list, group, next); +cleanup_events (group); + } + +pmu_arch_fini (); +free (rte_pmu.name); + rte_pmu.name =NULL ; + rte_pmu.num_group_events =0 ; +}lib/pmu/rte_pmu.h new file mode renamed from lib/pmu/rte_pmu.h +0
-0100644 index 0000000000..6b664c3336
lib/pmu/rte_pmu.h new file +19
-0@@ -0,0 +1,212 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2023 Marvell +*/ + +#ifndef _RTE_PMU_H_ +#define _RTE_PMU_H_ + +/** + * @file + * + * PMU event tracing operations + * + * This file defines generic API and types necessary to setup PMU +and + * read selected counters in runtime. + */ + +#ifdef __cplusplus +extern "C" {lib/pmu/rte_pmu.h line 29
Quoted 10 lines
lib/pmu/rte_pmu.h new file +10
-0+#endif + +#include <linux/perf_event.h> + +#include <rte_atomic.h> +#include <rte_branch_prediction.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_spinlock.h> + +/** Maximum number of events in a group */ #define +MAX_NUM_GROUP_EVENTS 8forgot
RTE_prefix.
In fact, do you really need number of events in group to be hard-coded? Couldn't mmap_pages[] and fds[] be allocated dynamically byenable_group()?Quoted 8 lines
Quoted 1 line
8 is reasonable number I think. X86/ARM have actually less that that (was that something like
4?).
Quoted 2 lines
Moreover events are scheduled as a group so there must be enough hw counters available for that to succeed. So this number should cover current needs.
If you think 8 will be enough to cover all possible future cases - I am ok either way. Still need
RTE_prefix for it.Okay that can be added.
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
lib/pmu/rte_pmu.h
Quoted 6 lines
lib/pmu/rte_pmu.h new file +6
-0+ +/** +* A structure describing a group of events. +*/ +struct rte_pmu_event_group { +struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS ];/**< array of user pages */
Quoted 11 lines
Quoted 8 lines
lib/pmu/rte_pmu.h line 41
Quoted 5 lines
lib/pmu/rte_pmu.h new file +5
-0+int fds[MAX_NUM_GROUP_EVENTS ];/**< array of event descriptors */ +bool enabled;/**< true if group was enabled on particular lcore */ +TAILQ_ENTRY (rte_pmu_event_group) next;/**< list entry */ } +__rte_cache_aligned; +Even if we'd decide to keep
rte_pmu_read()as static inline (still not sure it is a good idea),We want to save as much cpu cycles as we possibly can and inlining does helps in that matter.
Quoted 7 lines
Quoted 3 lines
Ok, so asking same question from different thread: how many cycles it will save? What is the difference in terms of performance when you have this function inlined vs not inlined?
On x86 setup which is not under load, no cpusets configured, etc *just* not inlining
rte_pmu_read()decreases performance by roughly 24% (44 vs 58 cpu cycles). At least that is reported bytrace_perf_autotest.
From my perspective 14 cycles is not that much...
Considering that user will probably not call it very often, and by enabling measurements he probably already prepared to get some hit.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
why these two struct below (
rte_pmu_eventandrte_pmu) have to be public? I think both can be safely moved away from public headers.struct
rte_pmu_eventcan be hidden I guess.
structrte_pmuis used in this header hence cannot be moved elsewhere.
Quoted 7 lines
Quoted 4 lines
Not sure why?
Is that because you use it insiderte_pmu_read()?
But that check I think can be safely moved into__rte_pmu_enable_group()or probably even intorte_pmu_add_event().No, we should not do that. Otherwise we'll need to call function. Even though check will happen early on still function prologue/epilogue will happen. This takes cycles.
Not necessary. You can store this value in pmu_group itself, and by this value decide is pmu and group initialized, etc.
Quoted 100 lines
Quoted 100 lines
Quoted 99 lines
Quoted 56 lines
Quoted 56 lines
lib/pmu/rte_pmu.h new file +53
-0+/** +* A structure describing an event. +*/ +struct rte_pmu_event { +char *name;/**< name of an event */ +unsigned int index;/**< event index into fds/mmap_pages */ +TAILQ_ENTRY (rte_pmu_event) next;/**< list entry */ }; + +/** +* A PMU state container. +*/ +struct rte_pmu { +char *name;/**< name of core PMU listed under /sys/bus/event_source/devices */ +rte_spinlock_t lock;/**< serialize access to event group list */ + TAILQ_HEAD(, rte_pmu_event_group) event_group_list; /**< list of event groups */ +unsigned int num_group_events;/**< number of events in a group */ + TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */ +unsigned int initialized;/**< initialization counter */ }; + +/** lcore event group */ +RTE_DECLARE_PER_LCORE(struct rte_pmu_event_group, _event_group); + +/** PMU state container */ +extern struct rte_pmu rte_pmu; + +/** Each architecture supporting PMU needs to provide its own +version */ #ifndef rte_pmu_pmc_read#define +rte_pmu_pmc_read (index) ({ 0; }) #endif + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Read PMU counter. +* +* @warning This should be not called directly. +* +* @param pc +* Pointer to the mmapped user page. +* @return +* Counter value read from hardware. +*/ +static __rte_always_inline uint64_t __rte_pmu_read_userpage(struct +perf_event_mmap_page *pc) { +uint64_t width, offset; +uint32_t seq, index; +int64_t pmc; + +for (;;) { + seq = pc->lock; +rte_compiler_barrier (); + index = pc->index; + offset = pc->offset;Quoted 9 lines
lib/pmu/rte_pmu.h line 102
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+ width = pc->pmc_width; + +/* index set to 0 means that particular counter cannot be used */ +if (likely (pc->cap_user_rdpmc && index)) {In
mmap_events()you return EPERM ifcap_user_rdpmcis not enabled. Do you need another check here? Or this capability can be disabled by kernel at run-time?That extra check in
mmap_event()may be removed actually. Some archs
allow disabling reading rdpmc (I think that on x86 one can do that) so this check needs to stay.Quoted 17 lines
Quoted 17 lines
lib/pmu/rte_pmu.h new file +16
-0+ pmc =rte_pmu_pmc_read (index -1 ); + pmc <<=64 - width; + pmc >>=64 - width; + offset += pmc; + } + +rte_compiler_barrier (); + +if (likely (pc->lock == seq)) +return offset; + } + +return 0 ; +} + +/** Quoted 9 lines
Quoted 6 lines
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enable group of events on the calling lcore.
+ *
+ * @warning This should be not called directly.
__rte_internal?No this cannot be internal because that will make functions calling it internal as well hence apps won't be able to use that. This has already been brought up by one of the reviewers.
Quoted 5 lines
Quoted 2 lines
Ok, then we probably can mark it with ' @internal' tag in formal comments?
I added a warning not to call that directly. Since function is not internal (in DPDK parlance) per se I don’t think we should add more confusion that extra tag.
We doing it in other places, why not to add it here?
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */lib/pmu/rte_pmu.h new file +6
-0+__rte_experimental +int +__rte_pmu_enable_group (void ); + +/** +* @warning Quoted 9 lines
Quoted 5 lines
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initialize PMU library.
+ *
+ * @warning This should be not called directly.Hmm.. then who should call it?
If it not supposed to be called directly, why to declare it here?This is inlined and has one caller i.e
rte_pmu_read().
Quoted 8 lines
Quoted 3 lines
I thought we are talking here about
rte_pmu_init().
I don't see where it is inlined and still not clear why it can't be called directly.No this cannot be called by init because groups are configured in runtime. That is why
__rte_pmu_enable_group()is called once inrte_pmu_read().*Other* code should not call that directly. And yes, that is not inlined - my mistake.
Once again: we are discussing comments for rte_pmu_init() function. Why it can't be called directly?
In test_pmu_read() you do call it directly.
Quoted 19 lines
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
+ *
+ * @return
+ * 0 in case of success, negative value otherwise.
+ */Quoted 7 lines
Quoted 5 lines
Probably worth to mention that this function is not MT safe. Same for _fini_ and
add_event.
Also worth to mention that all control-path functions (init/fini/add_event) and data-path (pmu_read) can't be called concurrently.Yes they are meant to be called from main thread.
Ok, then please add that into formal API comments.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/pmu/rte_pmu.h new file +4
-0+__rte_experimental +int +rte_pmu_init (void ); +