Skip to contentSkip to conversation

add support for self monitoring

PATCH Superseded Tomasz Duszynski 14 participants 222 messages Nov 11, 2022 – Jun 10, 2025 Page 1 of 4

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover add support for self monitoring

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 ++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  37 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 103 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 518 +++++++++++++++++++++++
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  11 +
 lib/eal/include/rte_pmu.h                | 207 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   4 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  32 ++
 20 files changed, 1067 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.25.1

#1 lore raw

PATCH 1/4 eal: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 app/test/meson.build                  |   1 +
 app/test/test_pmu.c                   |  41 +++
 doc/guides/prog_guide/profile_app.rst |   8 +
 lib/eal/common/meson.build            |   3 +
 lib/eal/common/pmu_private.h          |  41 +++
 lib/eal/common/rte_pmu.c              | 455 ++++++++++++++++++++++++++
 lib/eal/include/meson.build           |   1 +
 lib/eal/include/rte_pmu.h             | 204 ++++++++++++
 lib/eal/linux/eal.c                   |   4 +
 lib/eal/version.map                   |   3 +
 10 files changed, 761 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
app/test/meson.build +1 -0
index f34d19e3c3..93b3300309 100644
@@ -143,6 +143,7 @@ test_sources = files(
         'test_timer_racecond.c',
         'test_timer_secondary.c',
         'test_ticketlock.c',
+        'test_pmu.c',
         'test_trace.c',
         'test_trace_register.c',
         'test_trace_perf.c',
app/test/test_pmu.c new file +41 -0

41 lines not on this page · raw message

doc/guides/prog_guide/profile_app.rst +8 -0
index bd6700ef85..8fc1b20cab 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
         'rte_service.c',
         'rte_version.c',
 )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
 if is_linux or is_windows
     sources += files('eal_common_dynmem.c')
 endif
lib/eal/common/pmu_private.h new file +41 -0

41 lines not on this page · raw message

lib/eal/common/rte_pmu.c new file +455 -0

455 lines not on this page · raw message

lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+        'rte_pmu.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +204 -0

204 lines not on this page · raw message

lib/eal/linux/eal.c +4 -0
index 8c118d0d9f..751a13b597 100644
@@ -53,6 +53,7 @@
 #include "eal_options.h"
 #include "eal_vfio.h"
 #include "hotplug_mp.h"
+#include "pmu_private.h"
 
 #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	eal_pmu_init();
+
 	if (rte_eal_tailqs_init() < 0) {
 		rte_eal_init_alert("Cannot init tail queues for objects");
 		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
 	eal_bus_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
+	eal_pmu_fini();
 	/* after this point, any DPDK pointers will become dangling */
 	rte_eal_memory_detach();
 	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map +3 -0
index 7ad12a7dc9..e870c87493 100644
@@ -432,6 +432,8 @@ EXPERIMENTAL {
 	rte_thread_set_priority;
 
 	# added in 22.11
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
 	rte_thread_attr_get_affinity;
 	rte_thread_attr_init;
 	rte_thread_attr_set_affinity;
@@ -483,4 +485,5 @@ INTERNAL {
 	rte_mem_map;
 	rte_mem_page_size;
 	rte_mem_unmap;
+	rte_pmu_enable_group;
 };
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH 1/4] eal: add generic support for reading PMU events

Quoted 722 lines · Tomasz Duszynski

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
  app/test/meson.build                  |   1 +
  app/test/test_pmu.c                   |  41 +++
  doc/guides/prog_guide/profile_app.rst |   8 +
  lib/eal/common/meson.build            |   3 +
  lib/eal/common/pmu_private.h          |  41 +++
  lib/eal/common/rte_pmu.c              | 455 ++++++++++++++++++++++++++
  lib/eal/include/meson.build           |   1 +
  lib/eal/include/rte_pmu.h             | 204 ++++++++++++
  lib/eal/linux/eal.c                   |   4 +
  lib/eal/version.map                   |   3 +
  10 files changed, 761 insertions(+)

create mode 100644 app/test/test_pmu.c
create mode 100644 lib/eal/common/pmu_private.h
create mode 100644 lib/eal/common/rte_pmu.c
create mode 100644 lib/eal/include/rte_pmu.h

app/test/meson.build +1 -0
index f34d19e3c3..93b3300309 100644
@@ -143,6 +143,7 @@ test_sources = files(
          'test_timer_racecond.c',
          'test_timer_secondary.c',
          'test_ticketlock.c',
+        'test_pmu.c',
          'test_trace.c',
          'test_trace_register.c',
          'test_trace_perf.c',
app/test/test_pmu.c new file +41 -0
new file mode 100644
index 0000000000..fd331af9ee
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <rte_pmu.h>
+
+#include "test.h"
+
+static int
+test_pmu_read(void)
+{
+	uint64_t val = 0;
+	int tries = 10;
+	int event = -1;
+
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	if (val == 0)
+		return TEST_FAILED;
+
+	return TEST_SUCCESS;
+}
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_pmu(void)
+{
+	return unit_test_suite_runner(&pmu_tests);
+}
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/guides/prog_guide/profile_app.rst +8 -0
index bd6700ef85..8fc1b20cab 100644
@@ -7,6 +7,14 @@ Profile Your Application
  The following sections describe methods of profiling DPDK applications on
  different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
  Profiling on x86
  ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
          'rte_service.c',
          'rte_version.c',
  )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
  if is_linux or is_windows
      sources += files('eal_common_dynmem.c')
  endif
lib/eal/common/pmu_private.h new file +41 -0
new file mode 100644
index 0000000000..cade4245e6
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+/**
+ * Initialize PMU tracing internals.
+ */
+void
+eal_pmu_init(void);
+
+/**
+ * Cleanup PMU internals.
+ */
+void
+eal_pmu_fini(void);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/eal/common/rte_pmu.c new file +455 -0
new file mode 100644
index 0000000000..7d3bd57d1d
@@ -0,0 +1,455 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_pmu.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h)))))
+#endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m))
+#endif
+
+struct rte_pmu *pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op. Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t config[3])
+{
+	RTE_SET_USED(config);
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask)
+{
+	char *config = NULL;
+	char path[PATH_MAX];
+	int high, low, ret;
+	FILE *fp;
+
+	/* quiesce -Wmaybe-uninitialized warning */
+	*num = 0;
+	*mask = 0;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", pmu->name, name);
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3])
+{
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3])
+{
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", pmu->name, name);
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int lcore_id, int group_fd)
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, rte_gettid(), rte_lcore_to_cpu_id(lcore_id),
+		       group_fd, 0);
+}
+
+static int
+open_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &pmu->group[lcore_id];
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &pmu->event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to get %s event config\n", event->name);
+			continue;
+		}
+
+		ret = do_perf_event_open(config, lcore_id, group->fds[0]);
+		if (ret == -1) {
+			if (errno == EOPNOTSUPP)
+				RTE_LOG(ERR, EAL, "64 bit counters not supported\n");
+
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &pmu->group[lcore_id];
+	void *addr;
+	int ret, i;
+
+	for (i = 0; i < pmu->num_group_events; i++) {
+		addr = mmap(0, rte_mem_page_size(), PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], rte_mem_page_size());
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &pmu->group[lcore_id];
+	int i;
+
+	if (!group->fds)
+		return;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < pmu->num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], rte_mem_page_size());
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	rte_free(group->mmap_pages);
+	rte_free(group->fds);
+
+	group->mmap_pages = NULL;
+	group->fds = NULL;
+	group->enabled = false;
+}
+
+int __rte_noinline
+rte_pmu_enable_group(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &pmu->group[lcore_id];
+	int ret;
+
+	if (pmu->num_group_events == 0) {
+		RTE_LOG(DEBUG, EAL, "no matching PMU events\n");
+
+		return 0;
+	}
+
+	group->fds = rte_zmalloc(NULL, pmu->num_group_events, sizeof(*group->fds));
+	if (!group->fds) {
+		RTE_LOG(ERR, EAL, "failed to alloc descriptor memory\n");
+
+		return -ENOMEM;
+	}
+
+	group->mmap_pages = rte_zmalloc(NULL, pmu->num_group_events, sizeof(*group->mmap_pages));
+	if (!group->mmap_pages) {
+		RTE_LOG(ERR, EAL, "failed to alloc userpage memory\n");
+
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ret = open_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to open events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	ret = mmap_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to map events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		RTE_LOG(ERR, EAL, "failed to enable events on lcore-worker-%d\n", lcore_id);
+
+		ret = -errno;
+		goto out;
+	}
+
+	return 0;
+
+out:
+	cleanup_events(lcore_id);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (!dirp)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	closedir(dirp);
+
+	if (dent) {
+		pmu->name = strdup(name);
+		if (!pmu->name)
+			return -ENOMEM;
+	}
+
+	return pmu->name ? 0 : -ENODEV;
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", pmu->name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &pmu->event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = rte_zmalloc(NULL, 1, sizeof(*event));
+	if (!event)
+		return -ENOMEM;
+
+	event->name = strdup(name);
+	if (!event->name) {
+		rte_free(event);
+
+		return -ENOMEM;
+	}
+
+	event->index = pmu->num_group_events++;
+	TAILQ_INSERT_TAIL(&pmu->event_list, event, next);
+
+	RTE_LOG(DEBUG, EAL, "%s even added at index %d\n", name, event->index);
+
+	return event->index;
+}
+
+void
+eal_pmu_init(void)
+{
+	int ret;
+
+	pmu = rte_calloc(NULL, 1, sizeof(*pmu), RTE_CACHE_LINE_SIZE);
+	if (!pmu) {
+		RTE_LOG(ERR, EAL, "failed to alloc PMU\n");
+
+		return;
+	}
+
+	TAILQ_INIT(&pmu->event_list);
+
+	ret = scan_pmus();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to find core pmu\n");
+		goto out;
+	}
+
+	ret = pmu_arch_init();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to setup arch for PMU\n");
+		goto out;
+	}
+
+	return;
+out:
+	free(pmu->name);
+	rte_free(pmu);
+}
+
+void
+eal_pmu_fini(void)
+{
+	struct rte_pmu_event *event, *tmp;
+	int lcore_id;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &pmu->event_list, next, tmp) {
+		TAILQ_REMOVE(&pmu->event_list, event, next);
+		free(event->name);
+		rte_free(event);
+	}
+
+	RTE_LCORE_FOREACH_WORKER(lcore_id)
+		cleanup_events(lcore_id);
+
+	pmu_arch_fini();
+	free(pmu->name);
+	rte_free(pmu);
+}
lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
          'rte_pci_dev_features.h',
          'rte_per_lcore.h',
          'rte_pflock.h',
+        'rte_pmu.h',
          'rte_random.h',
          'rte_reciprocal.h',
          'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +79 -0
new file mode 100644
index 0000000000..5955c22779
@@ -0,0 +1,204 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef RTE_EXEC_ENV_LINUX
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_lcore.h>
+#include <rte_log.h>
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped perf_event_attr structures */
+	bool enabled; /**< true if group was enabled on particular lcore */
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */
+};
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under /sys/bus/event_source/devices */
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore event group data */
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+};
+
+/** Pointer to the PMU state container */
+extern struct rte_pmu *pmu;
+
+/** Each architecture supporting PMU needs to provide its own version */
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal

lib/eal/include/rte_pmu.h line 89

Quoted 10 lines
lib/eal/include/rte_pmu.h new file +10 -0
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();

I'm guessing this should be a load-acquire instead. Less heavy-handed than a compiler barrier on TSO CPUs, and works on weakly ordered systems as well (unlike the compiler barrier).

This looks like an open-coded sequence lock, so take a look in rte_seqcount.h for inspiration.

Quoted 163 lines
lib/eal/include/rte_pmu.h new file +115 -0
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return pmc + offset;
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get perf_event_mmap_page lock\n");
+			break;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!pmu)
+		return 0;
+
+	group = &pmu->group[lcore_id];
+	if (!group->enabled) {
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}
+
+	if (index < 0 || index >= pmu->num_group_events)
+		return 0;
+
+	return rte_pmu_read_userpage(group->mmap_pages[index]);
+}
+
+#else /* !RTE_EXEC_ENV_LINUX */
+
+__rte_experimental
+static int __rte_unused
+rte_pmu_add_event(__rte_unused const char *name)
+{
+	return -1;
+}
+
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(__rte_unused int index)
+{
+	return 0;
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/eal/linux/eal.c +4 -0
index 8c118d0d9f..751a13b597 100644
@@ -53,6 +53,7 @@
  #include "eal_options.h"
  #include "eal_vfio.h"
  #include "hotplug_mp.h"
+#include "pmu_private.h"
 
  #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
  		return -1;
  	}
 
+	eal_pmu_init();
+
  	if (rte_eal_tailqs_init() < 0) {
  		rte_eal_init_alert("Cannot init tail queues for objects");
  		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
  	eal_bus_cleanup();
  	rte_trace_save();
  	eal_trace_fini();
+	eal_pmu_fini();
  	/* after this point, any DPDK pointers will become dangling */
  	rte_eal_memory_detach();
  	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map +3 -0
index 7ad12a7dc9..e870c87493 100644
@@ -432,6 +432,8 @@ EXPERIMENTAL {
  	rte_thread_set_priority;
 
  	# added in 22.11
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
  	rte_thread_attr_get_affinity;
  	rte_thread_attr_init;
  	rte_thread_attr_set_affinity;
@@ -483,4 +485,5 @@ INTERNAL {
  	rte_mem_map;
  	rte_mem_page_size;
  	rte_mem_unmap;
+	rte_pmu_enable_group;
  };
#3 lore raw

PATCH 2/4 eal/arm: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c               |   4 ++
 lib/eal/arm/include/meson.build   |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h |  37 +++++++++++
 lib/eal/arm/meson.build           |   4 ++
 lib/eal/arm/rte_pmu.c             | 103 ++++++++++++++++++++++++++++++
 lib/eal/include/rte_pmu.h         |   3 +
 6 files changed, 152 insertions(+)
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
app/test/test_pmu.c +4 -0
index fd331af9ee..f94866dff9 100644
@@ -13,6 +13,10 @@ test_pmu_read(void)
 	int tries = 10;
 	int event = -1;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/eal/arm/include/meson.build +1 -0
index 657bf58569..ab13b0220a 100644
@@ -20,6 +20,7 @@ arch_headers = files(
         'rte_pause_32.h',
         'rte_pause_64.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch_32.h',
         'rte_prefetch_64.h',
lib/eal/arm/include/rte_pmu_pmc.h new file +37 -0
new file mode 100644
index 0000000000..5efc851cb8
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_ARM_H_
+#define _RTE_PMU_PMC_ARM_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_ARM_H_ */
lib/eal/arm/meson.build +4 -0
index dca1106aae..0c5575b197 100644
@@ -9,3 +9,7 @@ sources += files(
         'rte_hypervisor.c',
         'rte_power_intrinsics.c',
 )
+
+if is_linux
+    sources += files('rte_pmu.c')
+endif
lib/eal/arm/rte_pmu.c new file +103 -0

103 lines not on this page · raw message

lib/eal/include/rte_pmu.h +3 -0
index 5955c22779..67b1194a2a 100644
@@ -20,6 +20,9 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
+#if defined(RTE_ARCH_ARM64)
+#include <rte_pmu_pmc.h>
+#endif
 
 /**
  * @file
Signature · 1 line
2.25.1
#4 lore raw

PATCH 3/4 eal/x86: support reading Intel PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel PMU events in runtime.

---
 app/test/test_pmu.c               |  2 ++
 lib/eal/include/rte_pmu.h         |  2 +-
 lib/eal/x86/include/meson.build   |  1 +
 lib/eal/x86/include/rte_pmu_pmc.h | 32 +++++++++++++++++++++++++++++++
 4 files changed, 36 insertions(+), 1 deletion(-)
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h
app/test/test_pmu.c +2 -0
index f94866dff9..016204c083 100644
@@ -15,6 +15,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/eal/include/rte_pmu.h +1 -1
index 67b1194a2a..bbe12d100d 100644
@@ -20,7 +20,7 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
-#if defined(RTE_ARCH_ARM64)
+#if defined(RTE_ARCH_ARM64) || defined(RTE_ARCH_X86_64)
 #include <rte_pmu_pmc.h>
 #endif
 
lib/eal/x86/include/meson.build +1 -0
index 52d2f8e969..03d286ed25 100644
@@ -9,6 +9,7 @@ arch_headers = files(
         'rte_io.h',
         'rte_memcpy.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch.h',
         'rte_rtm.h',
lib/eal/x86/include/rte_pmu_pmc.h new file +32 -0
new file mode 100644
index 0000000000..6ecb27a1eb
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_X86_H_
+#define _RTE_PMU_PMC_X86_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint32_t high, low;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return ((uint64_t)high << 32) | (uint64_t)low;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_X86_H_ */
Signature · 1 line
2.25.1
#5 lore raw

PATCH 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               |  4 ++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 ++++++++++++
 lib/eal/common/eal_common_trace_points.c |  3 ++
 lib/eal/common/rte_pmu.c                 | 63 ++++++++++++++++++++++++
 lib/eal/include/rte_eal_trace.h          | 11 +++++
 lib/eal/version.map                      |  1 +
 7 files changed, 119 insertions(+)
app/test/test_trace_perf.c +4 -0
index 46ae7d8074..4851b6852f 100644
@@ -114,6 +114,8 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +124,7 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+WORKER_DEFINE(READ_PMU)
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +177,7 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index 8fc1b20cab..977800ea01 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..9a845fd86f 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace_points.c +3 -0
index 0b0b254615..de918ca618 100644
@@ -75,3 +75,6 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
lib/eal/common/rte_pmu.c +63 -0

86 lines not on this page · raw message

lib/eal/include/rte_eal_trace.h +11 -0
index 5ef4398230..2a10f63e97 100644
@@ -17,6 +17,7 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#include <rte_pmu.h>
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +280,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+/* PMU */
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(int index),
+	uint64_t val;
+	rte_trace_point_emit_int(index);
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/version.map +1 -0
index e870c87493..d6ec3f3b0e 100644
@@ -432,6 +432,7 @@ EXPERIMENTAL {
 	rte_thread_set_priority;
 
 	# added in 22.11
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 	rte_pmu_add_event; # WINDOWS_NO_EXPORT
 	rte_pmu_read; # WINDOWS_NO_EXPORT
 	rte_thread_attr_get_affinity;
Signature · 1 line
2.25.1
#6 lore raw

v2 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 ++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 103 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 519 +++++++++++++++++++++++
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  11 +
 lib/eal/include/rte_pmu.h                | 207 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   6 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1073 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.25.1

#7 lore raw

PATCH v2 1/4 eal: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 app/test/meson.build                  |   1 +
 app/test/test_pmu.c                   |  41 +++
 doc/guides/prog_guide/profile_app.rst |   8 +
 lib/eal/common/meson.build            |   3 +
 lib/eal/common/pmu_private.h          |  41 +++
 lib/eal/common/rte_pmu.c              | 456 ++++++++++++++++++++++++++
 lib/eal/include/meson.build           |   1 +
 lib/eal/include/rte_pmu.h             | 204 ++++++++++++
 lib/eal/linux/eal.c                   |   4 +
 lib/eal/version.map                   |   5 +
 10 files changed, 764 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
app/test/meson.build +1 -0
index f34d19e3c3..93b3300309 100644
@@ -143,6 +143,7 @@ test_sources = files(
         'test_timer_racecond.c',
         'test_timer_secondary.c',
         'test_ticketlock.c',
+        'test_pmu.c',
         'test_trace.c',
         'test_trace_register.c',
         'test_trace_perf.c',
app/test/test_pmu.c new file +41 -0

41 lines not on this page · raw message

doc/guides/prog_guide/profile_app.rst +8 -0
index bd6700ef85..8fc1b20cab 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
         'rte_service.c',
         'rte_version.c',
 )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
 if is_linux or is_windows
     sources += files('eal_common_dynmem.c')
 endif
lib/eal/common/pmu_private.h new file +41 -0

41 lines not on this page · raw message

lib/eal/common/rte_pmu.c new file +456 -0

456 lines not on this page · raw message

lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+        'rte_pmu.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +204 -0

204 lines not on this page · raw message

lib/eal/linux/eal.c +4 -0
index 8c118d0d9f..751a13b597 100644
@@ -53,6 +53,7 @@
 #include "eal_options.h"
 #include "eal_vfio.h"
 #include "hotplug_mp.h"
+#include "pmu_private.h"
 
 #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	eal_pmu_init();
+
 	if (rte_eal_tailqs_init() < 0) {
 		rte_eal_init_alert("Cannot init tail queues for objects");
 		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
 	eal_bus_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
+	eal_pmu_fini();
 	/* after this point, any DPDK pointers will become dangling */
 	rte_eal_memory_detach();
 	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map +5 -0
index 7ad12a7dc9..1ebd842f34 100644
@@ -440,6 +440,10 @@ EXPERIMENTAL {
 	rte_thread_detach;
 	rte_thread_equal;
 	rte_thread_join;
+
+	# added in 23.03
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
 };
 
 INTERNAL {
@@ -483,4 +487,5 @@ INTERNAL {
 	rte_mem_map;
 	rte_mem_page_size;
 	rte_mem_unmap;
+	rte_pmu_enable_group;
 };
Signature · 1 line
2.25.1
#8 lore raw

PATCH v2 2/4 eal/arm: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c               |   4 ++
 lib/eal/arm/include/meson.build   |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h |  39 +++++++++++
 lib/eal/arm/meson.build           |   4 ++
 lib/eal/arm/rte_pmu.c             | 103 ++++++++++++++++++++++++++++++
 lib/eal/include/rte_pmu.h         |   3 +
 6 files changed, 154 insertions(+)
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
app/test/test_pmu.c +4 -0
index fd331af9ee..f94866dff9 100644
@@ -13,6 +13,10 @@ test_pmu_read(void)
 	int tries = 10;
 	int event = -1;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/eal/arm/include/meson.build +1 -0
index 657bf58569..ab13b0220a 100644
@@ -20,6 +20,7 @@ arch_headers = files(
         'rte_pause_32.h',
         'rte_pause_64.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch_32.h',
         'rte_prefetch_64.h',
lib/eal/arm/include/rte_pmu_pmc.h new file +39 -0
new file mode 100644
index 0000000000..10e2984813
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_ARM_H_
+#define _RTE_PMU_PMC_ARM_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_ARM_H_ */
lib/eal/arm/meson.build +4 -0
index dca1106aae..0c5575b197 100644
@@ -9,3 +9,7 @@ sources += files(
         'rte_hypervisor.c',
         'rte_power_intrinsics.c',
 )
+
+if is_linux
+    sources += files('rte_pmu.c')
+endif
lib/eal/arm/rte_pmu.c new file +103 -0

103 lines not on this page · raw message

lib/eal/include/rte_pmu.h +3 -0
index 5955c22779..67b1194a2a 100644
@@ -20,6 +20,9 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
+#if defined(RTE_ARCH_ARM64)
+#include <rte_pmu_pmc.h>
+#endif
 
 /**
  * @file
Signature · 1 line
2.25.1
#9 lore raw

PATCH v2 3/4 eal/x86: support reading Intel PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel PMU events in runtime.

---
 app/test/test_pmu.c               |  2 ++
 lib/eal/include/rte_pmu.h         |  2 +-
 lib/eal/x86/include/meson.build   |  1 +
 lib/eal/x86/include/rte_pmu_pmc.h | 33 +++++++++++++++++++++++++++++++
 4 files changed, 37 insertions(+), 1 deletion(-)
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h
app/test/test_pmu.c +2 -0
index f94866dff9..016204c083 100644
@@ -15,6 +15,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/eal/include/rte_pmu.h +1 -1
index 67b1194a2a..bbe12d100d 100644
@@ -20,7 +20,7 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
-#if defined(RTE_ARCH_ARM64)
+#if defined(RTE_ARCH_ARM64) || defined(RTE_ARCH_X86_64)
 #include <rte_pmu_pmc.h>
 #endif
 
lib/eal/x86/include/meson.build +1 -0
index 52d2f8e969..03d286ed25 100644
@@ -9,6 +9,7 @@ arch_headers = files(
         'rte_io.h',
         'rte_memcpy.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch.h',
         'rte_rtm.h',
lib/eal/x86/include/rte_pmu_pmc.h new file +33 -0
new file mode 100644
index 0000000000..a2cd849fb1
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_X86_H_
+#define _RTE_PMU_PMC_X86_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint32_t high, low;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return ((uint64_t)high << 32) | (uint64_t)low;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_X86_H_ */
Signature · 1 line
2.25.1
#10 lore raw

PATCH v2 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               |  4 ++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 ++++++++++++
 lib/eal/common/eal_common_trace_points.c |  3 ++
 lib/eal/common/rte_pmu.c                 | 63 ++++++++++++++++++++++++
 lib/eal/include/rte_eal_trace.h          | 11 +++++
 lib/eal/version.map                      |  1 +
 7 files changed, 119 insertions(+)
app/test/test_trace_perf.c +4 -0
index 46ae7d8074..4851b6852f 100644
@@ -114,6 +114,8 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +124,7 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+WORKER_DEFINE(READ_PMU)
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +177,7 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index 8fc1b20cab..977800ea01 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..9a845fd86f 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace_points.c +3 -0
index 0b0b254615..de918ca618 100644
@@ -75,3 +75,6 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
lib/eal/common/rte_pmu.c +63 -0

86 lines not on this page · raw message

lib/eal/include/rte_eal_trace.h +11 -0
index 5ef4398230..2a10f63e97 100644
@@ -17,6 +17,7 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#include <rte_pmu.h>
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +280,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+/* PMU */
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(int index),
+	uint64_t val;
+	rte_trace_point_emit_int(index);
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/version.map +1 -0
index 1ebd842f34..b49a430c84 100644
@@ -442,6 +442,7 @@ EXPERIMENTAL {
 	rte_thread_join;
 
 	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 	rte_pmu_add_event; # WINDOWS_NO_EXPORT
 	rte_pmu_read; # WINDOWS_NO_EXPORT
 };
Signature · 1 line
2.25.1
#11 lore raw

v3 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 ++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 104 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 520 +++++++++++++++++++++++
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  11 +
 lib/eal/include/rte_pmu.h                | 207 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   7 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1076 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.25.1

#12 lore raw

PATCH v3 1/4 eal: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 app/test/meson.build                  |   1 +
 app/test/test_pmu.c                   |  41 +++
 doc/guides/prog_guide/profile_app.rst |   8 +
 lib/eal/common/meson.build            |   3 +
 lib/eal/common/pmu_private.h          |  41 +++
 lib/eal/common/rte_pmu.c              | 457 ++++++++++++++++++++++++++
 lib/eal/include/meson.build           |   1 +
 lib/eal/include/rte_pmu.h             | 204 ++++++++++++
 lib/eal/linux/eal.c                   |   4 +
 lib/eal/version.map                   |   6 +
 10 files changed, 766 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
app/test/meson.build +1 -0
index f34d19e3c3..93b3300309 100644
@@ -143,6 +143,7 @@ test_sources = files(
         'test_timer_racecond.c',
         'test_timer_secondary.c',
         'test_ticketlock.c',
+        'test_pmu.c',
         'test_trace.c',
         'test_trace_register.c',
         'test_trace_perf.c',
app/test/test_pmu.c new file +41 -0

41 lines not on this page · raw message

doc/guides/prog_guide/profile_app.rst +8 -0
index bd6700ef85..8fc1b20cab 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
         'rte_service.c',
         'rte_version.c',
 )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
 if is_linux or is_windows
     sources += files('eal_common_dynmem.c')
 endif
lib/eal/common/pmu_private.h new file +41 -0

41 lines not on this page · raw message

lib/eal/common/rte_pmu.c new file +457 -0

457 lines not on this page · raw message

lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+        'rte_pmu.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +204 -0

204 lines not on this page · raw message

lib/eal/linux/eal.c +4 -0
index 8c118d0d9f..751a13b597 100644
@@ -53,6 +53,7 @@
 #include "eal_options.h"
 #include "eal_vfio.h"
 #include "hotplug_mp.h"
+#include "pmu_private.h"
 
 #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	eal_pmu_init();
+
 	if (rte_eal_tailqs_init() < 0) {
 		rte_eal_init_alert("Cannot init tail queues for objects");
 		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
 	eal_bus_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
+	eal_pmu_fini();
 	/* after this point, any DPDK pointers will become dangling */
 	rte_eal_memory_detach();
 	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map +6 -0
index 7ad12a7dc9..9225f46f67 100644
@@ -440,6 +440,11 @@ EXPERIMENTAL {
 	rte_thread_detach;
 	rte_thread_equal;
 	rte_thread_join;
+
+	# added in 23.03
+	rte_pmu; # WINDOWS_NO_EXPORT
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
 };
 
 INTERNAL {
@@ -483,4 +488,5 @@ INTERNAL {
 	rte_mem_map;
 	rte_mem_page_size;
 	rte_mem_unmap;
+	rte_pmu_enable_group;
 };
Signature · 1 line
2.25.1
#13 lore raw

Re: [PATCH v3 1/4] eal: add generic support for reading PMU events

Hi Tomasz,

Quoted 617 lines · Tomasz Duszynski

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
  app/test/meson.build                  |   1 +
  app/test/test_pmu.c                   |  41 +++
  doc/guides/prog_guide/profile_app.rst |   8 +
  lib/eal/common/meson.build            |   3 +
  lib/eal/common/pmu_private.h          |  41 +++
  lib/eal/common/rte_pmu.c              | 457 ++++++++++++++++++++++++++
  lib/eal/include/meson.build           |   1 +
  lib/eal/include/rte_pmu.h             | 204 ++++++++++++
  lib/eal/linux/eal.c                   |   4 +
  lib/eal/version.map                   |   6 +
  10 files changed, 766 insertions(+)

create mode 100644 app/test/test_pmu.c
create mode 100644 lib/eal/common/pmu_private.h
create mode 100644 lib/eal/common/rte_pmu.c
create mode 100644 lib/eal/include/rte_pmu.h

app/test/meson.build +1 -0
index f34d19e3c3..93b3300309 100644
@@ -143,6 +143,7 @@ test_sources = files(
          'test_timer_racecond.c',
          'test_timer_secondary.c',
          'test_ticketlock.c',
+        'test_pmu.c',
          'test_trace.c',
          'test_trace_register.c',
          'test_trace_perf.c',
app/test/test_pmu.c new file +41 -0
new file mode 100644
index 0000000000..fd331af9ee
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <rte_pmu.h>
+
+#include "test.h"
+
+static int
+test_pmu_read(void)
+{
+	uint64_t val = 0;
+	int tries = 10;
+	int event = -1;
+
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	if (val == 0)
+		return TEST_FAILED;
+
+	return TEST_SUCCESS;
+}
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_pmu(void)
+{
+	return unit_test_suite_runner(&pmu_tests);
+}
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/guides/prog_guide/profile_app.rst +8 -0
index bd6700ef85..8fc1b20cab 100644
@@ -7,6 +7,14 @@ Profile Your Application
  The following sections describe methods of profiling DPDK applications on
  different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
  Profiling on x86
  ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
          'rte_service.c',
          'rte_version.c',
  )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
  if is_linux or is_windows
      sources += files('eal_common_dynmem.c')
  endif
lib/eal/common/pmu_private.h new file +41 -0
new file mode 100644
index 0000000000..cade4245e6
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+/**
+ * Initialize PMU tracing internals.
+ */
+void
+eal_pmu_init(void);
+
+/**
+ * Cleanup PMU internals.
+ */
+void
+eal_pmu_fini(void);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/eal/common/rte_pmu.c new file +447 -0
new file mode 100644
index 0000000000..6763005903
@@ -0,0 +1,457 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
+#include <rte_pmu.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h)))))
+#endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m))
+#endif
+
+struct rte_pmu *rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op. Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t config[3])
+{
+	RTE_SET_USED(config);
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask)
+{
+	char *config = NULL;
+	char path[PATH_MAX];
+	int high, low, ret;
+	FILE *fp;
+
+	/* quiesce -Wmaybe-uninitialized warning */
+	*num = 0;
+	*mask = 0;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu->name, name);
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3])
+{
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3])
+{
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu->name, name);
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int lcore_id, int group_fd)
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, rte_gettid(), rte_lcore_to_cpu_id(lcore_id),
+		       group_fd, 0);
+}
+
+static int
+open_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to get %s event config\n", event->name);
+			continue;
+		}
+
+		ret = do_perf_event_open(config, lcore_id, group->fds[0]);
+		if (ret == -1) {
+			if (errno == EOPNOTSUPP)
+				RTE_LOG(ERR, EAL, "64 bit counters not supported\n");
+
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	void *addr;
+	int ret, i;
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		addr = mmap(0, rte_mem_page_size(), PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], rte_mem_page_size());
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int i;
+
+	if (!group->fds)
+		return;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], rte_mem_page_size());
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	rte_free(group->mmap_pages);
+	rte_free(group->fds);
+
+	group->mmap_pages = NULL;
+	group->fds = NULL;
+	group->enabled = false;
+}
+
+int __rte_noinline
+rte_pmu_enable_group(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int ret;
+
+	if (rte_pmu->num_group_events == 0) {
+		RTE_LOG(DEBUG, EAL, "no matching PMU events\n");
+
+		return 0;
+	}
+
+	group->fds = rte_calloc(NULL, rte_pmu->num_group_events, sizeof(*group->fds), 0);
+	if (!group->fds) {
+		RTE_LOG(ERR, EAL, "failed to alloc descriptor memory\n");
+
+		return -ENOMEM;
+	}
+
+	group->mmap_pages = rte_calloc(NULL, rte_pmu->num_group_events, sizeof(*group->mmap_pages), 0);
+	if (!group->mmap_pages) {
+		RTE_LOG(ERR, EAL, "failed to alloc userpage memory\n");
+
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ret = open_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to open events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	ret = mmap_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to map events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		RTE_LOG(ERR, EAL, "failed to enable events on lcore-worker-%d\n", lcore_id);
+
+		ret = -errno;
+		goto out;
+	}
+
+	return 0;
+
+out:
+	cleanup_events(lcore_id);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (!dirp)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	closedir(dirp);
+
+	if (dent) {
+		rte_pmu->name = strdup(name);
+		if (!rte_pmu->name)
+			return -ENOMEM;
+	}
+
+	return rte_pmu->name ? 0 : -ENODEV;
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu->name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = rte_calloc(NULL, 1, sizeof(*event), 0);
+	if (!event)
+		return -ENOMEM;
+
+	event->name = strdup(name);
+	if (!event->name) {
+		rte_free(event);
+
+		return -ENOMEM;
+	}
+
+	event->index = rte_pmu->num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu->event_list, event, next);
+
+	RTE_LOG(DEBUG, EAL, "%s even added at index %d\n", name, event->index);
+
+	return event->index;
+}
+
+void
+eal_pmu_init(void)
+{
+	int ret;
+
+	rte_pmu = rte_calloc(NULL, 1, sizeof(*rte_pmu), RTE_CACHE_LINE_SIZE);
+	if (!rte_pmu) {
+		RTE_LOG(ERR, EAL, "failed to alloc PMU\n");
+
+		return;
+	}
+
+	TAILQ_INIT(&rte_pmu->event_list);
+
+	ret = scan_pmus();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to find core pmu\n");
+		goto out;
+	}
+
+	ret = pmu_arch_init();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to setup arch for PMU\n");
+		goto out;
+	}
+
+	return;
+out:
+	free(rte_pmu->name);
+	rte_free(rte_pmu);
+}
+
+void
+eal_pmu_fini(void)
+{
+	struct rte_pmu_event *event, *tmp;
+	int lcore_id;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu->event_list, next, tmp) {
+		TAILQ_REMOVE(&rte_pmu->event_list, event, next);
+		free(event->name);

lib/eal/common/rte_pmu.c line 457

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+		rte_free(event);
+	}
+
+	RTE_LCORE_FOREACH_WORKER(lcore_id)
+		cleanup_events(lcore_id);
+
+	pmu_arch_fini();
+	free(rte_pmu->name);
+	rte_free(rte_pmu);
+}

There may be some problems with the implementation of eal_pmu_fini(), but I'm not sure.

I checked some test reports for this series. It seems that the test case of debug_autotest in

the DPDK unit test has an issue when the child process in this test case calls the function of  rte_exit().

The call chain is as follows:

    test_debug() -> test_exit() -> test_exit_val() -> rte_exit() -> rte_eal_cleanup() -> eal_pmu_fini().

The issue may be related to memory free from the error message as follows:

test_exit_valEAL: Error: Invalid memory
EAL: Error - exiting with code: 1
  Cause: test_exit_valEAL: Error: Invalid memory
EAL: Error - exiting with code: 2
  Cause: test_exit_valEAL: Error: Invalid memory
EAL: Error - exiting with code: 255
  Cause: test_exit_valEAL: Error: Invalid memory
EAL: Error - exiting with code: -1
  Cause: test_exit_valEAL: Error: Invalid memory

The above error message will disappear when I comment out the calling to the eal_pmu_fini() in

the rte_eal_cleanup().

Quoted 268 lines
lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
          'rte_pci_dev_features.h',
          'rte_per_lcore.h',
          'rte_pflock.h',
+        'rte_pmu.h',
          'rte_random.h',
          'rte_reciprocal.h',
          'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +204 -0
new file mode 100644
index 0000000000..e4b4f6b052
@@ -0,0 +1,204 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef RTE_EXEC_ENV_LINUX
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_lcore.h>
+#include <rte_log.h>
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped perf_event_attr structures */
+	bool enabled; /**< true if group was enabled on particular lcore */
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */
+};
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under /sys/bus/event_source/devices */
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore event group data */
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+};
+
+/** Pointer to the PMU state container */
+extern struct rte_pmu *rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version */
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return pmc + offset;
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get perf_event_mmap_page lock\n");
+			break;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}
+
+	if (index < 0 || index >= rte_pmu->num_group_events)
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page *)group->mmap_pages[index]);
+}
+
+#else /* !RTE_EXEC_ENV_LINUX */
+
+__rte_experimental
+static int __rte_unused
+rte_pmu_add_event(__rte_unused const char *name)
+{
+	return -1;
+}
+
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(__rte_unused int index)
+{
+	return 0;
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/eal/linux/eal.c +4 -0
index 8c118d0d9f..751a13b597 100644
@@ -53,6 +53,7 @@
  #include "eal_options.h"
  #include "eal_vfio.h"
  #include "hotplug_mp.h"
+#include "pmu_private.h"
 
  #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
  		return -1;
  	}
 
+	eal_pmu_init();
+
  	if (rte_eal_tailqs_init() < 0) {
  		rte_eal_init_alert("Cannot init tail queues for objects");
  		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
  	eal_bus_cleanup();
  	rte_trace_save();
  	eal_trace_fini();
+	eal_pmu_fini();
  	/* after this point, any DPDK pointers will become dangling */
  	rte_eal_memory_detach();
  	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map +5 -0
index 7ad12a7dc9..9225f46f67 100644
@@ -440,6 +440,11 @@ EXPERIMENTAL {
  	rte_thread_detach;
  	rte_thread_equal;
  	rte_thread_join;
+
+	# added in 23.03
+	rte_pmu; # WINDOWS_NO_EXPORT
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
  };
 
  INTERNAL {

lib/eal/version.map INTERNAL { · line 492

Quoted 5 lines
lib/eal/version.map +1 -0
@@ -483,4 +488,5 @@ INTERNAL {
  	rte_mem_map;
  	rte_mem_page_size;
  	rte_mem_unmap;
+	rte_pmu_enable_group;
  };

Best regards,

Min Zhou

#14 lore raw

RE: [EXT] Re: [PATCH v3 1/4] eal: add generic support for reading PMU events

Hello Min,

Quoted 12 lines

-----Original Message-----
From: zhoumin <[email protected]>
Sent: Wednesday, November 30, 2022 9:33 AM
To: Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Subject: [EXT] Re: [PATCH v3 1/4] eal: add generic support for reading PMU events

External Email

----------------------------------------------------------------------
Hi Tomasz,

[...]

Quoted 44 lines
Quoted 9 lines
lib/eal/common/rte_pmu.c new file +9 -0
+void
+eal_pmu_fini(void)
+{
+	struct rte_pmu_event *event, *tmp;
+	int lcore_id;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu->event_list, next, tmp) {
+		TAILQ_REMOVE(&rte_pmu->event_list, event, next);
+		free(event->name);

lib/eal/common/rte_pmu.c line 457

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+		rte_free(event);
+	}
+
+	RTE_LCORE_FOREACH_WORKER(lcore_id)
+		cleanup_events(lcore_id);
+
+	pmu_arch_fini();
+	free(rte_pmu->name);
+	rte_free(rte_pmu);
+}

There may be some problems with the implementation of eal_pmu_fini(), but I'm not sure.

I checked some test reports for this series. It seems that the test case of debug_autotest in

the DPDK unit test has an issue when the child process in this test case calls the function of  rte_exit().

The call chain is as follows:

    test_debug() -> test_exit() -> test_exit_val() -> rte_exit() ->
rte_eal_cleanup() -> eal_pmu_fini().

The issue may be related to memory free from the error message as follows:

test_exit_valEAL: Error: Invalid memory
EAL: Error - exiting with code: 1
  Cause: test_exit_valEAL: Error: Invalid memory
EAL: Error - exiting with code: 2
  Cause: test_exit_valEAL: Error: Invalid memory
EAL: Error - exiting with code: 255
  Cause: test_exit_valEAL: Error: Invalid memory
EAL: Error - exiting with code: -1
  Cause: test_exit_valEAL: Error: Invalid memory

Quoted 4 lines

The above error message will disappear when I comment out the calling to the eal_pmu_fini() in

the rte_eal_cleanup().

Thanks for pointing this out. This was apparently happening due to freeing same hugepage memory in forked process multiple times.

#15 lore raw

PATCH v3 2/4 eal/arm: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c               |   4 ++
 lib/eal/arm/include/meson.build   |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h |  39 +++++++++++
 lib/eal/arm/meson.build           |   4 ++
 lib/eal/arm/rte_pmu.c             | 104 ++++++++++++++++++++++++++++++
 lib/eal/include/rte_pmu.h         |   3 +
 6 files changed, 155 insertions(+)
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
app/test/test_pmu.c +4 -0
index fd331af9ee..f94866dff9 100644
@@ -13,6 +13,10 @@ test_pmu_read(void)
 	int tries = 10;
 	int event = -1;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/eal/arm/include/meson.build +1 -0
index 657bf58569..ab13b0220a 100644
@@ -20,6 +20,7 @@ arch_headers = files(
         'rte_pause_32.h',
         'rte_pause_64.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch_32.h',
         'rte_prefetch_64.h',
lib/eal/arm/include/rte_pmu_pmc.h new file +39 -0
new file mode 100644
index 0000000000..10e2984813
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_ARM_H_
+#define _RTE_PMU_PMC_ARM_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_ARM_H_ */
lib/eal/arm/meson.build +4 -0
index dca1106aae..0c5575b197 100644
@@ -9,3 +9,7 @@ sources += files(
         'rte_hypervisor.c',
         'rte_power_intrinsics.c',
 )
+
+if is_linux
+    sources += files('rte_pmu.c')
+endif
lib/eal/arm/rte_pmu.c new file +104 -0

104 lines not on this page · raw message

lib/eal/include/rte_pmu.h +3 -0
index e4b4f6b052..158a616b83 100644
@@ -20,6 +20,9 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
+#if defined(RTE_ARCH_ARM64)
+#include <rte_pmu_pmc.h>
+#endif
 
 /**
  * @file
Signature · 1 line
2.25.1
#16 lore raw

PATCH v3 3/4 eal/x86: support reading Intel PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel PMU events in runtime.

---
 app/test/test_pmu.c               |  2 ++
 lib/eal/include/rte_pmu.h         |  2 +-
 lib/eal/x86/include/meson.build   |  1 +
 lib/eal/x86/include/rte_pmu_pmc.h | 33 +++++++++++++++++++++++++++++++
 4 files changed, 37 insertions(+), 1 deletion(-)
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h
app/test/test_pmu.c +2 -0
index f94866dff9..016204c083 100644
@@ -15,6 +15,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/eal/include/rte_pmu.h +1 -1
index 158a616b83..3d90f4baf7 100644
@@ -20,7 +20,7 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
-#if defined(RTE_ARCH_ARM64)
+#if defined(RTE_ARCH_ARM64) || defined(RTE_ARCH_X86_64)
 #include <rte_pmu_pmc.h>
 #endif
 
lib/eal/x86/include/meson.build +1 -0
index 52d2f8e969..03d286ed25 100644
@@ -9,6 +9,7 @@ arch_headers = files(
         'rte_io.h',
         'rte_memcpy.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch.h',
         'rte_rtm.h',
lib/eal/x86/include/rte_pmu_pmc.h new file +33 -0
new file mode 100644
index 0000000000..a2cd849fb1
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_X86_H_
+#define _RTE_PMU_PMC_X86_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint32_t high, low;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return ((uint64_t)high << 32) | (uint64_t)low;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_X86_H_ */
Signature · 1 line
2.25.1
#17 lore raw

PATCH v3 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               |  4 ++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 ++++++++++++
 lib/eal/common/eal_common_trace_points.c |  3 ++
 lib/eal/common/rte_pmu.c                 | 63 ++++++++++++++++++++++++
 lib/eal/include/rte_eal_trace.h          | 11 +++++
 lib/eal/version.map                      |  1 +
 7 files changed, 119 insertions(+)
app/test/test_trace_perf.c +4 -0
index 46ae7d8074..4851b6852f 100644
@@ -114,6 +114,8 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +124,7 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+WORKER_DEFINE(READ_PMU)
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +177,7 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index 8fc1b20cab..977800ea01 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..9a845fd86f 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace_points.c +3 -0
index 0b0b254615..de918ca618 100644
@@ -75,3 +75,6 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
lib/eal/common/rte_pmu.c +63 -0

86 lines not on this page · raw message

lib/eal/include/rte_eal_trace.h +11 -0
index 5ef4398230..2a10f63e97 100644
@@ -17,6 +17,7 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#include <rte_pmu.h>
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +280,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+/* PMU */
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(int index),
+	uint64_t val;
+	rte_trace_point_emit_int(index);
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/version.map +1 -0
index 9225f46f67..73803f9601 100644
@@ -442,6 +442,7 @@ EXPERIMENTAL {
 	rte_thread_join;
 
 	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 	rte_pmu; # WINDOWS_NO_EXPORT
 	rte_pmu_add_event; # WINDOWS_NO_EXPORT
 	rte_pmu_read; # WINDOWS_NO_EXPORT
Signature · 1 line
2.25.1
#18 lore raw
In reply to Tomasz Duszynski
Quoted 16 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Tuesday, 29 November 2022 10.28

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on
isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Quoted 7 lines

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

Hi Thomasz,

I am very interested in this patch series for fast path profiling purposes. (Not using EAL trace, but our proprietary profiler.)

However, it seems that rte_pmu_read() is quite longwinded, compared to rte_pmu_pmc_read().

But perhaps I am just worrying too much, so I will ask: What is the performance cost of using rte_pmu_read() - compared to rte_pmu_pmc_read() - in the fast path?

If there is a non-negligible difference, could you please provide an example of how to configure PMU events and use rte_pmu_pmc_read() in an application?

I would primarily be interested in data cache misses and branch mispredictions. But feel free to make your own choices for the example.

#19 lore raw

Hi Morten,

Quoted 33 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Tuesday, November 29, 2022 11:43 AM
To: Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Subject: [EXT] RE: [PATCH v3 0/4] add support for self monitoring

External Email

----------------------------------------------------------------------

Quoted 15 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Tuesday, 29 November 2022 10.28

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Quoted 7 lines

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

Quoted 7 lines

Hi Thomasz,

I am very interested in this patch series for fast path profiling purposes. (Not using EAL trace, but our proprietary profiler.)

However, it seems that rte_pmu_read() is quite longwinded, compared to rte_pmu_pmc_read().

We need some bit of extra logic to set thigs up before performing reading actual counter but in reality cycles are mostly consumed by rte_pmu_pmc_read(). This obviously differs among platforms so if you want precise measurements you need to get your hands dirty.

That said, below are results coming from dpdk-test after running trace_perf_autotest - just to give you some idea.

X86-64

RTE>>trace_perf_autotest
Timer running at 3000.00MHz

            void: cycles=17.739375 ns=5.913125
             u64: cycles=17.348296 ns=5.782765
             int: cycles=17.098724 ns=5.699575
           float: cycles=17.099946 ns=5.699982
          double: cycles=17.229702 ns=5.743234
          string: cycles=31.159907 ns=10.386636
         void_fp: cycles=0.679842 ns=0.226614
        read_pmu: cycles=49.325117 ns=16.441706

ARM64 with RTE_ARM_EAL_RDTSC_USE_PMU

RTE>>trace_perf_autotest
Timer running at 2480.00MHz

            void: cycles=9.413568 ns=3.795793
             u64: cycles=9.386003 ns=3.784678
             int: cycles=9.438701 ns=3.805928
           float: cycles=9.359377 ns=3.773942
          double: cycles=9.372279 ns=3.779145
          string: cycles=24.474899 ns=9.868911
         void_fp: cycles=0.505513 ns=0.203836
        read_pmu: cycles=17.442853 ns=7.033409
Quoted 6 lines

But perhaps I am just worrying too much, so I will ask: What is the performance cost of using rte_pmu_read() - compared to rte_pmu_pmc_read() - in the fast path?

If there is a non-negligible difference, could you please provide an example of how to configure PMU events and use rte_pmu_pmc_read() in an application?

Series come with some docs so you can check there how to run it.

Quoted 2 lines

I would primarily be interested in data cache misses and branch mispredictions. But feel free to make your own choices for the example.

Raw events are not supported right now which means you don't have fine control over all events. You can use only events from CPU PMU (/sys/bus/event_source/devices/<PMU>/events).

#20 lore raw

v4 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 ++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 104 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 519 +++++++++++++++++++++++
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  11 +
 lib/eal/include/rte_pmu.h                | 207 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   7 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1075 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.25.1

#21 lore raw

PATCH v4 1/4 eal: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 app/test/meson.build                  |   1 +
 app/test/test_pmu.c                   |  41 +++
 doc/guides/prog_guide/profile_app.rst |   8 +
 lib/eal/common/meson.build            |   3 +
 lib/eal/common/pmu_private.h          |  41 +++
 lib/eal/common/rte_pmu.c              | 456 ++++++++++++++++++++++++++
 lib/eal/include/meson.build           |   1 +
 lib/eal/include/rte_pmu.h             | 204 ++++++++++++
 lib/eal/linux/eal.c                   |   4 +
 lib/eal/version.map                   |   6 +
 10 files changed, 765 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
app/test/meson.build +1 -0
index f34d19e3c3..93b3300309 100644
@@ -143,6 +143,7 @@ test_sources = files(
         'test_timer_racecond.c',
         'test_timer_secondary.c',
         'test_ticketlock.c',
+        'test_pmu.c',
         'test_trace.c',
         'test_trace_register.c',
         'test_trace_perf.c',
app/test/test_pmu.c new file +41 -0

41 lines not on this page · raw message

doc/guides/prog_guide/profile_app.rst +8 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
         'rte_service.c',
         'rte_version.c',
 )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
 if is_linux or is_windows
     sources += files('eal_common_dynmem.c')
 endif
lib/eal/common/pmu_private.h new file +41 -0

41 lines not on this page · raw message

lib/eal/common/rte_pmu.c new file +456 -0

456 lines not on this page · raw message

lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+        'rte_pmu.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +204 -0

204 lines not on this page · raw message

lib/eal/linux/eal.c +4 -0
index 8c118d0d9f..751a13b597 100644
@@ -53,6 +53,7 @@
 #include "eal_options.h"
 #include "eal_vfio.h"
 #include "hotplug_mp.h"
+#include "pmu_private.h"
 
 #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	eal_pmu_init();
+
 	if (rte_eal_tailqs_init() < 0) {
 		rte_eal_init_alert("Cannot init tail queues for objects");
 		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
 	eal_bus_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
+	eal_pmu_fini();
 	/* after this point, any DPDK pointers will become dangling */
 	rte_eal_memory_detach();
 	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map +6 -0
index 7ad12a7dc9..9225f46f67 100644
@@ -440,6 +440,11 @@ EXPERIMENTAL {
 	rte_thread_detach;
 	rte_thread_equal;
 	rte_thread_join;
+
+	# added in 23.03
+	rte_pmu; # WINDOWS_NO_EXPORT
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
 };
 
 INTERNAL {
@@ -483,4 +488,5 @@ INTERNAL {
 	rte_mem_map;
 	rte_mem_page_size;
 	rte_mem_unmap;
+	rte_pmu_enable_group;
 };
Signature · 1 line
2.25.1
#22 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Quoted 548 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Tuesday, 13 December 2022 11.44

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---

+++ b/lib/eal/common/rte_pmu.c

lib/eal/common/rte_pmu.c new file +26 -0
@@ -0,0 +1,456 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_eal_paging.h>
+#include <rte_pmu.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>

((64 - 1 - (h)))))

lib/eal/common/rte_pmu.c new file +4 -0
+#endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1))

& (m))

lib/eal/common/rte_pmu.c new file +6 -0
+#endif
+
+struct rte_pmu *rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op. Architectures

should override them if
+ * necessary.
+ */

lib/eal/common/rte_pmu.c new file +31 -0
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t config[3])
+{
+	RTE_SET_USED(config);
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask)
+{
+	char *config = NULL;
+	char path[PATH_MAX];
+	int high, low, ret;
+	FILE *fp;
+
+	/* quiesce -Wmaybe-uninitialized warning */
+	*num = 0;
+	*mask = 0;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH

"/%s/format/%s", rte_pmu->name, name);

lib/eal/common/rte_pmu.c new file +20 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is

implied. */

lib/eal/common/rte_pmu.c new file +52 -0
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3])
+{
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3])
+{
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH

"/%s/events/%s", rte_pmu->name, name);

lib/eal/common/rte_pmu.c new file +34 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int lcore_id, int group_fd)
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, rte_gettid(),

rte_lcore_to_cpu_id(lcore_id),

lib/eal/common/rte_pmu.c new file +18 -0
+		       group_fd, 0);
+}
+
+static int
+open_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to get %s event config\n",

event->name);

lib/eal/common/rte_pmu.c new file +7 -0
+			continue;
+		}
+
+		ret = do_perf_event_open(config, lcore_id, group->fds[0]);
+		if (ret == -1) {
+			if (errno == EOPNOTSUPP)
+				RTE_LOG(ERR, EAL, "64 bit counters not

supported\n");

lib/eal/common/rte_pmu.c new file +29 -0
+
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	void *addr;
+	int ret, i;
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		addr = mmap(0, rte_mem_page_size(), PROT_READ, MAP_SHARED,

group->fds[i], 0);

lib/eal/common/rte_pmu.c new file +29 -0
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], rte_mem_page_size());
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int i;
+
+	if (!group->fds)
+		return;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE,

PERF_IOC_FLAG_GROUP);

lib/eal/common/rte_pmu.c new file +34 -0
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], rte_mem_page_size());
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	free(group->mmap_pages);
+	free(group->fds);
+
+	group->mmap_pages = NULL;
+	group->fds = NULL;
+	group->enabled = false;
+}
+
+int __rte_noinline
+rte_pmu_enable_group(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int ret;
+
+	if (rte_pmu->num_group_events == 0) {
+		RTE_LOG(DEBUG, EAL, "no matching PMU events\n");
+
+		return 0;
+	}
+
+	group->fds = calloc(rte_pmu->num_group_events, sizeof(*group-
Quoted 1 line

fds));

lib/eal/common/rte_pmu.c new file +7 -0
+	if (!group->fds) {
+		RTE_LOG(ERR, EAL, "failed to alloc descriptor memory\n");
+
+		return -ENOMEM;
+	}
+
+	group->mmap_pages = calloc(rte_pmu->num_group_events,

sizeof(*group->mmap_pages));

lib/eal/common/rte_pmu.c new file +10 -0
+	if (!group->mmap_pages) {
+		RTE_LOG(ERR, EAL, "failed to alloc userpage memory\n");
+
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ret = open_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to open events on lcore-worker-

%d\n", lcore_id);

lib/eal/common/rte_pmu.c new file +6 -0
+		goto out;
+	}
+
+	ret = mmap_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to map events on lcore-worker-

%d\n", lcore_id);

lib/eal/common/rte_pmu.c new file +4 -0
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE,

PERF_IOC_FLAG_GROUP) == -1) {

(quoted diff) +1 -0
+		RTE_LOG(ERR, EAL, "failed to enable events on lcore-worker-

%d\n", lcore_id);

lib/eal/common/rte_pmu.c new file +35 -0
+
+		ret = -errno;
+		goto out;
+	}
+
+	return 0;
+
+out:
+	cleanup_events(lcore_id);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (!dirp)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH

"/%s/cpus", name);

lib/eal/common/rte_pmu.c new file +22 -0
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	closedir(dirp);
+
+	if (dent) {
+		rte_pmu->name = strdup(name);
+		if (!rte_pmu->name)
+			return -ENOMEM;
+	}
+
+	return rte_pmu->name ? 0 : -ENODEV;
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH

"/%s/events/%s", rte_pmu->name, name);

lib/eal/common/rte_pmu.c new file +24 -0
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = calloc(1, sizeof(*event));
+	if (!event)
+		return -ENOMEM;
+
+	event->name = strdup(name);
+	if (!event->name) {
+		free(event);
+
+		return -ENOMEM;
+	}
+
+	event->index = rte_pmu->num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu->event_list, event, next);
+
+	RTE_LOG(DEBUG, EAL, "%s even added at index %d\n", name, event-
Quoted 1 line

index);

lib/eal/common/rte_pmu.c new file +55 -0
+
+	return event->index;
+}
+
+void
+eal_pmu_init(void)
+{
+	int ret;
+
+	rte_pmu = calloc(1, sizeof(*rte_pmu));
+	if (!rte_pmu) {
+		RTE_LOG(ERR, EAL, "failed to alloc PMU\n");
+
+		return;
+	}
+
+	TAILQ_INIT(&rte_pmu->event_list);
+
+	ret = scan_pmus();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to find core pmu\n");
+		goto out;
+	}
+
+	ret = pmu_arch_init();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to setup arch for PMU\n");
+		goto out;
+	}
+
+	return;
+out:
+	free(rte_pmu->name);
+	free(rte_pmu);
+}
+
+void
+eal_pmu_fini(void)
+{
+	struct rte_pmu_event *event, *tmp;
+	int lcore_id;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu->event_list, next, tmp) {
+		TAILQ_REMOVE(&rte_pmu->event_list, event, next);
+		free(event->name);
+		free(event);
+	}
+
+	RTE_LCORE_FOREACH_WORKER(lcore_id)
+		cleanup_events(lcore_id);
+
+	pmu_arch_fini();
+	free(rte_pmu->name);
+	free(rte_pmu);
+}
lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+        'rte_pmu.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +38 -0
new file mode 100644
index 0000000000..e4b4f6b052
@@ -0,0 +1,204 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef RTE_EXEC_ENV_LINUX
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_lcore.h>
+#include <rte_log.h>
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped
Quoted 1 line

perf_event_attr structures */

There seems to be a lot of indirection involved here. Why are these arrays not statically sized, instead of dynamically allocated?

Also, what is the reason for hiding the type struct perf_event_mmap_page **mmap_pages opaque by using void **mmap_pages instead?

Quoted 24 lines
lib/eal/include/rte_pmu.h new file +1 -0
+	bool enabled; /**< true if group was enabled on particular lcore

*/

lib/eal/include/rte_pmu.h new file +16 -0
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */
+};
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

lib/eal/include/rte_pmu.h new file +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore

event group data */

lib/eal/include/rte_pmu.h new file +2 -0
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching

events */

lib/eal/include/rte_pmu.h line 62

Quoted 4 lines
lib/eal/include/rte_pmu.h new file +4 -0
+};
+
+/** Pointer to the PMU state container */
+extern struct rte_pmu *rte_pmu;

Again, why not just extern struct rte_pmu, instead of dynamic allocation?

Quoted 106 lines
lib/eal/include/rte_pmu.h new file +2 -0
+
+/** Each architecture supporting PMU needs to provide its own version

*/

lib/eal/include/rte_pmu.h new file +42 -0
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return pmc + offset;
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get

perf_event_mmap_page lock\n");

lib/eal/include/rte_pmu.h new file +29 -0
+			break;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under

/sys/bus/event_source/devices/pmu/events.
+ * @return
+ * Event index in case of success, negative value otherwise.
+ */

lib/eal/include/rte_pmu.h new file +14 -0
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of

support
+ * 0 is returned. In other words, stream of zeros in a trace file
+ * indicates problem with reading particular PMU event register.
+ */

lib/eal/include/rte_pmu.h new file +9 -0
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)

lib/eal/include/rte_pmu.h line 174

Quoted 10 lines
lib/eal/include/rte_pmu.h new file +10 -0
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}

Why is the group not enabled in the setup function, rte_pmu_add_event(), instead of here, in the hot path?

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +5 -0
+
+	if (index < 0 || index >= rte_pmu->num_group_events)
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page

*)group->mmap_pages[index]);

Using fixed size arrays instead of multiple indirections via pointers is faster. It could be:

return rte_pmu_read_userpage((struct perf_event_mmap_page *)rte_pmu.group[lcore_id].mmap_pages[index]);

With our without suggested performance improvements...

#23 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Hello Morten,

Thanks for review. Answers inline.

[...]

Quoted 11 lines
Quoted 11 lines
lib/eal/include/rte_pmu.h new file +11 -0
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+/**
+ * A structure describing a group of events.
Quoted 9 lines
Quoted 5 lines

+ */

lib/eal/include/rte_pmu.h new file +3 -0
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped

perf_event_attr structures */

There seems to be a lot of indirection involved here. Why are these arrays not statically sized, instead of dynamically allocated?

Different architectures/pmus impose limits on number of simultaneously enabled counters. So in order relief the pain of thinking about it and adding macros for each and every arch I decided to allocate the number user wants dynamically. Also assumption holds that user knows about tradeoffs of using too many counters hence will not enable too many events at once.

Quoted 2 lines

Also, what is the reason for hiding the type struct perf_event_mmap_page **mmap_pages opaque by using void **mmap_pages instead?

I think, that part doing mmap/munmap was written first hence void ** was chosen in the first place.

Quoted 22 lines
Quoted 22 lines
(quoted diff) +1 -0
+	bool enabled; /**< true if group was enabled on particular lcore

*/

lib/eal/include/rte_pmu.h new file +15 -0
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

(quoted diff) +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore

event group data */

lib/eal/include/rte_pmu.h new file +1 -0
+	int num_group_events; /**< number of events in a group */
Quoted 9 lines

app/test/test_pmu.c

Quoted 6 lines
(quoted diff) +1 -0
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching

events */

app/test/test_pmu.c new file +4 -0
+};
+
+/** Pointer to the PMU state container */ extern struct rte_pmu
+*rte_pmu;

Again, why not just extern struct rte_pmu, instead of dynamic allocation?

No strong opinions here since this is a matter of personal preference. Can be removed in the next version.

Quoted 109 lines
Quoted 109 lines
(quoted diff) +2 -0
+
+/** Each architecture supporting PMU needs to provide its own version

*/

lib/eal/include/rte_pmu.h new file +40 -0
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return pmc + offset;
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get

perf_event_mmap_page lock\n");

lib/eal/include/rte_pmu.h new file +29 -0
+			break;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under

/sys/bus/event_source/devices/pmu/events.
+ * @return
+ * Event index in case of success, negative value otherwise.
+ */

lib/eal/include/rte_pmu.h new file +14 -0
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of

support
+ * 0 is returned. In other words, stream of zeros in a trace file
+ * indicates problem with reading particular PMU event register.
+ */

lib/eal/include/rte_pmu.h new file +14 -0
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {
+		ret = rte_pmu_enable_group(lcore_id);
Quoted 9 lines

lib/eal/include/rte_pmu.h line 174

Quoted 5 lines
lib/eal/include/rte_pmu.h new file +5 -0
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}

Why is the group not enabled in the setup function, rte_pmu_add_event(), instead of here, in the hot path?

When this is executed for the very first time then cpu will have obviously more work to do but afterwards setup path is not taken hence much less cpu cycles are required.

Setup is executed by main lcore solely, before lcores are executed hence some info passed to SYS_perf_event_open ioctl() is missing, pid (via rte_gettid()) being an example here.

Quoted 15 lines
Quoted 6 lines
lib/eal/include/rte_pmu.h new file +5 -0
+
+	if (index < 0 || index >= rte_pmu->num_group_events)
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page

*)group->mmap_pages[index]);

Using fixed size arrays instead of multiple indirections via pointers is faster. It could be:

return rte_pmu_read_userpage((struct perf_event_mmap_page *)rte_pmu.group[lcore_id].mmap_pages[index]);

With our without suggested performance improvements...

#24 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

+CC: Mattias, see my comment below about per-thread constructor for this

Quoted 32 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 10.39

Hello Morten,

Thanks for review. Answers inline.

[...]

Quoted 6 lines
Quoted 6 lines

+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU

and

Quoted 4 lines
Quoted 4 lines

+ * read selected counters in runtime.
+ */

(quoted diff) +2 -0
+
+/**
Quoted 8 lines
Quoted 6 lines

+ * A structure describing a group of events.
+ */

lib/eal/include/rte_pmu.h new file +3 -0
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped

perf_event_attr structures */

There seems to be a lot of indirection involved here. Why are these

arrays not statically sized,

Quoted 2 lines

instead of dynamically allocated?

Quoted 7 lines

Different architectures/pmus impose limits on number of simultaneously enabled counters. So in order
relief the pain of thinking about it and adding macros for each and every arch I decided to allocate
the number user wants dynamically. Also assumption holds that user knows about tradeoffs of using
too many counters hence will not enable too many events at once.

The DPDK convention is to use fixed size arrays (with a maximum size, e.g. RTE_MAX_ETHPORTS) in the fast path, for performance reasons.

Please use fixed size arrays instead of dynamically allocated arrays.

Quoted 7 lines
Quoted 2 lines

Also, what is the reason for hiding the type struct

perf_event_mmap_page **mmap_pages opaque by

Quoted 1 line

using void **mmap_pages instead?

I think, that part doing mmap/munmap was written first hence void ** was chosen in the first place.

Please update it, so the actual type is reflected here.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
(quoted diff) +1 -0
+	bool enabled; /**< true if group was enabled on particular lcore

*/

lib/eal/include/rte_pmu.h new file +13 -0
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */ };
+
+/**
+ * A PMU state container.
+ */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(quoted diff) +2 -0
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

(quoted diff) +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore

event group data */

(quoted diff) +2 -0
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching

events */

The event_list is used in slow path only, so it can remain a list - i.e. no change requested here. :-)

Quoted 6 lines
Quoted 6 lines

app/test/test_pmu.c

Quoted 4 lines
app/test/test_pmu.c new file +4 -0
+};
+
+/** Pointer to the PMU state container */ extern struct rte_pmu
+*rte_pmu;

Again, why not just extern struct rte_pmu, instead of dynamic

Quoted 6 lines

allocation?

Quoted 1 line

No strong opinions here since this is a matter of personal preference. Can be removed
in the next version.

Yes, please.

Quoted 18 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+
+/** Each architecture supporting PMU needs to provide its own

version

Quoted 14 lines
Quoted 14 lines

*/

lib/eal/include/rte_pmu.h new file +13 -0
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
Quoted 8 lines
Quoted 8 lines

lib/eal/include/rte_pmu.h line 87

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +8 -0
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {

As a matter of personal preference, I would write this loop differently:

+ for (tries = 100; tries != 0; tries--) {

Quoted 7 lines
Quoted 7 lines

lib/eal/include/rte_pmu.h line 94

Quoted 7 lines
lib/eal/include/rte_pmu.h new file +7 -0
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {

Why "&& index"? The way I read [man perf_event_open], index 0 is perfectly valid.

[man perf_event_open]: https://man7.org/linux/man-pages/man2/perf_event_open.2.html

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
lib/eal/include/rte_pmu.h new file +7 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +5 -0
+		if (likely(pc->lock == seq))
+			return pmc + offset;
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get

perf_event_mmap_page lock\n");

(quoted diff) +2 -0
+			break;
+		}

- Remove the 4 above lines of code, and move the debug log message to the end of the function instead.

Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+	}

+ RTE_LOG(DEBUG, EAL, "failed to get perf_event_mmap_page lock\n");

Quoted 47 lines
Quoted 31 lines
Quoted 31 lines
lib/eal/include/rte_pmu.h new file +26 -0
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under

/sys/bus/event_source/devices/pmu/events.
+ * @return
+ * Event index in case of success, negative value otherwise.
+ */
+__rte_experimental

Quoted 8 lines

lib/eal/include/rte_pmu.h

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +8 -0
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an

event.

Quoted 7 lines
Quoted 7 lines

+ *
+ * @param index
+ * Index of an event to be read.
+ * @return
+ * Event value read from register. In case of errors or lack of
support
+ * 0 is returned. In other words, stream of zeros in a trace

Quoted 6 lines

file

Quoted 5 lines

lib/eal/include/rte_pmu.h line 158

Quoted 5 lines

+ * indicates problem with reading particular PMU event register.
+ */

lib/eal/include/rte_pmu.h new file +3 -0
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)

The index type can be changed from int to uint32_t. This also eliminates the "(index < 0" part of the comparison further below in this function.

Quoted 10 lines
Quoted 10 lines

lib/eal/include/rte_pmu.h line 168

Quoted 10 lines
lib/eal/include/rte_pmu.h new file +10 -0
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {

Optimized: if (unlikely(!group->enabled)) {

Quoted 17 lines
Quoted 8 lines

lib/eal/include/rte_pmu.h line 174

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}

Why is the group not enabled in the setup function,

rte_pmu_add_event(), instead of here, in the

Quoted 2 lines

hot path?

When this is executed for the very first time then cpu will have obviously more work to do
but afterwards setup path is not taken hence much less cpu cycles are required.

Quoted 4 lines

Setup is executed by main lcore solely, before lcores are executed hence some info passed to
SYS_perf_event_open ioctl() is missing, pid (via rte_gettid()) being an example here.

OK. Thank you for the explanation. Since impossible at setup, it has to be done at runtime.

@Mattias: Another good example of something that would belong in per-thread constructors, as my suggested feature creep in [1].

[1]: http://inbox.dpdk.org/dev/[email protected]/

Quoted 3 lines
Quoted 3 lines

lib/eal/include/rte_pmu.h line 176

Quoted 3 lines
lib/eal/include/rte_pmu.h new file +2 -0
+
+	if (index < 0 || index >= rte_pmu->num_group_events)

Optimized: if (unlikely(index >= rte_pmu.num_group_events))

Quoted 15 lines
Quoted 6 lines
Quoted 4 lines
lib/eal/common/rte_pmu.c new file +3 -0
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page

*)group->mmap_pages[index]);

Using fixed size arrays instead of multiple indirections via pointers

is faster. It could be:

Quoted 7 lines

return rte_pmu_read_userpage((struct perf_event_mmap_page *)rte_pmu.group[lcore_id].mmap_pages[index]);

With our without suggested performance improvements...

#25 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Quoted 62 lines

From: Morten Brørup [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 11.41

+CC: Mattias, see my comment below about per-thread constructor for this

Quoted 5 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 10.39

Hello Morten,

Quoted 7 lines

Thanks for review. Answers inline.

[...]

Quoted 3 lines

lib/eal/include/rte_pmu.h line 158

Quoted 3 lines
lib/eal/include/rte_pmu.h new file +3 -0
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)

The index type can be changed from int to uint32_t. This also eliminates the "(index < 0" part of the comparison further below in this function.

Quoted 10 lines
Quoted 10 lines

lib/eal/include/rte_pmu.h line 168

Quoted 10 lines
lib/eal/include/rte_pmu.h new file +10 -0
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {

Optimized: if (unlikely(!group->enabled)) {

Quoted 12 lines
Quoted 8 lines

lib/eal/include/rte_pmu.h line 174

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}

Why is the group not enabled in the setup function,

rte_pmu_add_event(), instead of here, in the

Quoted 2 lines

hot path?

Quoted 8 lines

When this is executed for the very first time then cpu will have obviously more work to do
but afterwards setup path is not taken hence much less cpu cycles are required.

Setup is executed by main lcore solely, before lcores are executed hence some info passed to
SYS_perf_event_open ioctl() is missing, pid (via rte_gettid()) being

an

Quoted 1 line

example here.

OK. Thank you for the explanation. Since impossible at setup, it has to be done at runtime.

Quoted 6 lines

@Mattias: Another good example of something that would belong in per- thread constructors, as my suggested feature creep in [1].

[1]:
http://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35D87553@smarts
erver.smartshare.dk/

I just realized that this initialization is per-lcore (not per thread), so you can use rte_lcore_callback_register() to register a per-lcore initialization function, and move rte_pmu_enable_group(lcore_id) there.

-Morten

#26 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Quoted 71 lines

From: Morten Brørup [mailto:[email protected]]
Sent: Thursday, 15 December 2022 09.22

Quoted 23 lines

From: Morten Brørup [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 11.41

+CC: Mattias, see my comment below about per-thread constructor for this

Quoted 5 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 10.39

Hello Morten,

Quoted 7 lines

Thanks for review. Answers inline.

[...]

Quoted 3 lines

lib/eal/include/rte_pmu.h line 158

Quoted 3 lines
lib/eal/include/rte_pmu.h new file +3 -0
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)

The index type can be changed from int to uint32_t. This also eliminates the "(index < 0" part of the comparison further below in this function.

Quoted 28 lines
Quoted 10 lines
Quoted 10 lines

lib/eal/include/rte_pmu.h line 168

Quoted 10 lines
lib/eal/include/rte_pmu.h new file +10 -0
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {

Optimized: if (unlikely(!group->enabled)) {

Quoted 15 lines
Quoted 8 lines

lib/eal/include/rte_pmu.h line 174

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}

Why is the group not enabled in the setup function,

rte_pmu_add_event(), instead of here, in the

Quoted 2 lines

hot path?

When this is executed for the very first time then cpu will have obviously more work to do
but afterwards setup path is not taken hence much less cpu cycles

are

Quoted 5 lines
Quoted 5 lines

required.

Setup is executed by main lcore solely, before lcores are executed hence some info passed to
SYS_perf_event_open ioctl() is missing, pid (via rte_gettid())

being

Quoted 4 lines

an

Quoted 1 line

example here.

OK. Thank you for the explanation. Since impossible at setup, it has

to

Quoted 5 lines

be done at runtime.

@Mattias: Another good example of something that would belong in per- thread constructors, as my suggested feature creep in [1].

Quoted 8 lines
Quoted 2 lines

[1]:

http://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35D87553@smarts

Quoted 1 line

erver.smartshare.dk/

I just realized that this initialization is per-lcore (not per thread), so you can use rte_lcore_callback_register() to register a per-lcore initialization function, and move rte_pmu_enable_group(lcore_id) there.

Sorry, Thomasz!

You can't use rte_lcore_callback_register()... it doesn't provide per-lcore thread constructors/destructors the way I thought. :-(

#27 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

In reply to Morten Brørup

Hi Morten,

A few comments inline.

Quoted 52 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Wednesday, December 14, 2022 11:41 AM
To: Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]
Subject: [EXT] RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

External Email

----------------------------------------------------------------------
+CC: Mattias, see my comment below about per-thread constructor for this

Quoted 32 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 10.39

Hello Morten,

Thanks for review. Answers inline.

[...]

Quoted 6 lines
Quoted 6 lines

+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU

and

Quoted 4 lines
Quoted 4 lines

+ * read selected counters in runtime.
+ */

(quoted diff) +2 -0
+
+/**
Quoted 8 lines
Quoted 6 lines

+ * A structure describing a group of events.
+ */

lib/eal/include/rte_pmu.h new file +3 -0
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped

perf_event_attr structures */

There seems to be a lot of indirection involved here. Why are these

arrays not statically sized,

Quoted 2 lines

instead of dynamically allocated?

Quoted 6 lines

Different architectures/pmus impose limits on number of simultaneously enabled counters. So in order relief the pain of thinking about it and adding macros for each and every arch I decided to allocate the number user wants dynamically. Also assumption holds that user knows about tradeoffs of using too many counters hence will not enable too many events at once.

Quoted 5 lines

The DPDK convention is to use fixed size arrays (with a maximum size, e.g. RTE_MAX_ETHPORTS) in the fast path, for performance reasons.

Please use fixed size arrays instead of dynamically allocated arrays.

I do agree that from maintenance angle fixed arrays are much more convenient but when optimization kicks in then that statement does not necessarily hold true anymore.

For example, in this case performance dropped by ~0.3% which is insignificant imo. So given simpler code, next patchset will use fixed arrays.

Quoted 85 lines
Quoted 7 lines
Quoted 2 lines

Also, what is the reason for hiding the type struct

perf_event_mmap_page **mmap_pages opaque by

Quoted 1 line

using void **mmap_pages instead?

I think, that part doing mmap/munmap was written first hence void ** was chosen in the first place.

Please update it, so the actual type is reflected here.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
(quoted diff) +1 -0
+	bool enabled; /**< true if group was enabled on particular lcore

*/

lib/eal/include/rte_pmu.h new file +13 -0
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */ };
+
+/**
+ * A PMU state container.
+ */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(quoted diff) +2 -0
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

(quoted diff) +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore

event group data */

(quoted diff) +2 -0
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching

events */

The event_list is used in slow path only, so it can remain a list - i.e. no change requested here. :-)

Quoted 6 lines
Quoted 6 lines

app/test/test_pmu.c

Quoted 4 lines
app/test/test_pmu.c new file +4 -0
+};
+
+/** Pointer to the PMU state container */ extern struct rte_pmu
+*rte_pmu;

Again, why not just extern struct rte_pmu, instead of dynamic

Quoted 6 lines

allocation?

Quoted 1 line

No strong opinions here since this is a matter of personal preference. Can be removed
in the next version.

Yes, please.

Quoted 18 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+
+/** Each architecture supporting PMU needs to provide its own

version

Quoted 14 lines
Quoted 14 lines

*/

lib/eal/include/rte_pmu.h new file +13 -0
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
Quoted 8 lines
Quoted 8 lines

lib/eal/include/rte_pmu.h line 87

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +8 -0
+__rte_internal
+static __rte_always_inline uint64_t rte_pmu_read_userpage(struct
+perf_event_mmap_page *pc) {
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {

As a matter of personal preference, I would write this loop differently:

+ for (tries = 100; tries != 0; tries--) {

Quoted 10 lines
Quoted 7 lines
Quoted 7 lines

lib/eal/include/rte_pmu.h line 94

Quoted 7 lines
lib/eal/include/rte_pmu.h new file +7 -0
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {

Why "&& index"? The way I read [man perf_event_open], index 0 is perfectly valid.

Valid index starts at 1. 0 means that either hw counter is stopped or isn't active. Maybe this is not initially clear from man but there's example later on how to get actual number.

Quoted 76 lines

[man perf_event_open]: https://urldefense.proofpoint.com/v2/url?u=https-3A__man7.org_linux_man-
2Dpages_man2_perf-5Fevent- 5Fopen.2.html&d=DwIFAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNXgrbjdlXxVEEGYkxIxRndyEUwWU_ad5ce22YI6Is&m=tny gBVwOnoZDV7hItku1HtmsI8R3F6vPJdr7ON3hE5iAds96T2C9JTNcnt6ptN4Q&s=s10yJogwRRXHqAuIay47H- aWl9SL5wpQ9tCjfiQUgrY&e=

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
lib/eal/include/rte_pmu.h new file +7 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +5 -0
+		if (likely(pc->lock == seq))
+			return pmc + offset;
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get

perf_event_mmap_page lock\n");

(quoted diff) +2 -0
+			break;
+		}

- Remove the 4 above lines of code, and move the debug log message to the end of the function instead.

Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+	}

+ RTE_LOG(DEBUG, EAL, "failed to get perf_event_mmap_page lock\n");

Quoted 47 lines
Quoted 31 lines
Quoted 31 lines
lib/eal/include/rte_pmu.h new file +26 -0
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under

/sys/bus/event_source/devices/pmu/events.
+ * @return
+ * Event index in case of success, negative value otherwise.
+ */
+__rte_experimental

Quoted 8 lines

lib/eal/include/rte_pmu.h

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +8 -0
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an

event.

Quoted 7 lines
Quoted 7 lines

+ *
+ * @param index
+ * Index of an event to be read.
+ * @return
+ * Event value read from register. In case of errors or lack of
support
+ * 0 is returned. In other words, stream of zeros in a trace

Quoted 9 lines
Quoted 5 lines

file

Quoted 4 lines

Review

Quoted 4 lines

+ * indicates problem with reading particular PMU event register.
+ */

(quoted diff) +2 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(int index)

The index type can be changed from int to uint32_t. This also eliminates the "(index < 0" part of the comparison further below in this function.

That's true.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/eal/include/rte_pmu.h new file +4 -0
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

lib/eal/include/rte_pmu.h line 168

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {

Optimized: if (unlikely(!group->enabled)) {

Compiler will optimize the branch itself correctly. Extra hint is not required.

Quoted 53 lines
Quoted 12 lines
Quoted 8 lines

lib/eal/include/rte_pmu.h line 174

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}

Why is the group not enabled in the setup function,

rte_pmu_add_event(), instead of here, in the

Quoted 2 lines

hot path?

Quoted 7 lines

When this is executed for the very first time then cpu will have obviously more work to do but afterwards setup path is not taken hence much less cpu cycles are required.

Setup is executed by main lcore solely, before lcores are executed hence some info passed to SYS_perf_event_open ioctl() is missing, pid (via rte_gettid()) being an example here.

OK. Thank you for the explanation. Since impossible at setup, it has to be done at runtime.

@Mattias: Another good example of something that would belong in per-thread constructors, as my suggested feature creep in [1].

[1]: https://urldefense.proofpoint.com/v2/url?u=http-
3A__inbox.dpdk.org_dev_98CBD80474FA8B44BF855DF32C47DC35D87553- 40smartserver.smartshare.dk_&d=DwIFAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNXgrbjdlXxVEEGYkxIxRndyEUwWU_ad5 ce22YI6Is&m=tnygBVwOnoZDV7hItku1HtmsI8R3F6vPJdr7ON3hE5iAds96T2C9JTNcnt6ptN4Q&s=aSAnYqgVnrgDp6yyMtGC uWgJjDlgqj1wHf1nGWyHCNo&e=

Quoted 3 lines
Quoted 3 lines

lib/eal/include/rte_pmu.h line 176

Quoted 3 lines
lib/eal/include/rte_pmu.h new file +2 -0
+
+	if (index < 0 || index >= rte_pmu->num_group_events)

Optimized: if (unlikely(index >= rte_pmu.num_group_events))

Quoted 16 lines
Quoted 7 lines
Quoted 4 lines
lib/eal/common/rte_pmu.c new file +3 -0
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page

*)group->mmap_pages[index]);

Using fixed size arrays instead of multiple indirections via pointers

is faster. It could be:

Quoted 7 lines

return rte_pmu_read_userpage((struct perf_event_mmap_page *)rte_pmu.group[lcore_id].mmap_pages[index]);

With our without suggested performance improvements...

#28 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Quoted 67 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Thursday, 5 January 2023 22.14

Hi Morten,

A few comments inline.

Quoted 7 lines

From: Morten Brørup <[email protected]>
Sent: Wednesday, December 14, 2022 11:41 AM

External Email

----------------------------------------------------------------------
+CC: Mattias, see my comment below about per-thread constructor for

this

Quoted 8 lines
Quoted 8 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 10.39

Hello Morten,

Thanks for review. Answers inline.

Quoted 8 lines
Quoted 8 lines

[...]

Quoted 6 lines
Quoted 6 lines

+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup

PMU

Quoted 13 lines
Quoted 13 lines

and

Quoted 12 lines
Quoted 10 lines

+ * read selected counters in runtime.
+ */

lib/eal/include/rte_pmu.h new file +7 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped

perf_event_attr structures */

There seems to be a lot of indirection involved here. Why are

these

Quoted 5 lines
Quoted 5 lines

arrays not statically sized,

Quoted 2 lines

instead of dynamically allocated?

Different architectures/pmus impose limits on number of

simultaneously

Quoted 1 line
Quoted 1 line

enabled counters. So in order relief the pain of thinking about it

and

Quoted 1 line
Quoted 1 line

adding macros for each and every arch I decided to allocate the

number

Quoted 5 lines
Quoted 3 lines

user wants dynamically. Also assumption holds that user knows about tradeoffs of using too many counters hence will not enable too many events at once.

The DPDK convention is to use fixed size arrays (with a maximum size,

e.g. RTE_MAX_ETHPORTS) in the

Quoted 4 lines

fast path, for performance reasons.

Please use fixed size arrays instead of dynamically allocated arrays.

Quoted 8 lines

I do agree that from maintenance angle fixed arrays are much more convenient
but when optimization kicks in then that statement does not necessarily hold true anymore.

For example, in this case performance dropped by ~0.3% which is insignificant imo. So
given simpler code, next patchset will use fixed arrays.

I fail to understand how pointer chasing can perform better than obtaining an address by multiplying by a constant. Modern CPUs work in mysterious ways, and you obviously tested this, so I believe your test results. But in theory, pointer chasing touches more cache lines, and should perform worse in a loaded system where pointers in the chain have been evicted from the cache.

Anyway, you agreed to use fixed arrays, so I am happy. :-)

Quoted 102 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Also, what is the reason for hiding the type struct

perf_event_mmap_page **mmap_pages opaque by

Quoted 1 line

using void **mmap_pages instead?

Quoted 8 lines
Quoted 2 lines

I think, that part doing mmap/munmap was written first hence void ** was chosen in the first place.

Please update it, so the actual type is reflected here.

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +1 -0
+	bool enabled; /**< true if group was enabled on particular

lcore

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

*/

lib/eal/include/rte_pmu.h new file +9 -0
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */ };
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +6 -0
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

(quoted diff) +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per

lcore

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines

event group data */

(quoted diff) +2 -0
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of

matching

Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

events */

The event_list is used in slow path only, so it can remain a list -

i.e. no change requested here.

Quoted 12 lines

:-)

Quoted 10 lines
Quoted 6 lines

app/test/test_pmu.c

Quoted 4 lines
app/test/test_pmu.c new file +4 -0
+};
+
+/** Pointer to the PMU state container */ extern struct rte_pmu
+*rte_pmu;

Again, why not just extern struct rte_pmu, instead of dynamic

allocation?

Quoted 1 line

No strong opinions here since this is a matter of personal

preference.

Quoted 9 lines
Quoted 2 lines

Can be removed
in the next version.

Yes, please.

Quoted 4 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+
+/** Each architecture supporting PMU needs to provide its own

version

Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

*/

lib/eal/include/rte_pmu.h new file +7 -0
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines

+ *
+ * @param pc
+ * Pointer to the mmapped user page.
+ * @return
+ * Counter value read from hardware.
+ */

(quoted diff) +2 -0
+__rte_internal
+static __rte_always_inline uint64_t

rte_pmu_read_userpage(struct

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

lib/eal/include/rte_pmu.h line 87

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+perf_event_mmap_page *pc) {
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {

As a matter of personal preference, I would write this loop

differently:

Quoted 12 lines

+ for (tries = 100; tries != 0; tries--) {

Quoted 7 lines
Quoted 7 lines

lib/eal/include/rte_pmu.h line 94

Quoted 7 lines
lib/eal/include/rte_pmu.h new file +7 -0
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {

Why "&& index"? The way I read [man perf_event_open], index 0 is

Quoted 7 lines

perfectly valid.

Quoted 1 line

Valid index starts at 1. 0 means that either hw counter is stopped or isn't active. Maybe this is not
initially clear from man but there's example later on how to get actual number.

OK. Thanks for the reference.

Please add a comment about the special meaning of index 0 in the code.

Quoted 111 lines
Quoted 2 lines

[man perf_event_open]:

https://urldefense.proofpoint.com/v2/url?u=https-
3A__man7.org_linux_man-

Quoted 2 lines

2Dpages_man2_perf-5Fevent- 5Fopen.2.html&d=DwIFAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNXgrbjdlXxVEEGYkxI

xRndyEUwWU_ad5ce22YI6Is&m=tny

Quoted 1 line

gBVwOnoZDV7hItku1HtmsI8R3F6vPJdr7ON3hE5iAds96T2C9JTNcnt6ptN4Q&s=s10yJo

gwRRXHqAuIay47H-

Quoted 11 lines

aWl9SL5wpQ9tCjfiQUgrY&e=

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
lib/eal/include/rte_pmu.h new file +9 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return pmc + offset;
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

Review

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +3 -0
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get

perf_event_mmap_page lock\n");

(quoted diff) +2 -0
+			break;
+		}

- Remove the 4 above lines of code, and move the debug log message to

the end of the function

Quoted 51 lines

instead.

Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+	}

+ RTE_LOG(DEBUG, EAL, "failed to get perf_event_mmap_page lock\n");

Quoted 45 lines
Quoted 31 lines
Quoted 31 lines
lib/eal/include/rte_pmu.h new file +26 -0
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under

/sys/bus/event_source/devices/pmu/events.
+ * @return
+ * Event index in case of success, negative value otherwise.
+ */
+__rte_experimental

Quoted 8 lines

lib/eal/include/rte_pmu.h

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +8 -0
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an

event.

Quoted 5 lines
Quoted 5 lines

+ *
+ * @param index
+ * Index of an event to be read.
+ * @return
+ * Event value read from register. In case of errors or lack

of

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

support
+ * 0 is returned. In other words, stream of zeros in a trace

file

Quoted 1 line
Quoted 1 line

+ * indicates problem with reading particular PMU event

register.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines

+ */

(quoted diff) +2 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(int index)

The index type can be changed from int to uint32_t. This also

eliminates the "(index < 0" part of

Quoted 2 lines

the comparison further below in this function.

That's true.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/eal/include/rte_pmu.h new file +4 -0
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

lib/eal/include/rte_pmu.h line 168

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {

Optimized: if (unlikely(!group->enabled)) {

Quoted 2 lines

Compiler will optimize the branch itself correctly. Extra hint is not required.

I haven't reviewed the output from this, so I'll take your word for it. I suggested the unlikely() because I previously tested some very simple code, and it optimized for taking the "if":

void testb(bool b)

{
    if (!b)
        exit(1);
    
    exit(99);
}

I guess I should experiment with more realistic code, and update my optimization notes!

You could add the unlikely() for readability purposes. ;-)

Quoted 61 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/eal/include/rte_pmu.h line 174

Quoted 7 lines
lib/eal/include/rte_pmu.h new file +6 -0
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}

Why is the group not enabled in the setup function,

Quoted 6 lines
Quoted 6 lines

rte_pmu_add_event(), instead of here, in the

Quoted 2 lines

hot path?

When this is executed for the very first time then cpu will have obviously more work to do but afterwards setup path is not taken

hence

Quoted 4 lines
Quoted 4 lines

much less cpu cycles are required.

Setup is executed by main lcore solely, before lcores are executed hence some info passed to SYS_perf_event_open ioctl() is missing,

pid

Quoted 3 lines
Quoted 1 line

(via rte_gettid()) being an example here.

OK. Thank you for the explanation. Since impossible at setup, it has

to be done at runtime.

Quoted 2 lines

@Mattias: Another good example of something that would belong in per-

thread constructors, as my

Quoted 5 lines

suggested feature creep in [1].

[1]: https://urldefense.proofpoint.com/v2/url?u=http-
3A__inbox.dpdk.org_dev_98CBD80474FA8B44BF855DF32C47DC35D87553- 40smartserver.smartshare.dk_&d=DwIFAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNXg

rbjdlXxVEEGYkxIxRndyEUwWU_ad5

Quoted 1 line

ce22YI6Is&m=tnygBVwOnoZDV7hItku1HtmsI8R3F6vPJdr7ON3hE5iAds96T2C9JTNcnt

6ptN4Q&s=aSAnYqgVnrgDp6yyMtGC

Quoted 24 lines

uWgJjDlgqj1wHf1nGWyHCNo&e=

Quoted 3 lines
Quoted 3 lines

lib/eal/include/rte_pmu.h line 176

Quoted 3 lines
lib/eal/include/rte_pmu.h new file +2 -0
+
+	if (index < 0 || index >= rte_pmu->num_group_events)

Optimized: if (unlikely(index >= rte_pmu.num_group_events))

Quoted 16 lines
Quoted 7 lines
Quoted 4 lines
lib/eal/common/rte_pmu.c new file +3 -0
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page

*)group->mmap_pages[index]);

Using fixed size arrays instead of multiple indirections via pointers

is faster. It could be:

Quoted 7 lines

return rte_pmu_read_userpage((struct perf_event_mmap_page *)rte_pmu.group[lcore_id].mmap_pages[index]);

With our without suggested performance improvements...

#29 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Quoted 327 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Thursday, January 5, 2023 11:08 PM
To: Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]
Subject: [EXT] RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 68 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Thursday, 5 January 2023 22.14

Hi Morten,

A few comments inline.

Review

Quoted 8 lines

From: Morten Brørup <[email protected]>
Sent: Wednesday, December 14, 2022 11:41 AM

External Email

---------------------------------------------------------------------

(quoted diff) +1 -1
-
+CC: Mattias, see my comment below about per-thread constructor for

this

Quoted 8 lines
Quoted 8 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 10.39

Hello Morten,

Thanks for review. Answers inline.

Quoted 8 lines
Quoted 8 lines

[...]

Quoted 6 lines
Quoted 6 lines

+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup

PMU

Quoted 13 lines
Quoted 13 lines

and

Quoted 12 lines
Quoted 10 lines

+ * read selected counters in runtime.
+ */

lib/eal/include/rte_pmu.h new file +7 -0
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped

perf_event_attr structures */

There seems to be a lot of indirection involved here. Why are

these

Quoted 5 lines
Quoted 5 lines

arrays not statically sized,

Quoted 2 lines

instead of dynamically allocated?

Different architectures/pmus impose limits on number of

simultaneously

Quoted 1 line
Quoted 1 line

enabled counters. So in order relief the pain of thinking about it

and

Quoted 1 line
Quoted 1 line

adding macros for each and every arch I decided to allocate the

number

Quoted 5 lines
Quoted 3 lines

user wants dynamically. Also assumption holds that user knows about tradeoffs of using too many counters hence will not enable too many events at once.

The DPDK convention is to use fixed size arrays (with a maximum size,

e.g. RTE_MAX_ETHPORTS) in the

Quoted 4 lines

fast path, for performance reasons.

Please use fixed size arrays instead of dynamically allocated arrays.

Quoted 7 lines

I do agree that from maintenance angle fixed arrays are much more convenient but when optimization kicks in then that statement does not necessarily hold true anymore.

For example, in this case performance dropped by ~0.3% which is insignificant imo. So given simpler code, next patchset will use fixed arrays.

I fail to understand how pointer chasing can perform better than obtaining an address by multiplying by a constant. Modern CPUs work in mysterious ways, and you obviously tested this, so I believe your test results. But in theory, pointer chasing touches more cache lines, and should perform worse in a loaded system where pointers in the chain have been evicted from the cache.

Anyway, you agreed to use fixed arrays, so I am happy. :-)

Quoted 102 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Also, what is the reason for hiding the type struct

perf_event_mmap_page **mmap_pages opaque by

Quoted 1 line

using void **mmap_pages instead?

Quoted 8 lines
Quoted 2 lines

I think, that part doing mmap/munmap was written first hence void ** was chosen in the first place.

Please update it, so the actual type is reflected here.

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +1 -0
+	bool enabled; /**< true if group was enabled on particular

lcore

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

*/

lib/eal/include/rte_pmu.h new file +9 -0
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */ };
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +6 -0
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

(quoted diff) +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per

lcore

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines

event group data */

(quoted diff) +2 -0
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of

matching

Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

events */

The event_list is used in slow path only, so it can remain a list -

i.e. no change requested here.

Quoted 12 lines

:-)

Quoted 10 lines
Quoted 6 lines

app/test/test_pmu.c

Quoted 4 lines
app/test/test_pmu.c new file +4 -0
+};
+
+/** Pointer to the PMU state container */ extern struct
+rte_pmu *rte_pmu;

Again, why not just extern struct rte_pmu, instead of dynamic

allocation?

Quoted 1 line

No strong opinions here since this is a matter of personal

preference.

Quoted 9 lines
Quoted 2 lines

Can be removed
in the next version.

Yes, please.

Quoted 4 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+
+/** Each architecture supporting PMU needs to provide its own

version

Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

*/

lib/eal/include/rte_pmu.h new file +7 -0
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; }) #endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines

+ *
+ * @param pc
+ * Pointer to the mmapped user page.
+ * @return
+ * Counter value read from hardware.
+ */

(quoted diff) +2 -0
+__rte_internal
+static __rte_always_inline uint64_t

rte_pmu_read_userpage(struct

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

lib/eal/include/rte_pmu.h line 87

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+perf_event_mmap_page *pc) {
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {

As a matter of personal preference, I would write this loop

differently:

Quoted 12 lines

+ for (tries = 100; tries != 0; tries--) {

Quoted 7 lines
Quoted 7 lines

lib/eal/include/rte_pmu.h line 94

Quoted 7 lines
lib/eal/include/rte_pmu.h new file +7 -0
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {

Why "&& index"? The way I read [man perf_event_open], index 0 is

Quoted 6 lines

perfectly valid.

Quoted 1 line

Valid index starts at 1. 0 means that either hw counter is stopped or isn't active. Maybe this is not initially clear from man but there's example later on how to get actual number.

OK. Thanks for the reference.

Please add a comment about the special meaning of index 0 in the code.

Quoted 113 lines
Quoted 2 lines

[man perf_event_open]:

https://urldefense.proofpoint.com/v2/url?u=https-
3A__man7.org_linux_man-

Quoted 3 lines

2Dpages_man2_perf-5Fevent- 5Fopen.2.html&d=DwIFAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNXgrbjdlXxVEEGYkx I

xRndyEUwWU_ad5ce22YI6Is&m=tny

Quoted 2 lines

gBVwOnoZDV7hItku1HtmsI8R3F6vPJdr7ON3hE5iAds96T2C9JTNcnt6ptN4Q&s=s10yJ o

gwRRXHqAuIay47H-

Quoted 11 lines

aWl9SL5wpQ9tCjfiQUgrY&e=

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
lib/eal/include/rte_pmu.h new file +9 -0
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return pmc + offset;
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

Review

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +3 -0
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get

perf_event_mmap_page lock\n");

(quoted diff) +2 -0
+			break;
+		}

- Remove the 4 above lines of code, and move the debug log message to

the end of the function

Quoted 51 lines

instead.

Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+	}

+ RTE_LOG(DEBUG, EAL, "failed to get perf_event_mmap_page lock\n");

Quoted 45 lines
Quoted 31 lines
Quoted 31 lines
lib/eal/include/rte_pmu.h new file +26 -0
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under

/sys/bus/event_source/devices/pmu/events.
+ * @return
+ * Event index in case of success, negative value otherwise.
+ */
+__rte_experimental

Quoted 8 lines

lib/eal/include/rte_pmu.h

Quoted 8 lines
lib/eal/include/rte_pmu.h new file +8 -0
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an

event.

Quoted 5 lines
Quoted 5 lines

+ *
+ * @param index
+ * Index of an event to be read.
+ * @return
+ * Event value read from register. In case of errors or lack

of

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

support
+ * 0 is returned. In other words, stream of zeros in a trace

file

Quoted 1 line
Quoted 1 line

+ * indicates problem with reading particular PMU event

register.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines

+ */

(quoted diff) +2 -0
+__rte_experimental
+static __rte_always_inline uint64_t rte_pmu_read(int index)

The index type can be changed from int to uint32_t. This also

eliminates the "(index < 0" part of

Quoted 2 lines

the comparison further below in this function.

That's true.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/eal/include/rte_pmu.h new file +4 -0
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

lib/eal/include/rte_pmu.h line 168

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {

Optimized: if (unlikely(!group->enabled)) {

Quoted 2 lines

Compiler will optimize the branch itself correctly. Extra hint is not required.

I haven't reviewed the output from this, so I'll take your word for it. I suggested the unlikely() because I previously tested some very simple code, and it optimized for taking the "if":

void testb(bool b)

Quoted 9 lines
{
   if (!b)
       exit(1);

   exit(99);
}

I guess I should experiment with more realistic code, and update my optimization notes!

I think this may be too simple to draw far-reaching conclusions from it. Compiler will make the fall-through path more likely. If I recall Intel Optimization Reference Manual has some more info on this.

Lets take a different example.

int main(int argc, char *argv[])

{
        int *p;

        p = malloc(sizeof(*p));
        if (!p)
                return 1;
        *p = atoi(argv[1]);
        if (*p < 0)
                return 2;
        free(p);

        return 0;
}

If compiled with -O3 and disassembled I got below.

00000000000010a0 <main>:

    10a0:       f3 0f 1e fa             endbr64
    10a4:       55                      push   %rbp
    10a5:       bf 04 00 00 00          mov    $0x4,%edi
    10aa:       53                      push   %rbx
    10ab:       48 89 f3                mov    %rsi,%rbx
    10ae:       48 83 ec 08             sub    $0x8,%rsp
    10b2:       e8 d9 ff ff ff          call   1090 <malloc@plt>
    10b7:       48 85 c0                test   %rax,%rax
    10ba:       74 31                   je     10ed <main+0x4d>
    10bc:       48 8b 7b 08             mov    0x8(%rbx),%rdi
    10c0:       ba 0a 00 00 00          mov    $0xa,%edx
    10c5:       31 f6                   xor    %esi,%esi
    10c7:       48 89 c5                mov    %rax,%rbp
    10ca:       e8 b1 ff ff ff          call   1080 <strtol@plt>
    10cf:       49 89 c0                mov    %rax,%r8
    10d2:       b8 02 00 00 00          mov    $0x2,%eax
    10d7:       45 85 c0                test   %r8d,%r8d
    10da:       78 0a                   js     10e6 <main+0x46>
    10dc:       48 89 ef                mov    %rbp,%rdi
    10df:       e8 8c ff ff ff          call   1070 <free@plt>
    10e4:       31 c0                   xor    %eax,%eax
    10e6:       48 83 c4 08             add    $0x8,%rsp
    10ea:       5b                      pop    %rbx
    10eb:       5d                      pop    %rbp
    10ec:       c3                      ret
    10ed:       b8 01 00 00 00          mov    $0x1,%eax
    10f2:       eb f2                   jmp    10e6 <main+0x46>

Looking at both 10ba and 10da suggests that code was laid out in a way that jumping is frowned upon. Also potentially lest frequently executed code (at 10ed) is pushed further down the memory hence optimizing cache line usage.

That said, each and every scenario needs analysis on its own.

Quoted 2 lines

You could add the unlikely() for readability purposes. ;-)

Sure. That won't hurt performance.

Quoted 63 lines
Quoted 63 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/eal/include/rte_pmu.h line 174

Quoted 7 lines
lib/eal/include/rte_pmu.h new file +6 -0
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}

Why is the group not enabled in the setup function,

Quoted 6 lines
Quoted 6 lines

rte_pmu_add_event(), instead of here, in the

Quoted 2 lines

hot path?

When this is executed for the very first time then cpu will have obviously more work to do but afterwards setup path is not taken

hence

Quoted 4 lines
Quoted 4 lines

much less cpu cycles are required.

Setup is executed by main lcore solely, before lcores are executed hence some info passed to SYS_perf_event_open ioctl() is missing,

pid

Quoted 3 lines
Quoted 1 line

(via rte_gettid()) being an example here.

OK. Thank you for the explanation. Since impossible at setup, it has

to be done at runtime.

Quoted 2 lines

@Mattias: Another good example of something that would belong in per-

thread constructors, as my

Quoted 6 lines

suggested feature creep in [1].

[1]: https://urldefense.proofpoint.com/v2/url?u=http-
3A__inbox.dpdk.org_dev_98CBD80474FA8B44BF855DF32C47DC35D87553- 40smartserver.smartshare.dk_&d=DwIFAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNX g

rbjdlXxVEEGYkxIxRndyEUwWU_ad5

Quoted 2 lines

ce22YI6Is&m=tnygBVwOnoZDV7hItku1HtmsI8R3F6vPJdr7ON3hE5iAds96T2C9JTNcn t

6ptN4Q&s=aSAnYqgVnrgDp6yyMtGC

Quoted 24 lines

uWgJjDlgqj1wHf1nGWyHCNo&e=

Quoted 3 lines
Quoted 3 lines

lib/eal/include/rte_pmu.h line 176

Quoted 3 lines
lib/eal/include/rte_pmu.h new file +2 -0
+
+	if (index < 0 || index >= rte_pmu->num_group_events)

Optimized: if (unlikely(index >= rte_pmu.num_group_events))

Quoted 16 lines
Quoted 7 lines
Quoted 4 lines
lib/eal/common/rte_pmu.c new file +3 -0
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page

*)group->mmap_pages[index]);

Using fixed size arrays instead of multiple indirections via pointers

is faster. It could be:

Quoted 7 lines

return rte_pmu_read_userpage((struct perf_event_mmap_page *)rte_pmu.group[lcore_id].mmap_pages[index]);

With our without suggested performance improvements...

#30 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Quoted 31 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Sunday, 8 January 2023 16.41

Quoted 16 lines

From: Morten Brørup <[email protected]>
Sent: Thursday, January 5, 2023 11:08 PM

Quoted 13 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Thursday, 5 January 2023 22.14

Hi Morten,

A few comments inline.

Quoted 6 lines

From: Morten Brørup <[email protected]>
Sent: Wednesday, December 14, 2022 11:41 AM

External Email

-------------------------------------------------------------------

--

Quoted 11 lines
Quoted 11 lines

Review

Quoted 2 lines
(quoted diff) +1 -1
-
+CC: Mattias, see my comment below about per-thread constructor for

this

Quoted 8 lines
Quoted 8 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 14 December 2022 10.39

Hello Morten,

Thanks for review. Answers inline.

[...]

Quoted 32 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 10 lines
Quoted 10 lines

Review

Quoted 10 lines
(quoted diff) +10 -0
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {

Optimized: if (unlikely(!group->enabled)) {

Quoted 1 line
Quoted 1 line

Compiler will optimize the branch itself correctly. Extra hint is

not

Quoted 3 lines
Quoted 1 line

required.

I haven't reviewed the output from this, so I'll take your word for

it. I suggested the unlikely()

Quoted 1 line

because I previously tested some very simple code, and it optimized

for taking the "if":

Quoted 10 lines

void testb(bool b)

{
   if (!b)
       exit(1);

   exit(99);
}

I guess I should experiment with more realistic code, and update my

Quoted 8 lines

optimization notes!

Quoted 1 line

I think this may be too simple to draw far-reaching conclusions from it. Compiler will make the
fall-through path more likely. If I recall Intel Optimization Reference Manual has some more
info on this.

IIRC, the Intel Optimization Reference Manual discusses branch optimization for assembler, not C.

Quoted 49 lines

Lets take a different example.

int main(int argc, char *argv[])

{
        int *p;

        p = malloc(sizeof(*p));
        if (!p)
                return 1;
        *p = atoi(argv[1]);
        if (*p < 0)
                return 2;
        free(p);

        return 0;
}

If compiled with -O3 and disassembled I got below.

00000000000010a0 <main>:

    10a0:       f3 0f 1e fa             endbr64
    10a4:       55                      push   %rbp
    10a5:       bf 04 00 00 00          mov    $0x4,%edi
    10aa:       53                      push   %rbx
    10ab:       48 89 f3                mov    %rsi,%rbx
    10ae:       48 83 ec 08             sub    $0x8,%rsp
    10b2:       e8 d9 ff ff ff          call   1090 <malloc@plt>
    10b7:       48 85 c0                test   %rax,%rax
    10ba:       74 31                   je     10ed <main+0x4d>
    10bc:       48 8b 7b 08             mov    0x8(%rbx),%rdi
    10c0:       ba 0a 00 00 00          mov    $0xa,%edx
    10c5:       31 f6                   xor    %esi,%esi
    10c7:       48 89 c5                mov    %rax,%rbp
    10ca:       e8 b1 ff ff ff          call   1080 <strtol@plt>
    10cf:       49 89 c0                mov    %rax,%r8
    10d2:       b8 02 00 00 00          mov    $0x2,%eax
    10d7:       45 85 c0                test   %r8d,%r8d
    10da:       78 0a                   js     10e6 <main+0x46>
    10dc:       48 89 ef                mov    %rbp,%rdi
    10df:       e8 8c ff ff ff          call   1070 <free@plt>
    10e4:       31 c0                   xor    %eax,%eax
    10e6:       48 83 c4 08             add    $0x8,%rsp
    10ea:       5b                      pop    %rbx
    10eb:       5d                      pop    %rbp
    10ec:       c3                      ret
    10ed:       b8 01 00 00 00          mov    $0x1,%eax
    10f2:       eb f2                   jmp    10e6 <main+0x46>
Quoted 4 lines

Looking at both 10ba and 10da suggests that code was laid out in a way that jumping is frowned upon. Also
potentially lest frequently executed code (at 10ed) is pushed further down the memory hence optimizing cache line usage.

In my notes, I have (ptr == NULL) marked as considered unlikely, but (int == 0) marked as considered likely. Since group->enabled is bool, I guessed the compiler would treat it like int and consider (!group->enabled) as likely.

Like in your example here, I also have (int < 0) marked as considered unlikely.

Quoted 2 lines

That said, each and every scenario needs analysis on its own.

Agree. Theory is good, validation is better. ;-)

Quoted 5 lines
Quoted 3 lines

You could add the unlikely() for readability purposes. ;-)

Sure. That won't hurt performance.

I think we are both in agreement about the intentions here, so I won't hold you back with further academic discussions at this point. I might resume the discussion with your next patch version, though. ;-)

#31 lore raw

Re: [PATCH v4 1/4] eal: add generic support for reading PMU events

In reply to Tomasz Duszynski
Quoted 229 lines · Tomasz Duszynski

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
  app/test/meson.build                  |   1 +
  app/test/test_pmu.c                   |  41 +++
  doc/guides/prog_guide/profile_app.rst |   8 +
  lib/eal/common/meson.build            |   3 +
  lib/eal/common/pmu_private.h          |  41 +++
  lib/eal/common/rte_pmu.c              | 456 ++++++++++++++++++++++++++
  lib/eal/include/meson.build           |   1 +
  lib/eal/include/rte_pmu.h             | 204 ++++++++++++
  lib/eal/linux/eal.c                   |   4 +
  lib/eal/version.map                   |   6 +
  10 files changed, 765 insertions(+)

create mode 100644 app/test/test_pmu.c
create mode 100644 lib/eal/common/pmu_private.h
create mode 100644 lib/eal/common/rte_pmu.c
create mode 100644 lib/eal/include/rte_pmu.h

app/test/meson.build +1 -0
index f34d19e3c3..93b3300309 100644
@@ -143,6 +143,7 @@ test_sources = files(
          'test_timer_racecond.c',
          'test_timer_secondary.c',
          'test_ticketlock.c',
+        'test_pmu.c',
          'test_trace.c',
          'test_trace_register.c',
          'test_trace_perf.c',
app/test/test_pmu.c new file +41 -0
new file mode 100644
index 0000000000..fd331af9ee
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <rte_pmu.h>
+
+#include "test.h"
+
+static int
+test_pmu_read(void)
+{
+	uint64_t val = 0;
+	int tries = 10;
+	int event = -1;
+
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	if (val == 0)
+		return TEST_FAILED;
+
+	return TEST_SUCCESS;
+}
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_pmu(void)
+{
+	return unit_test_suite_runner(&pmu_tests);
+}
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/guides/prog_guide/profile_app.rst +8 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
  The following sections describe methods of profiling DPDK applications on
  different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
  Profiling on x86
  ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
          'rte_service.c',
          'rte_version.c',
  )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
  if is_linux or is_windows
      sources += files('eal_common_dynmem.c')
  endif
lib/eal/common/pmu_private.h new file +41 -0
new file mode 100644
index 0000000000..cade4245e6
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+/**
+ * Initialize PMU tracing internals.
+ */
+void
+eal_pmu_init(void);
+
+/**
+ * Cleanup PMU internals.
+ */
+void
+eal_pmu_fini(void);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/eal/common/rte_pmu.c new file +59 -0
new file mode 100644
index 0000000000..049fe19fe3
@@ -0,0 +1,456 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_eal_paging.h>
+#include <rte_pmu.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64 - 1 - (h)))))
+#endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) & (m))
+#endif
+
+struct rte_pmu *rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op. Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t config[3])
+{
+	RTE_SET_USED(config);
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask)
+{

lib/eal/common/rte_pmu.c line 69

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+	char *config = NULL;
+	char path[PATH_MAX];
+	int high, low, ret;
+	FILE *fp;
+
+	/* quiesce -Wmaybe-uninitialized warning */
+	*num = 0;
+	*mask = 0;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu->name, name)

This code might crash in case a long name is supplied, which is maybe not want you want. A trunacte and a "file not found" is probably better. I believe there is a snprintf lookalike with these properties in DPDK.

Quoted 170 lines
lib/eal/common/rte_pmu.c new file +170 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3])
+{
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3])
+{
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu->name, name);
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int lcore_id, int group_fd)
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, rte_gettid(), rte_lcore_to_cpu_id(lcore_id),
+		       group_fd, 0);
+}
+
+static int
+open_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to get %s event config\n", event->name);
+			continue;
+		}
+
+		ret = do_perf_event_open(config, lcore_id, group->fds[0]);
+		if (ret == -1) {
+			if (errno == EOPNOTSUPP)
+				RTE_LOG(ERR, EAL, "64 bit counters not supported\n");
+
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	void *addr;
+	int ret, i;
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		addr = mmap(0, rte_mem_page_size(), PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+	}
+
+	return 0;
+out:

lib/eal/common/rte_pmu.c line 249

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], rte_mem_page_size());
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(int lcore_id)

lcore_id is an unsigned int.

lib/eal/common/rte_pmu.c line 255

Quoted 6 lines
lib/eal/common/rte_pmu.c new file +6 -0
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int i;
+
+	if (!group->fds)
+		return;

group->fds == NULL

This coding style violating appears throughput the patch set.

Quoted 16 lines
lib/eal/common/rte_pmu.c new file +16 -0
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], rte_mem_page_size());
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+

lib/eal/common/rte_pmu.c line 281

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+	free(group->mmap_pages);
+	free(group->fds);
+
+	group->mmap_pages = NULL;
+	group->fds = NULL;
+	group->enabled = false;
+}
+
+int __rte_noinline
+rte_pmu_enable_group(int lcore_id)

unsigned

Quoted 160 lines
lib/eal/common/rte_pmu.c new file +160 -0
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int ret;
+
+	if (rte_pmu->num_group_events == 0) {
+		RTE_LOG(DEBUG, EAL, "no matching PMU events\n");
+
+		return 0;
+	}
+
+	group->fds = calloc(rte_pmu->num_group_events, sizeof(*group->fds));
+	if (!group->fds) {
+		RTE_LOG(ERR, EAL, "failed to alloc descriptor memory\n");
+
+		return -ENOMEM;
+	}
+
+	group->mmap_pages = calloc(rte_pmu->num_group_events, sizeof(*group->mmap_pages));
+	if (!group->mmap_pages) {
+		RTE_LOG(ERR, EAL, "failed to alloc userpage memory\n");
+
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ret = open_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to open events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	ret = mmap_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to map events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		RTE_LOG(ERR, EAL, "failed to enable events on lcore-worker-%d\n", lcore_id);
+
+		ret = -errno;
+		goto out;
+	}
+
+	return 0;
+
+out:
+	cleanup_events(lcore_id);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (!dirp)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	closedir(dirp);
+
+	if (dent) {
+		rte_pmu->name = strdup(name);
+		if (!rte_pmu->name)
+			return -ENOMEM;
+	}
+
+	return rte_pmu->name ? 0 : -ENODEV;
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu->name, name);
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = calloc(1, sizeof(*event));
+	if (!event)
+		return -ENOMEM;
+
+	event->name = strdup(name);
+	if (!event->name) {
+		free(event);
+
+		return -ENOMEM;
+	}
+
+	event->index = rte_pmu->num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu->event_list, event, next);
+
+	RTE_LOG(DEBUG, EAL, "%s even added at index %d\n", name, event->index);
+
+	return event->index;
+}
+
+void
+eal_pmu_init(void)
+{
+	int ret;
+
+	rte_pmu = calloc(1, sizeof(*rte_pmu));
+	if (!rte_pmu) {
+		RTE_LOG(ERR, EAL, "failed to alloc PMU\n");
+
+		return;
+	}
+
+	TAILQ_INIT(&rte_pmu->event_list);
+
+	ret = scan_pmus();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to find core pmu\n");
+		goto out;
+	}
+
+	ret = pmu_arch_init();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to setup arch for PMU\n");
+		goto out;
+	}
+
+	return;
+out:
+	free(rte_pmu->name);
+	free(rte_pmu);
+}
+
+void
+eal_pmu_fini(void)
+{
+	struct rte_pmu_event *event, *tmp;

lib/eal/common/rte_pmu.c line 451

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+	int lcore_id;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu->event_list, next, tmp) {
+		TAILQ_REMOVE(&rte_pmu->event_list, event, next);
+		free(event->name);
+		free(event);
+	}
+
+	RTE_LCORE_FOREACH_WORKER(lcore_id)
+		cleanup_events(lcore_id);

Why is the main lcore left out?

Quoted 278 lines
lib/eal/common/rte_pmu.c new file +5 -0
+
+	pmu_arch_fini();
+	free(rte_pmu->name);
+	free(rte_pmu);
+}
lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
          'rte_pci_dev_features.h',
          'rte_per_lcore.h',
          'rte_pflock.h',
+        'rte_pmu.h',
          'rte_random.h',
          'rte_reciprocal.h',
          'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +204 -0
new file mode 100644
index 0000000000..e4b4f6b052
@@ -0,0 +1,204 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef RTE_EXEC_ENV_LINUX
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_lcore.h>
+#include <rte_log.h>
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped perf_event_attr structures */
+	bool enabled; /**< true if group was enabled on particular lcore */
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */
+};
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under /sys/bus/event_source/devices */
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore event group data */
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events */
+};
+
+/** Pointer to the PMU state container */
+extern struct rte_pmu *rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version */
+#ifndef rte_pmu_pmc_read
+#define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return pmc + offset;
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get perf_event_mmap_page lock\n");
+			break;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}
+
+	if (index < 0 || index >= rte_pmu->num_group_events)
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page *)group->mmap_pages[index]);
+}
+
+#else /* !RTE_EXEC_ENV_LINUX */
+
+__rte_experimental
+static int __rte_unused
+rte_pmu_add_event(__rte_unused const char *name)
+{
+	return -1;
+}
+
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(__rte_unused int index)
+{
+	return 0;
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/eal/linux/eal.c +4 -0
index 8c118d0d9f..751a13b597 100644
@@ -53,6 +53,7 @@
  #include "eal_options.h"
  #include "eal_vfio.h"
  #include "hotplug_mp.h"
+#include "pmu_private.h"
 
  #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
  		return -1;
  	}
 
+	eal_pmu_init();
+
  	if (rte_eal_tailqs_init() < 0) {
  		rte_eal_init_alert("Cannot init tail queues for objects");
  		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
  	eal_bus_cleanup();
  	rte_trace_save();
  	eal_trace_fini();
+	eal_pmu_fini();
  	/* after this point, any DPDK pointers will become dangling */
  	rte_eal_memory_detach();
  	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map +6 -0
index 7ad12a7dc9..9225f46f67 100644
@@ -440,6 +440,11 @@ EXPERIMENTAL {
  	rte_thread_detach;
  	rte_thread_equal;
  	rte_thread_join;
+
+	# added in 23.03
+	rte_pmu; # WINDOWS_NO_EXPORT
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
  };
 
  INTERNAL {
@@ -483,4 +488,5 @@ INTERNAL {
  	rte_mem_map;
  	rte_mem_page_size;
  	rte_mem_unmap;
+	rte_pmu_enable_group;
  };
#32 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Quoted 247 lines

-----Original Message-----
From: Mattias Rönnblom <[email protected]>
Sent: Thursday, December 15, 2022 9:46 AM
To: Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v4 1/4] eal: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 90 lines · Tomasz Duszynski

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
  app/test/meson.build                  |   1 +
  app/test/test_pmu.c                   |  41 +++
  doc/guides/prog_guide/profile_app.rst |   8 +
  lib/eal/common/meson.build            |   3 +
  lib/eal/common/pmu_private.h          |  41 +++
  lib/eal/common/rte_pmu.c              | 456 ++++++++++++++++++++++++++
  lib/eal/include/meson.build           |   1 +
  lib/eal/include/rte_pmu.h             | 204 ++++++++++++
  lib/eal/linux/eal.c                   |   4 +
  lib/eal/version.map                   |   6 +
  10 files changed, 765 insertions(+)

create mode 100644 app/test/test_pmu.c
create mode 100644 lib/eal/common/pmu_private.h
create mode 100644 lib/eal/common/rte_pmu.c
create mode 100644 lib/eal/include/rte_pmu.h

app/test/meson.build index renamed from app/test/meson.build +0 -0

f34d19e3c3..93b3300309 100644

app/test/meson.build +1 -0
@@ -143,6 +143,7 @@ test_sources = files(
          'test_timer_racecond.c',
          'test_timer_secondary.c',
          'test_ticketlock.c',
+        'test_pmu.c',
          'test_trace.c',
          'test_trace_register.c',
          'test_trace_perf.c',
app/test/test_pmu.c new file mode renamed from app/test/test_pmu.c +0 -0

100644 index 0000000000..fd331af9ee

app/test/test_pmu.c new file +41 -0
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <rte_pmu.h>
+
+#include "test.h"
+
+static int
+test_pmu_read(void)
+{
+	uint64_t val = 0;
+	int tries = 10;
+	int event = -1;
+
+	while (tries--)
+		val += rte_pmu_read(event);
+
+	if (val == 0)
+		return TEST_FAILED;
+
+	return TEST_SUCCESS;
+}
+
+static struct unit_test_suite pmu_tests = {
+	.suite_name = "pmu autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_pmu_read),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_pmu(void)
+{
+	return unit_test_suite_runner(&pmu_tests);
+}
+
+REGISTER_TEST_COMMAND(pmu_autotest, test_pmu);
doc/guides/prog_guide/profile_app.rst +0 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
  The following sections describe methods of profiling DPDK applications on

doc/guides/prog_guide/profile_app.rst Profile Your Application · line 17

Quoted 10 lines
doc/guides/prog_guide/profile_app.rst +8 -0
  different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit
+which provides a set of programmable counters that monitor specific
+events. There are different tools which can gather that information,
+perf being an example here. Though in some scenarios, eg. when CPU
+cores are isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such

cases one can read specific events directly from application via ``rte_pmu_read()``.

Quoted 134 lines
doc/guides/prog_guide/profile_app.rst +0 -0
  Profiling on x86
  ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
          'rte_service.c',
          'rte_version.c',
  )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
  if is_linux or is_windows
      sources += files('eal_common_dynmem.c')
  endif
(quoted diff) +0 -0

b/lib/eal/common/pmu_private.h new file mode 100644 index 0000000000..cade4245e6

lib/eal/common/pmu_private.h new file +41 -0
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+/**
+ * Initialize PMU tracing internals.
+ */
+void
+eal_pmu_init(void);
+
+/**
+ * Cleanup PMU internals.
+ */
+void
+eal_pmu_fini(void);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/eal/common/rte_pmu.c new renamed from lib/eal/common/rte_pmu.c +0 -0

file mode 100644 index 0000000000..049fe19fe3

lib/eal/common/rte_pmu.c new file +65 -0
@@ -0,0 +1,456 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_eal_paging.h>
+#include <rte_pmu.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64 - 1 - (h))))) #endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1))
+& (m)) #endif
+
+struct rte_pmu *rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t config[3]) {
+	RTE_SET_USED(config);
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char *config = NULL;
+	char path[PATH_MAX];
+	int high, low, ret;
+	FILE *fp;
+
+	/* quiesce -Wmaybe-uninitialized warning */
+	*num = 0;
Quoted 9 lines

lib/eal/common/rte_pmu.c

Quoted 4 lines
lib/eal/common/rte_pmu.c new file +4 -0
+	*mask = 0;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH
+"/%s/format/%s", rte_pmu->name, name)

This code might crash in case a long name is supplied, which is maybe not want you want. A trunacte and a "file not found" is probably better.
I believe there is a snprintf lookalike with these properties in DPDK.

In scenario, which is pretty unlikely especially because sysfs files have sane names, where 'path' cannot accommodate the whole string there will be NUL implicitly appended by snprintf. Hence fopen will fail. Not sure how this may go wrong.

Quoted 172 lines
Quoted 60 lines
lib/eal/common/rte_pmu.c new file +60 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+

lib/eal/common/rte_pmu.c

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu->name,

name);

Quoted 101 lines
lib/eal/common/rte_pmu.c new file +101 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int lcore_id, int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, rte_gettid(), rte_lcore_to_cpu_id(lcore_id),
+		       group_fd, 0);
+}
+
+static int
+open_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to get %s event config\n", event->name);
+			continue;
+		}
+
+		ret = do_perf_event_open(config, lcore_id, group->fds[0]);
+		if (ret == -1) {
+			if (errno == EOPNOTSUPP)
+				RTE_LOG(ERR, EAL, "64 bit counters not supported\n");
+
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	void *addr;
+	int ret, i;
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		addr = mmap(0, rte_mem_page_size(), PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], rte_mem_page_size());
+		group->mmap_pages[i - 1] = NULL;
+	}
Quoted 9 lines

lib/eal/common/rte_pmu.c line 249

Quoted 6 lines
lib/eal/common/rte_pmu.c new file +6 -0
+
+	return ret;
+}
+
+static void
+cleanup_events(int lcore_id)

lcore_id is an unsigned int.

True, unsigned seems to be more common.

Quoted 11 lines

lib/eal/common/rte_pmu.c line 255

Quoted 6 lines
lib/eal/common/rte_pmu.c new file +6 -0
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int i;
+
+	if (!group->fds)
+		return;

group->fds == NULL

This coding style violating appears throughput the patch set.

Good point.

Quoted 196 lines
Quoted 16 lines
lib/eal/common/rte_pmu.c new file +16 -0
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], rte_mem_page_size());
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+

lib/eal/common/rte_pmu.c line 281

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+	free(group->mmap_pages);
+	free(group->fds);
+
+	group->mmap_pages = NULL;
+	group->fds = NULL;
+	group->enabled = false;
+}
+
+int __rte_noinline
+rte_pmu_enable_group(int lcore_id)

unsigned

Quoted 87 lines
lib/eal/common/rte_pmu.c new file +87 -0
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int ret;
+
+	if (rte_pmu->num_group_events == 0) {
+		RTE_LOG(DEBUG, EAL, "no matching PMU events\n");
+
+		return 0;
+	}
+
+	group->fds = calloc(rte_pmu->num_group_events, sizeof(*group->fds));
+	if (!group->fds) {
+		RTE_LOG(ERR, EAL, "failed to alloc descriptor memory\n");
+
+		return -ENOMEM;
+	}
+
+	group->mmap_pages = calloc(rte_pmu->num_group_events, sizeof(*group->mmap_pages));
+	if (!group->mmap_pages) {
+		RTE_LOG(ERR, EAL, "failed to alloc userpage memory\n");
+
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ret = open_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to open events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	ret = mmap_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to map events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		RTE_LOG(ERR, EAL, "failed to enable events on lcore-worker-%d\n",
+lcore_id);
+
+		ret = -errno;
+		goto out;
+	}
+
+	return 0;
+
+out:
+	cleanup_events(lcore_id);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (!dirp)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	closedir(dirp);
+
+	if (dent) {
+		rte_pmu->name = strdup(name);
+		if (!rte_pmu->name)
+			return -ENOMEM;
+	}
+

lib/eal/common/rte_pmu.c

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+	return rte_pmu->name ? 0 : -ENODEV;
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu->name,

name);

Quoted 69 lines
lib/eal/common/rte_pmu.c new file +69 -0
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = calloc(1, sizeof(*event));
+	if (!event)
+		return -ENOMEM;
+
+	event->name = strdup(name);
+	if (!event->name) {
+		free(event);
+
+		return -ENOMEM;
+	}
+
+	event->index = rte_pmu->num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu->event_list, event, next);
+
+	RTE_LOG(DEBUG, EAL, "%s even added at index %d\n", name,
+event->index);
+
+	return event->index;
+}
+
+void
+eal_pmu_init(void)
+{
+	int ret;
+
+	rte_pmu = calloc(1, sizeof(*rte_pmu));
+	if (!rte_pmu) {
+		RTE_LOG(ERR, EAL, "failed to alloc PMU\n");
+
+		return;
+	}
+
+	TAILQ_INIT(&rte_pmu->event_list);
+
+	ret = scan_pmus();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to find core pmu\n");
+		goto out;
+	}
+
+	ret = pmu_arch_init();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to setup arch for PMU\n");
+		goto out;
+	}
+
+	return;
+out:
+	free(rte_pmu->name);
+	free(rte_pmu);
+}
+
+void
+eal_pmu_fini(void)
+{
+	struct rte_pmu_event *event, *tmp;
+	int lcore_id;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu->event_list, next, tmp) {
+		TAILQ_REMOVE(&rte_pmu->event_list, event, next);
Quoted 9 lines

lib/eal/common/rte_pmu.c line 451

Quoted 6 lines
lib/eal/common/rte_pmu.c new file +6 -0
+		free(event->name);
+		free(event);
+	}
+
+	RTE_LCORE_FOREACH_WORKER(lcore_id)
+		cleanup_events(lcore_id);

Why is the main lcore left out?

Main lcore was omitted because it's pretty uncommon for it to do a heavy-lifting so usefulness of reading counters is questionable. It can be added for completeness though.

Do you have any specific use case on your mind?

Quoted 273 lines
Quoted 273 lines
lib/eal/common/rte_pmu.c new file +5 -0
+
+	pmu_arch_fini();
+	free(rte_pmu->name);
+	free(rte_pmu);
+}
lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
          'rte_pci_dev_features.h',
          'rte_per_lcore.h',
          'rte_pflock.h',
+        'rte_pmu.h',
          'rte_random.h',
          'rte_reciprocal.h',
          'rte_seqcount.h',
lib/eal/include/rte_pmu.h new renamed from lib/eal/include/rte_pmu.h +0 -0

file mode 100644 index 0000000000..e4b4f6b052

lib/eal/include/rte_pmu.h new file +200 -0
@@ -0,0 +1,204 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell
+ */
+
+#ifndef _RTE_PMU_H_
+#define _RTE_PMU_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef RTE_EXEC_ENV_LINUX
+
+#include <linux/perf_event.h>
+
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_lcore.h>
+#include <rte_log.h>
+
+/**
+ * @file
+ *
+ * PMU event tracing operations
+ *
+ * This file defines generic API and types necessary to setup PMU and
+ * read selected counters in runtime.
+ */
+
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int *fds; /**< array of event descriptors */
+	void **mmap_pages; /**< array of pointers to mmapped perf_event_attr structures */
+	bool enabled; /**< true if group was enabled on particular lcore */
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */ };
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under /sys/bus/event_source/devices */
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore event group data */
+	int num_group_events; /**< number of events in a group */
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching events
+*/ };
+
+/** Pointer to the PMU state container */ extern struct rte_pmu
+*rte_pmu;
+
+/** Each architecture supporting PMU needs to provide its own version
+*/ #ifndef rte_pmu_pmc_read #define rte_pmu_pmc_read(index) ({ 0; })
+#endif
+
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t offset, width, pmc = 0;
+	uint32_t seq, index;
+	int tries = 100;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return pmc + offset;
+
+		if (--tries == 0) {
+			RTE_LOG(DEBUG, EAL, "failed to get perf_event_mmap_page lock\n");
+			break;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * @internal
+ *
+ * Enable group of events for a given lcore.
+ *
+ * @param lcore_id
+ *   The identifier of the lcore.
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+__rte_internal
+int
+rte_pmu_enable_group(int lcore_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add event to the group of enabled events.
+ *
+ * @param name
+ *   Name of an event listed under /sys/bus/event_source/devices/pmu/events.
+ * @return
+ *   Event index in case of success, negative value otherwise.
+ */
+__rte_experimental
+int
+rte_pmu_add_event(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ *   Index of an event to be read.
+ * @return
+ *   Event value read from register. In case of errors or lack of support
+ *   0 is returned. In other words, stream of zeros in a trace file
+ *   indicates problem with reading particular PMU event register.
+ */
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(int index)
+{
+	int lcore_id = rte_lcore_id();
+	struct rte_pmu_event_group *group;
+	int ret;
+
+	if (!rte_pmu)
+		return 0;
+
+	group = &rte_pmu->group[lcore_id];
+	if (!group->enabled) {
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;
+	}
+
+	if (index < 0 || index >= rte_pmu->num_group_events)
+		return 0;
+
+	return rte_pmu_read_userpage((struct perf_event_mmap_page
+*)group->mmap_pages[index]); }
+
+#else /* !RTE_EXEC_ENV_LINUX */
+
+__rte_experimental
+static int __rte_unused
+rte_pmu_add_event(__rte_unused const char *name) {
+	return -1;
+}
+
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(__rte_unused int index)
+{
+	return 0;
+}
+
+#endif /* RTE_EXEC_ENV_LINUX */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_H_ */
lib/eal/linux/eal.c index renamed from lib/eal/linux/eal.c +0 -0

8c118d0d9f..751a13b597 100644

lib/eal/linux/eal.c +4 -0
@@ -53,6 +53,7 @@
  #include "eal_options.h"
  #include "eal_vfio.h"
  #include "hotplug_mp.h"
+#include "pmu_private.h"
 
  #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
  		return -1;
  	}
 
+	eal_pmu_init();
+
  	if (rte_eal_tailqs_init() < 0) {
  		rte_eal_init_alert("Cannot init tail queues for objects");
  		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
  	eal_bus_cleanup();
  	rte_trace_save();
  	eal_trace_fini();
+	eal_pmu_fini();
  	/* after this point, any DPDK pointers will become dangling */
  	rte_eal_memory_detach();
  	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map index renamed from lib/eal/version.map +0 -0

7ad12a7dc9..9225f46f67 100644

lib/eal/version.map +6 -0
@@ -440,6 +440,11 @@ EXPERIMENTAL {
  	rte_thread_detach;
  	rte_thread_equal;
  	rte_thread_join;
+
+	# added in 23.03
+	rte_pmu; # WINDOWS_NO_EXPORT
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
  };
 
  INTERNAL {
@@ -483,4 +488,5 @@ INTERNAL {
  	rte_mem_map;
  	rte_mem_page_size;
  	rte_mem_unmap;
+	rte_pmu_enable_group;
  };
#33 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

In reply to Tomasz Duszynski
Quoted 32 lines

-----Original Message-----
From: Tomasz Duszynski <[email protected]>
Sent: Tuesday, December 13, 2022 6:44 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Tomasz Duszynski <[email protected]>
Subject: [PATCH v4 1/4] eal: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 app/test/meson.build                  |   1 +
 app/test/test_pmu.c                   |  41 +++
 doc/guides/prog_guide/profile_app.rst |   8 +
 lib/eal/common/meson.build            |   3 +
 lib/eal/common/pmu_private.h          |  41 +++
 lib/eal/common/rte_pmu.c              | 456 ++++++++++++++++++++++++++
 lib/eal/include/meson.build           |   1 +
 lib/eal/include/rte_pmu.h             | 204 ++++++++++++
 lib/eal/linux/eal.c                   |   4 +
 lib/eal/version.map                   |   6 +
 10 files changed, 765 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h  create mode 100644

lib/eal/common/rte_pmu.c create mode 100644 lib/eal/include/rte_pmu.h

<snip>

Quoted 168 lines
lib/eal/common/rte_pmu.c new file +68 -0
index 0000000000..049fe19fe3
@@ -0,0 +1,456 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_eal_paging.h>
+#include <rte_pmu.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >> ((64
+- 1 - (h))))) #endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1)) &
+(m)) #endif
+
+struct rte_pmu *rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op. Architectures
+should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t config[3]) {
+	RTE_SET_USED(config);
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char *config = NULL;
+	char path[PATH_MAX];
+	int high, low, ret;
+	FILE *fp;
+
+	/* quiesce -Wmaybe-uninitialized warning */
+	*num = 0;
+	*mask = 0;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/format/%s", rte_pmu->name,

name);

lib/eal/common/rte_pmu.c new file +70 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu->name,

name);

lib/eal/common/rte_pmu.c new file +23 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int lcore_id, int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,

lib/eal/common/rte_pmu.c

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, rte_gettid(),

Looks like using '0' instead of rte_gettid() takes the same effect. A small optimization.

Quoted 204 lines

rte_lcore_to_cpu_id(lcore_id),

lib/eal/common/rte_pmu.c new file +203 -0
+		       group_fd, 0);
+}
+
+static int
+open_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to get %s event config\n", event->name);
+			continue;
+		}
+
+		ret = do_perf_event_open(config, lcore_id, group->fds[0]);
+		if (ret == -1) {
+			if (errno == EOPNOTSUPP)
+				RTE_LOG(ERR, EAL, "64 bit counters not supported\n");
+
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	void *addr;
+	int ret, i;
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		addr = mmap(0, rte_mem_page_size(), PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], rte_mem_page_size());
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int i;
+
+	if (!group->fds)
+		return;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], rte_mem_page_size());
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	free(group->mmap_pages);
+	free(group->fds);
+
+	group->mmap_pages = NULL;
+	group->fds = NULL;
+	group->enabled = false;
+}
+
+int __rte_noinline
+rte_pmu_enable_group(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int ret;
+
+	if (rte_pmu->num_group_events == 0) {
+		RTE_LOG(DEBUG, EAL, "no matching PMU events\n");
+
+		return 0;
+	}
+
+	group->fds = calloc(rte_pmu->num_group_events, sizeof(*group->fds));
+	if (!group->fds) {
+		RTE_LOG(ERR, EAL, "failed to alloc descriptor memory\n");
+
+		return -ENOMEM;
+	}
+
+	group->mmap_pages = calloc(rte_pmu->num_group_events, sizeof(*group->mmap_pages));
+	if (!group->mmap_pages) {
+		RTE_LOG(ERR, EAL, "failed to alloc userpage memory\n");
+
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ret = open_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to open events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	ret = mmap_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to map events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		RTE_LOG(ERR, EAL, "failed to enable events on lcore-worker-%d\n",
+lcore_id);
+
+		ret = -errno;
+		goto out;
+	}
+
+	return 0;
+
+out:
+	cleanup_events(lcore_id);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (!dirp)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	closedir(dirp);
+
+	if (dent) {
+		rte_pmu->name = strdup(name);
+		if (!rte_pmu->name)
+			return -ENOMEM;
+	}
+
+	return rte_pmu->name ? 0 : -ENODEV;
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/events/%s", rte_pmu->name,
Quoted 1 line

name);

Better to check if rte_pmu is available.
See below.

Quoted 49 lines
lib/eal/common/rte_pmu.c new file +49 -0
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = calloc(1, sizeof(*event));
+	if (!event)
+		return -ENOMEM;
+
+	event->name = strdup(name);
+	if (!event->name) {
+		free(event);
+
+		return -ENOMEM;
+	}
+
+	event->index = rte_pmu->num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu->event_list, event, next);
+
+	RTE_LOG(DEBUG, EAL, "%s even added at index %d\n", name,
+event->index);
+
+	return event->index;
+}
+
+void
+eal_pmu_init(void)
+{
+	int ret;
+
+	rte_pmu = calloc(1, sizeof(*rte_pmu));
+	if (!rte_pmu) {
+		RTE_LOG(ERR, EAL, "failed to alloc PMU\n");
+
+		return;
+	}
+
+	TAILQ_INIT(&rte_pmu->event_list);
+
+	ret = scan_pmus();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to find core pmu\n");
+		goto out;
+	}
+

lib/eal/common/rte_pmu.c line 435

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+	ret = pmu_arch_init();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to setup arch for PMU\n");
+		goto out;
+	}
+
+	return;
+out:
+	free(rte_pmu->name);
+	free(rte_pmu);

Set rte_pmu to NULL to prevent unintentional use?

lib/eal/common/rte_pmu.c line 444

Quoted 9 lines
lib/eal/common/rte_pmu.c new file +9 -0
+}
+
+void
+eal_pmu_fini(void)
+{
+	struct rte_pmu_event *event, *tmp;
+	int lcore_id;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu->event_list, next, tmp) {

rte_pmu can be unavailable if init fails. Better to check before accessing.

Quoted 12 lines
lib/eal/common/rte_pmu.c new file +12 -0
+		TAILQ_REMOVE(&rte_pmu->event_list, event, next);
+		free(event->name);
+		free(event);
+	}
+
+	RTE_LCORE_FOREACH_WORKER(lcore_id)
+		cleanup_events(lcore_id);
+
+	pmu_arch_fini();
+	free(rte_pmu->name);
+	free(rte_pmu);
+}

<snip>

#34 lore raw

RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

Hi Ruifeng,

Quoted 493 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Monday, January 9, 2023 8:37 AM
To: Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; nd <[email protected]>
Subject: [EXT] RE: [PATCH v4 1/4] eal: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 33 lines

-----Original Message-----
From: Tomasz Duszynski <[email protected]>
Sent: Tuesday, December 13, 2022 6:44 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Tomasz Duszynski <[email protected]>
Subject: [PATCH v4 1/4] eal: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 app/test/meson.build                  |   1 +
 app/test/test_pmu.c                   |  41 +++
 doc/guides/prog_guide/profile_app.rst |   8 +
 lib/eal/common/meson.build            |   3 +
 lib/eal/common/pmu_private.h          |  41 +++
 lib/eal/common/rte_pmu.c              | 456 ++++++++++++++++++++++++++
 lib/eal/include/meson.build           |   1 +
 lib/eal/include/rte_pmu.h             | 204 ++++++++++++
 lib/eal/linux/eal.c                   |   4 +
 lib/eal/version.map                   |   6 +
 10 files changed, 765 insertions(+)
 create mode 100644 app/test/test_pmu.c  create mode 100644

lib/eal/common/pmu_private.h create mode 100644 lib/eal/common/rte_pmu.c create mode 100644 lib/eal/include/rte_pmu.h

<snip>

Quoted 172 lines
lib/eal/common/rte_pmu.c new renamed from lib/eal/common/rte_pmu.c +0 -0

file mode 100644 index 0000000000..049fe19fe3

lib/eal/common/rte_pmu.c new file +71 -0
@@ -0,0 +1,456 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Marvell International Ltd.
+ */
+
+#include <ctype.h>
+#include <dirent.h>
+#include <errno.h>
+#include <regex.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+#include <rte_eal_paging.h>
+#include <rte_pmu.h>
+#include <rte_tailq.h>
+
+#include "pmu_private.h"
+
+#define EVENT_SOURCE_DEVICES_PATH "/sys/bus/event_source/devices"
+
+#ifndef GENMASK_ULL
+#define GENMASK_ULL(h, l) ((~0ULL - (1ULL << (l)) + 1) & (~0ULL >>
+((64
+- 1 - (h))))) #endif
+
+#ifndef FIELD_PREP
+#define FIELD_PREP(m, v) (((uint64_t)(v) << (__builtin_ffsll(m) - 1))
+&
+(m)) #endif
+
+struct rte_pmu *rte_pmu;
+
+/*
+ * Following __rte_weak functions provide default no-op.
+Architectures should override them if
+ * necessary.
+ */
+
+int
+__rte_weak pmu_arch_init(void)
+{
+	return 0;
+}
+
+void
+__rte_weak pmu_arch_fini(void)
+{
+}
+
+void
+__rte_weak pmu_arch_fixup_config(uint64_t config[3]) {
+	RTE_SET_USED(config);
+}
+
+static int
+get_term_format(const char *name, int *num, uint64_t *mask) {
+	char *config = NULL;
+	char path[PATH_MAX];
+	int high, low, ret;
+	FILE *fp;
+
+	/* quiesce -Wmaybe-uninitialized warning */
+	*num = 0;
+	*mask = 0;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH
+"/%s/format/%s", rte_pmu->name,

name);

lib/eal/common/rte_pmu.c new file +71 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	errno = 0;
+	ret = fscanf(fp, "%m[^:]:%d-%d", &config, &low, &high);
+	if (ret < 2) {
+		ret = -ENODATA;
+		goto out;
+	}
+	if (errno) {
+		ret = -errno;
+		goto out;
+	}
+
+	if (ret == 2)
+		high = low;
+
+	*mask = GENMASK_ULL(high, low);
+	/* Last digit should be [012]. If last digit is missing 0 is implied. */
+	*num = config[strlen(config) - 1];
+	*num = isdigit(*num) ? *num - '0' : 0;
+
+	ret = 0;
+out:
+	free(config);
+	fclose(fp);
+
+	return ret;
+}
+
+static int
+parse_event(char *buf, uint64_t config[3]) {
+	char *token, *term;
+	int num, ret, val;
+	uint64_t mask;
+
+	config[0] = config[1] = config[2] = 0;
+
+	token = strtok(buf, ",");
+	while (token) {
+		errno = 0;
+		/* <term>=<value> */
+		ret = sscanf(token, "%m[^=]=%i", &term, &val);
+		if (ret < 1)
+			return -ENODATA;
+		if (errno)
+			return -errno;
+		if (ret == 1)
+			val = 1;
+
+		ret = get_term_format(term, &num, &mask);
+		free(term);
+		if (ret)
+			return ret;
+
+		config[num] |= FIELD_PREP(mask, val);
+		token = strtok(NULL, ",");
+	}
+
+	return 0;
+}
+
+static int
+get_event_config(const char *name, uint64_t config[3]) {
+	char path[PATH_MAX], buf[BUFSIZ];
+	FILE *fp;
+	int ret;
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH
+"/%s/events/%s", rte_pmu->name,

name);

lib/eal/common/rte_pmu.c new file +23 -0
+	fp = fopen(path, "r");
+	if (!fp)
+		return -errno;
+
+	ret = fread(buf, 1, sizeof(buf), fp);
+	if (ret == 0) {
+		fclose(fp);
+
+		return -EINVAL;
+	}
+	fclose(fp);
+	buf[ret] = '\0';
+
+	return parse_event(buf, config);
+}
+
+static int
+do_perf_event_open(uint64_t config[3], int lcore_id, int group_fd) {
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,

lib/eal/common/rte_pmu.c

Quoted 10 lines
lib/eal/common/rte_pmu.c new file +10 -0
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, rte_gettid(),

Looks like using '0' instead of rte_gettid() takes the same effect. A small optimization.

Quoted 205 lines

rte_lcore_to_cpu_id(lcore_id),

lib/eal/common/rte_pmu.c new file +204 -0
+		       group_fd, 0);
+}
+
+static int
+open_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	struct rte_pmu_event *event;
+	uint64_t config[3];
+	int num = 0, ret;
+
+	/* group leader gets created first, with fd = -1 */
+	group->fds[0] = -1;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		ret = get_event_config(event->name, config);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to get %s event config\n", event->name);
+			continue;
+		}
+
+		ret = do_perf_event_open(config, lcore_id, group->fds[0]);
+		if (ret == -1) {
+			if (errno == EOPNOTSUPP)
+				RTE_LOG(ERR, EAL, "64 bit counters not supported\n");
+
+			ret = -errno;
+			goto out;
+		}
+
+		group->fds[event->index] = ret;
+		num++;
+	}
+
+	return 0;
+out:
+	for (--num; num >= 0; num--) {
+		close(group->fds[num]);
+		group->fds[num] = -1;
+	}
+
+
+	return ret;
+}
+
+static int
+mmap_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	void *addr;
+	int ret, i;
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		addr = mmap(0, rte_mem_page_size(), PROT_READ, MAP_SHARED, group->fds[i], 0);
+		if (addr == MAP_FAILED) {
+			ret = -errno;
+			goto out;
+		}
+
+		group->mmap_pages[i] = addr;
+	}
+
+	return 0;
+out:
+	for (; i; i--) {
+		munmap(group->mmap_pages[i - 1], rte_mem_page_size());
+		group->mmap_pages[i - 1] = NULL;
+	}
+
+	return ret;
+}
+
+static void
+cleanup_events(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int i;
+
+	if (!group->fds)
+		return;
+
+	if (group->fds[0] != -1)
+		ioctl(group->fds[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);
+
+	for (i = 0; i < rte_pmu->num_group_events; i++) {
+		if (group->mmap_pages[i]) {
+			munmap(group->mmap_pages[i], rte_mem_page_size());
+			group->mmap_pages[i] = NULL;
+		}
+
+		if (group->fds[i] != -1) {
+			close(group->fds[i]);
+			group->fds[i] = -1;
+		}
+	}
+
+	free(group->mmap_pages);
+	free(group->fds);
+
+	group->mmap_pages = NULL;
+	group->fds = NULL;
+	group->enabled = false;
+}
+
+int __rte_noinline
+rte_pmu_enable_group(int lcore_id)
+{
+	struct rte_pmu_event_group *group = &rte_pmu->group[lcore_id];
+	int ret;
+
+	if (rte_pmu->num_group_events == 0) {
+		RTE_LOG(DEBUG, EAL, "no matching PMU events\n");
+
+		return 0;
+	}
+
+	group->fds = calloc(rte_pmu->num_group_events, sizeof(*group->fds));
+	if (!group->fds) {
+		RTE_LOG(ERR, EAL, "failed to alloc descriptor memory\n");
+
+		return -ENOMEM;
+	}
+
+	group->mmap_pages = calloc(rte_pmu->num_group_events, sizeof(*group->mmap_pages));
+	if (!group->mmap_pages) {
+		RTE_LOG(ERR, EAL, "failed to alloc userpage memory\n");
+
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ret = open_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to open events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	ret = mmap_events(lcore_id);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to map events on lcore-worker-%d\n", lcore_id);
+		goto out;
+	}
+
+	if (ioctl(group->fds[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) == -1) {
+		RTE_LOG(ERR, EAL, "failed to enable events on lcore-worker-%d\n",
+lcore_id);
+
+		ret = -errno;
+		goto out;
+	}
+
+	return 0;
+
+out:
+	cleanup_events(lcore_id);
+
+	return ret;
+}
+
+static int
+scan_pmus(void)
+{
+	char path[PATH_MAX];
+	struct dirent *dent;
+	const char *name;
+	DIR *dirp;
+
+	dirp = opendir(EVENT_SOURCE_DEVICES_PATH);
+	if (!dirp)
+		return -errno;
+
+	while ((dent = readdir(dirp))) {
+		name = dent->d_name;
+		if (name[0] == '.')
+			continue;
+
+		/* sysfs entry should either contain cpus or be a cpu */
+		if (!strcmp(name, "cpu"))
+			break;
+
+		snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH "/%s/cpus", name);
+		if (access(path, F_OK) == 0)
+			break;
+	}
+
+	closedir(dirp);
+
+	if (dent) {
+		rte_pmu->name = strdup(name);
+		if (!rte_pmu->name)
+			return -ENOMEM;
+	}
+
+	return rte_pmu->name ? 0 : -ENODEV;
+}
+
+int
+rte_pmu_add_event(const char *name)
+{
+	struct rte_pmu_event *event;
+	char path[PATH_MAX];
+
+	snprintf(path, sizeof(path), EVENT_SOURCE_DEVICES_PATH
+"/%s/events/%s", rte_pmu->name,
Quoted 1 line

name);

Better to check if rte_pmu is available.
See below.

Quoted 53 lines
lib/eal/common/rte_pmu.c new file +53 -0
+	if (access(path, R_OK))
+		return -ENODEV;
+
+	TAILQ_FOREACH(event, &rte_pmu->event_list, next) {
+		if (!strcmp(event->name, name))
+			return event->index;
+		continue;
+	}
+
+	event = calloc(1, sizeof(*event));
+	if (!event)
+		return -ENOMEM;
+
+	event->name = strdup(name);
+	if (!event->name) {
+		free(event);
+
+		return -ENOMEM;
+	}
+
+	event->index = rte_pmu->num_group_events++;
+	TAILQ_INSERT_TAIL(&rte_pmu->event_list, event, next);
+
+	RTE_LOG(DEBUG, EAL, "%s even added at index %d\n", name,
+event->index);
+
+	return event->index;
+}
+
+void
+eal_pmu_init(void)
+{
+	int ret;
+
+	rte_pmu = calloc(1, sizeof(*rte_pmu));
+	if (!rte_pmu) {
+		RTE_LOG(ERR, EAL, "failed to alloc PMU\n");
+
+		return;
+	}
+
+	TAILQ_INIT(&rte_pmu->event_list);
+
+	ret = scan_pmus();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to find core pmu\n");
+		goto out;
+	}
+
+	ret = pmu_arch_init();
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to setup arch for PMU\n");
+		goto out;
Quoted 9 lines

lib/eal/common/rte_pmu.c line 435

Quoted 6 lines
lib/eal/common/rte_pmu.c new file +6 -0
+	}
+
+	return;
+out:
+	free(rte_pmu->name);
+	free(rte_pmu);

Set rte_pmu to NULL to prevent unintentional use?

Next series will take use of global pmu instance so this will no longer be required though your suggestions may be applied to other pointers around.

Quoted 12 lines

lib/eal/common/rte_pmu.c line 444

Quoted 9 lines
lib/eal/common/rte_pmu.c new file +9 -0
+}
+
+void
+eal_pmu_fini(void)
+{
+	struct rte_pmu_event *event, *tmp;
+	int lcore_id;
+
+	RTE_TAILQ_FOREACH_SAFE(event, &rte_pmu->event_list, next, tmp) {

rte_pmu can be unavailable if init fails. Better to check before accessing.

Yep.

Quoted 13 lines
Quoted 12 lines
lib/eal/common/rte_pmu.c new file +12 -0
+		TAILQ_REMOVE(&rte_pmu->event_list, event, next);
+		free(event->name);
+		free(event);
+	}
+
+	RTE_LCORE_FOREACH_WORKER(lcore_id)
+		cleanup_events(lcore_id);
+
+	pmu_arch_fini();
+	free(rte_pmu->name);
+	free(rte_pmu);
+}

<snip>

#35 lore raw

PATCH v4 2/4 eal/arm: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c               |   4 ++
 lib/eal/arm/include/meson.build   |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h |  39 +++++++++++
 lib/eal/arm/meson.build           |   4 ++
 lib/eal/arm/rte_pmu.c             | 104 ++++++++++++++++++++++++++++++
 lib/eal/include/rte_pmu.h         |   3 +
 6 files changed, 155 insertions(+)
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
app/test/test_pmu.c +4 -0
index fd331af9ee..f94866dff9 100644
@@ -13,6 +13,10 @@ test_pmu_read(void)
 	int tries = 10;
 	int event = -1;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/eal/arm/include/meson.build +1 -0
index 657bf58569..ab13b0220a 100644
@@ -20,6 +20,7 @@ arch_headers = files(
         'rte_pause_32.h',
         'rte_pause_64.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch_32.h',
         'rte_prefetch_64.h',
lib/eal/arm/include/rte_pmu_pmc.h new file +39 -0
new file mode 100644
index 0000000000..10e2984813
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_ARM_H_
+#define _RTE_PMU_PMC_ARM_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_ARM_H_ */
lib/eal/arm/meson.build +4 -0
index dca1106aae..0c5575b197 100644
@@ -9,3 +9,7 @@ sources += files(
         'rte_hypervisor.c',
         'rte_power_intrinsics.c',
 )
+
+if is_linux
+    sources += files('rte_pmu.c')
+endif
lib/eal/arm/rte_pmu.c new file +104 -0

104 lines not on this page · raw message

lib/eal/include/rte_pmu.h +3 -0
index e4b4f6b052..158a616b83 100644
@@ -20,6 +20,9 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
+#if defined(RTE_ARCH_ARM64)
+#include <rte_pmu_pmc.h>
+#endif
 
 /**
  * @file
Signature · 1 line
2.25.1
#36 lore raw

PATCH v4 3/4 eal/x86: support reading Intel PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel PMU events in runtime.

---
 app/test/test_pmu.c               |  2 ++
 lib/eal/include/rte_pmu.h         |  2 +-
 lib/eal/x86/include/meson.build   |  1 +
 lib/eal/x86/include/rte_pmu_pmc.h | 33 +++++++++++++++++++++++++++++++
 4 files changed, 37 insertions(+), 1 deletion(-)
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h
app/test/test_pmu.c +2 -0
index f94866dff9..016204c083 100644
@@ -15,6 +15,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/eal/include/rte_pmu.h +1 -1
index 158a616b83..3d90f4baf7 100644
@@ -20,7 +20,7 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
-#if defined(RTE_ARCH_ARM64)
+#if defined(RTE_ARCH_ARM64) || defined(RTE_ARCH_X86_64)
 #include <rte_pmu_pmc.h>
 #endif
 
lib/eal/x86/include/meson.build +1 -0
index 52d2f8e969..03d286ed25 100644
@@ -9,6 +9,7 @@ arch_headers = files(
         'rte_io.h',
         'rte_memcpy.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch.h',
         'rte_rtm.h',
lib/eal/x86/include/rte_pmu_pmc.h new file +33 -0
new file mode 100644
index 0000000000..a2cd849fb1
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_X86_H_
+#define _RTE_PMU_PMC_X86_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint32_t high, low;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return ((uint64_t)high << 32) | (uint64_t)low;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_X86_H_ */
Signature · 1 line
2.25.1
#37 lore raw

PATCH v4 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               |  4 ++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 ++++++++++++
 lib/eal/common/eal_common_trace_points.c |  3 ++
 lib/eal/common/rte_pmu.c                 | 63 ++++++++++++++++++++++++
 lib/eal/include/rte_eal_trace.h          | 11 +++++
 lib/eal/version.map                      |  1 +
 7 files changed, 119 insertions(+)
app/test/test_trace_perf.c +4 -0
index 46ae7d8074..4851b6852f 100644
@@ -114,6 +114,8 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +124,7 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+WORKER_DEFINE(READ_PMU)
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +177,7 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index a8b501fe0c..6a53341c6b 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..9a845fd86f 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace_points.c +3 -0
index 0b0b254615..de918ca618 100644
@@ -75,3 +75,6 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
lib/eal/common/rte_pmu.c +63 -0

86 lines not on this page · raw message

lib/eal/include/rte_eal_trace.h +11 -0
index 5ef4398230..2a10f63e97 100644
@@ -17,6 +17,7 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#include <rte_pmu.h>
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +280,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+/* PMU */
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(int index),
+	uint64_t val;
+	rte_trace_point_emit_int(index);
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/version.map +1 -0
index 9225f46f67..73803f9601 100644
@@ -442,6 +442,7 @@ EXPERIMENTAL {
 	rte_thread_join;
 
 	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 	rte_pmu; # WINDOWS_NO_EXPORT
 	rte_pmu_add_event; # WINDOWS_NO_EXPORT
 	rte_pmu_read; # WINDOWS_NO_EXPORT
Signature · 1 line
2.25.1
#38 lore raw

v5 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there v4:
- fix freeing mem detected by debug_autotest v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 +++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 104 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 504 +++++++++++++++++++++++
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  10 +
 lib/eal/include/rte_pmu.h                | 202 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   7 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1054 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.34.1

#39 lore raw

PATCH v5 1/4 eal: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 app/test/meson.build                  |   1 +
 app/test/test_pmu.c                   |  41 +++
 doc/guides/prog_guide/profile_app.rst |   8 +
 lib/eal/common/meson.build            |   3 +
 lib/eal/common/pmu_private.h          |  41 +++
 lib/eal/common/rte_pmu.c              | 435 ++++++++++++++++++++++++++
 lib/eal/include/meson.build           |   1 +
 lib/eal/include/rte_pmu.h             | 199 ++++++++++++
 lib/eal/linux/eal.c                   |   4 +
 lib/eal/version.map                   |   6 +
 10 files changed, 739 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
app/test/meson.build +1 -0
index f34d19e3c3..93b3300309 100644
@@ -143,6 +143,7 @@ test_sources = files(
         'test_timer_racecond.c',
         'test_timer_secondary.c',
         'test_ticketlock.c',
+        'test_pmu.c',
         'test_trace.c',
         'test_trace_register.c',
         'test_trace_perf.c',
app/test/test_pmu.c new file +41 -0

41 lines not on this page · raw message

doc/guides/prog_guide/profile_app.rst +8 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
lib/eal/common/meson.build +3 -0
index 917758cc65..d6d05b56f3 100644
@@ -38,6 +38,9 @@ sources += files(
         'rte_service.c',
         'rte_version.c',
 )
+if is_linux
+    sources += files('rte_pmu.c')
+endif
 if is_linux or is_windows
     sources += files('eal_common_dynmem.c')
 endif
lib/eal/common/pmu_private.h new file +41 -0

41 lines not on this page · raw message

lib/eal/common/rte_pmu.c new file +435 -0

435 lines not on this page · raw message

lib/eal/include/meson.build +1 -0
index cfcd40aaed..3bf830adee 100644
@@ -36,6 +36,7 @@ headers += files(
         'rte_pci_dev_features.h',
         'rte_per_lcore.h',
         'rte_pflock.h',
+        'rte_pmu.h',
         'rte_random.h',
         'rte_reciprocal.h',
         'rte_seqcount.h',
lib/eal/include/rte_pmu.h new file +199 -0

199 lines not on this page · raw message

lib/eal/linux/eal.c +4 -0
index 8c118d0d9f..751a13b597 100644
@@ -53,6 +53,7 @@
 #include "eal_options.h"
 #include "eal_vfio.h"
 #include "hotplug_mp.h"
+#include "pmu_private.h"
 
 #define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
 
@@ -1206,6 +1207,8 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	eal_pmu_init();
+
 	if (rte_eal_tailqs_init() < 0) {
 		rte_eal_init_alert("Cannot init tail queues for objects");
 		rte_errno = EFAULT;
@@ -1372,6 +1375,7 @@ rte_eal_cleanup(void)
 	eal_bus_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
+	eal_pmu_fini();
 	/* after this point, any DPDK pointers will become dangling */
 	rte_eal_memory_detach();
 	eal_mp_dev_hotplug_cleanup();
lib/eal/version.map +6 -0
index 7ad12a7dc9..1717b221b4 100644
@@ -440,6 +440,11 @@ EXPERIMENTAL {
 	rte_thread_detach;
 	rte_thread_equal;
 	rte_thread_join;
+
+	# added in 23.03
+	rte_pmu; # WINDOWS_NO_EXPORT
+	rte_pmu_add_event; # WINDOWS_NO_EXPORT
+	rte_pmu_read; # WINDOWS_NO_EXPORT
 };
 
 INTERNAL {
@@ -483,4 +488,5 @@ INTERNAL {
 	rte_mem_map;
 	rte_mem_page_size;
 	rte_mem_unmap;
+	rte_pmu_enable_group; # WINDOWS_NO_EXPORT
 };
Signature · 1 line
2.34.1
#40 lore raw

RE: [PATCH v5 1/4] eal: add generic support for reading PMU events

Quoted 12 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 11 January 2023 00.47

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---

[...]

Quoted 19 lines
(quoted diff) +2 -0
+static int
+do_perf_event_open(uint64_t config[3], unsigned int lcore_id, int

group_fd)

lib/eal/common/rte_pmu.c new file +16 -0
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0,
Quoted 2 lines

rte_lcore_to_cpu_id(lcore_id), group_fd, 0);
+}

If SYS_perf_event_open() must be called from the worker thread itself, then lcore_id must not be passed as a parameter to do_perf_event_open(). Otherwise, I would expect to be able to call do_perf_event_open() from the main thread and pass any lcore_id of a worker thread. This comment applies to all functions that must be called from the worker thread itself. It also applies to the functions that call such functions.

[...]

Quoted 9 lines
lib/eal/include/rte_pmu.h new file +6 -0
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS];

/**< array of user pages */

(quoted diff) +1 -0
+	bool enabled; /**< true if group was enabled on particular lcore

*/

lib/eal/include/rte_pmu.h line 52

Quoted 10 lines
lib/eal/include/rte_pmu.h new file +10 -0
+};
+
+/**
+ * A structure describing an event.
+ */
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	unsigned int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */
+};

Move the "enabled" field up, making it the first field in this structure. This might reduce the number of instructions required to check (!group->enabled) in rte_pmu_read().

Also, each instance of the structure is used individually per lcore, so the structure should be cache line aligned to avoid unnecessarily crossing cache lines.

I.e.:

struct rte_pmu_event_group {
bool enabled; /**< true if group was enabled on particular lcore */
int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */
} __rte_cache_aligned;

Quoted 13 lines
lib/eal/include/rte_pmu.h new file +6 -0
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

(quoted diff) +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore

event group data */

(quoted diff) +1 -0
+	unsigned int num_group_events; /**< number of events in a group

*/

(quoted diff) +1 -0
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching

events */

lib/eal/include/rte_pmu.h line 65

Quoted 4 lines
lib/eal/include/rte_pmu.h new file +4 -0
+};
+
+/** Pointer to the PMU state container */
+extern struct rte_pmu rte_pmu;

Just "The PMU state container". It is not a pointer anymore. :-)

[...]

Quoted 15 lines
lib/eal/include/rte_pmu.h new file +15 -0
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc)
+{
+	uint64_t width, offset;

lib/eal/include/rte_pmu.h line 96

Quoted 10 lines
lib/eal/include/rte_pmu.h new file +10 -0
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+

Please add a comment here about the special meaning of index == 0.

Quoted 15 lines
lib/eal/include/rte_pmu.h new file +15 -0
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}

[...]

Quoted 18 lines

+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ * Index of an event to be read.
+ * @return
+ * Event value read from register. In case of errors or lack of
support
+ * 0 is returned. In other words, stream of zeros in a trace file
+ * indicates problem with reading particular PMU event register.
+ */

lib/eal/include/rte_pmu.h new file +4 -0
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{

lib/eal/include/rte_pmu.h line 168

Quoted 10 lines
lib/eal/include/rte_pmu.h new file +10 -0
+	struct rte_pmu_event_group *group;
+	int ret, lcore_id = rte_lcore_id();
+
+	group = &rte_pmu.group[lcore_id];
+	if (unlikely(!group->enabled)) {
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;

Group->enabled should be set inside rte_pmu_enable_group(), not here.

Quoted 7 lines
lib/eal/include/rte_pmu.h new file +7 -0
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return rte_pmu_read_userpage(group->mmap_pages[index]);
+}
#41 lore raw

RE: [PATCH v5 1/4] eal: add generic support for reading PMU events

Quoted 47 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Wednesday, January 11, 2023 10:06 AM
To: Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXT] RE: [PATCH v5 1/4] eal: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 12 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 11 January 2023 00.47

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---

[...]

Quoted 19 lines
(quoted diff) +2 -0
+static int
+do_perf_event_open(uint64_t config[3], unsigned int lcore_id, int

group_fd)

lib/eal/common/rte_pmu.c new file +16 -0
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0,
Quoted 9 lines
Quoted 2 lines

rte_lcore_to_cpu_id(lcore_id), group_fd, 0);
+}

If SYS_perf_event_open() must be called from the worker thread itself, then lcore_id must not be passed as a parameter to do_perf_event_open(). Otherwise, I would expect to be able to call do_perf_event_open() from the main thread and pass any lcore_id of a worker thread. This comment applies to all functions that must be called from the worker thread itself. It also applies to the functions that call such functions.

Lcore_id is being passed around so that we don't need to call rte_lcore_id() each and every time.

Quoted 15 lines

[...]

Quoted 13 lines
lib/eal/include/rte_pmu.h new file +6 -0
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS];

/**< array of user pages */

(quoted diff) +1 -0
+	bool enabled; /**< true if group was enabled on particular lcore

*/

lib/eal/include/rte_pmu.h new file +4 -0
+};
+
+/**
+ * A structure describing an event.
Quoted 9 lines

lib/eal/include/rte_pmu.h

Quoted 5 lines

+ */

lib/eal/include/rte_pmu.h new file +4 -0
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	unsigned int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */ };

Move the "enabled" field up, making it the first field in this structure. This might reduce the number of instructions required to check (!group->enabled) in rte_pmu_read().

This will be called once and no this will not produce more instructions. Why should it? In both cases compiler will need to load data at some offset and archs do have instructions for that.

Quoted 10 lines

Also, each instance of the structure is used individually per lcore, so the structure should be cache line aligned to avoid unnecessarily crossing cache lines.

I.e.:

struct rte_pmu_event_group {
bool enabled; /**< true if group was enabled on particular lcore */
int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS]; /**< array of user pages */ }
__rte_cache_aligned;

Yes, this can be aligned. While at it, I'd be more inclined to move mmap_pages up instead of enable.

Quoted 12 lines
Quoted 12 lines
lib/eal/include/rte_pmu.h new file +6 -0
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

(quoted diff) +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore

event group data */

(quoted diff) +1 -0
+	unsigned int num_group_events; /**< number of events in a group

*/

Quoted 9 lines

app/test/test_pmu.c

Quoted 6 lines
(quoted diff) +1 -0
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching

events */

app/test/test_pmu.c new file +4 -0
+};
+
+/** Pointer to the PMU state container */ extern struct rte_pmu
+rte_pmu;

Just "The PMU state container". It is not a pointer anymore. :-)

Good catch.

Quoted 20 lines

[...]

Quoted 18 lines
lib/eal/include/rte_pmu.h new file +18 -0
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
Quoted 8 lines

lib/eal/include/rte_pmu.h line 96

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+

Please add a comment here about the special meaning of index == 0.

Okay.

Quoted 41 lines
Quoted 16 lines
lib/eal/include/rte_pmu.h new file +15 -0
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}

[...]

Quoted 22 lines

+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an event.
+ *
+ * @param index
+ * Index of an event to be read.
+ * @return
+ * Event value read from register. In case of errors or lack of
support
+ * 0 is returned. In other words, stream of zeros in a trace file
+ * indicates problem with reading particular PMU event register.
+ */

lib/eal/include/rte_pmu.h new file +8 -0
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+	struct rte_pmu_event_group *group;
+	int ret, lcore_id = rte_lcore_id();
+
+	group = &rte_pmu.group[lcore_id];
Quoted 9 lines

lib/eal/include/rte_pmu.h line 168

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+	if (unlikely(!group->enabled)) {
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;

Group->enabled should be set inside rte_pmu_enable_group(), not here.

This is easier to follow imo and not against coding guidelines so I prefer to leave it as is.

Quoted 8 lines
Quoted 7 lines
lib/eal/include/rte_pmu.h new file +7 -0
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return rte_pmu_read_userpage(group->mmap_pages[index]);
+}
#42 lore raw

RE: [PATCH v5 1/4] eal: add generic support for reading PMU events

Quoted 48 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 11 January 2023 17.21

Quoted 33 lines

From: Morten Brørup <[email protected]>
Sent: Wednesday, January 11, 2023 10:06 AM

Quoted 12 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 11 January 2023 00.47

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---

[...]

Quoted 15 lines
(quoted diff) +2 -0
+static int
+do_perf_event_open(uint64_t config[3], unsigned int lcore_id, int

group_fd)

lib/eal/common/rte_pmu.c new file +12 -0
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(struct perf_event_attr),
+		.type = PERF_TYPE_RAW,
+		.exclude_kernel = 1,
+		.exclude_hv = 1,
+		.disabled = 1,
+	};
+
+	pmu_arch_fixup_config(config);
+
+	attr.config = config[0];
Quoted 8 lines
Quoted 6 lines
lib/eal/common/rte_pmu.c new file +4 -0
+	attr.config1 = config[1];
+	attr.config2 = config[2];
+
+	return syscall(SYS_perf_event_open, &attr, 0,

rte_lcore_to_cpu_id(lcore_id), group_fd, 0);
+}

If SYS_perf_event_open() must be called from the worker thread itself,

then lcore_id must not be

Quoted 1 line

passed as a parameter to do_perf_event_open(). Otherwise, I would

expect to be able to call

Quoted 1 line

do_perf_event_open() from the main thread and pass any lcore_id of a

Quoted 8 lines

worker thread.

Quoted 1 line

This comment applies to all functions that must be called from the

worker thread itself. It also

Quoted 2 lines

applies to the functions that call such functions.

Lcore_id is being passed around so that we don't need to call rte_lcore_id() each and every time.

Please take a look at the rte_lcore_id() implementation. :-)

Regardless, my argument still stands: If a function cannot be called with the lcore_id parameter set to any valid lcore id, it should not be a parameter to the function.

Quoted 23 lines
Quoted 15 lines

[...]

Quoted 12 lines
lib/eal/include/rte_pmu.h new file +6 -0
+/**
+ * A structure describing a group of events.
+ */
+struct rte_pmu_event_group {
+	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
+	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS];

/**< array of user pages */

(quoted diff) +1 -0
+	bool enabled; /**< true if group was enabled on particular lcore

*/

lib/eal/include/rte_pmu.h new file +3 -0
+};
+
+/**
Quoted 8 lines

lib/eal/include/rte_pmu.h

Quoted 6 lines

+ * A structure describing an event.
+ */

lib/eal/include/rte_pmu.h new file +4 -0
+struct rte_pmu_event {
+	char *name; /** name of an event */
+	unsigned int index; /** event index into fds/mmap_pages */
+	TAILQ_ENTRY(rte_pmu_event) next; /** list entry */ };

Move the "enabled" field up, making it the first field in this

Quoted 7 lines

structure. This might reduce the

Quoted 1 line

number of instructions required to check (!group->enabled) in

rte_pmu_read().

Quoted 1 line

This will be called once and no this will not produce more instructions. Why should it?

It seems I was not clearly describing my intention here here. rte_pmu_read() a hot function, where the comparison "if (!group->enabled)" itself will be executed many times.

Quoted 2 lines

In both cases compiler will need to load data at some offset and archs do have instructions for that.

Yes, the instructions are: address = BASE + sizeof(struct rte_pmu_event_group) * lcore_id + offsetof(struct rte_pmu_event, enabled).

I meant you could avoid the extra instructions stemming from the addition: "+ offsetof()". But you are right... Both BASE and offsetof(struct rte_pmu_event, enabled) are known in advance, and can be merged at compile time to avoid the addition.

Quoted 9 lines
Quoted 2 lines

Also, each instance of the structure is used individually per lcore,

so the structure should be

Quoted 6 lines

cache line aligned to avoid unnecessarily crossing cache lines.

I.e.:

struct rte_pmu_event_group {
bool enabled; /**< true if group was enabled on particular lcore

Quoted 8 lines

*/

Quoted 2 lines
	int fds[MAX_NUM_GROUP_EVENTS]; /**< array of event descriptors */
	struct perf_event_mmap_page *mmap_pages[MAX_NUM_GROUP_EVENTS];

/**< array of user pages */ }

Quoted 1 line

__rte_cache_aligned;

Yes, this can be aligned. While at it, I'd be more inclined to move mmap_pages up instead of enable.

Yes, moving up mmap_pages is better.

Quoted 108 lines
Quoted 13 lines
Quoted 13 lines
lib/eal/include/rte_pmu.h new file +6 -0
+
+/**
+ * A PMU state container.
+ */
+struct rte_pmu {
+	char *name; /** name of core PMU listed under

/sys/bus/event_source/devices */

(quoted diff) +1 -0
+	struct rte_pmu_event_group group[RTE_MAX_LCORE]; /**< per lcore

event group data */

(quoted diff) +1 -0
+	unsigned int num_group_events; /**< number of events in a group

*/

Quoted 9 lines

app/test/test_pmu.c

Quoted 6 lines
(quoted diff) +1 -0
+	TAILQ_HEAD(, rte_pmu_event) event_list; /**< list of matching

events */

app/test/test_pmu.c new file +4 -0
+};
+
+/** Pointer to the PMU state container */ extern struct rte_pmu
+rte_pmu;

Just "The PMU state container". It is not a pointer anymore. :-)

Good catch.

Quoted 20 lines

[...]

Quoted 18 lines
lib/eal/include/rte_pmu.h new file +18 -0
+/**
+ * @internal
+ *
+ * Read PMU counter.
+ *
+ * @param pc
+ *   Pointer to the mmapped user page.
+ * @return
+ *   Counter value read from hardware.
+ */
+__rte_internal
+static __rte_always_inline uint64_t
+rte_pmu_read_userpage(struct perf_event_mmap_page *pc) {
+	uint64_t width, offset;
+	uint32_t seq, index;
+	int64_t pmc;
+
+	for (;;) {
Quoted 8 lines

lib/eal/include/rte_pmu.h line 96

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+		seq = pc->lock;
+		rte_compiler_barrier();
+		index = pc->index;
+		offset = pc->offset;
+		width = pc->pmc_width;
+

Please add a comment here about the special meaning of index == 0.

Okay.

Quoted 19 lines
Quoted 16 lines
lib/eal/include/rte_pmu.h new file +15 -0
+		if (likely(pc->cap_user_rdpmc && index)) {
+			pmc = rte_pmu_pmc_read(index - 1);
+			pmc <<= 64 - width;
+			pmc >>= 64 - width;
+			offset += pmc;
+		}
+
+		rte_compiler_barrier();
+
+		if (likely(pc->lock == seq))
+			return offset;
+	}
+
+	return 0;
+}

[...]

Quoted 5 lines
Quoted 5 lines

+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Read hardware counter configured to count occurrences of an

event.

Quoted 17 lines
Quoted 17 lines

+ *
+ * @param index
+ * Index of an event to be read.
+ * @return
+ * Event value read from register. In case of errors or lack of
support
+ * 0 is returned. In other words, stream of zeros in a trace file
+ * indicates problem with reading particular PMU event register.
+ */

lib/eal/include/rte_pmu.h new file +8 -0
+__rte_experimental
+static __rte_always_inline uint64_t
+rte_pmu_read(unsigned int index)
+{
+	struct rte_pmu_event_group *group;
+	int ret, lcore_id = rte_lcore_id();
+
+	group = &rte_pmu.group[lcore_id];
Quoted 9 lines

lib/eal/include/rte_pmu.h line 168

Quoted 6 lines
lib/eal/include/rte_pmu.h new file +6 -0
+	if (unlikely(!group->enabled)) {
+		ret = rte_pmu_enable_group(lcore_id);
+		if (ret)
+			return 0;
+
+		group->enabled = true;

Group->enabled should be set inside rte_pmu_enable_group(), not here.

Quoted 2 lines

This is easier to follow imo and not against coding guidelines so I prefer to leave it as is.

OK. It makes the rte_pmu_read() source code slightly shorter, but probably has zero effect on the generated code. No strong preference - feel free to follow your personal preference on this.

Quoted 10 lines
Quoted 9 lines
Quoted 8 lines
lib/eal/include/rte_pmu.h new file +7 -0
+	}
+
+	if (unlikely(index >= rte_pmu.num_group_events))
+		return 0;
+
+	return rte_pmu_read_userpage(group->mmap_pages[index]);
+}
#43 lore raw

PATCH v5 2/4 eal/arm: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c               |   4 ++
 lib/eal/arm/include/meson.build   |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h |  39 +++++++++++
 lib/eal/arm/meson.build           |   4 ++
 lib/eal/arm/rte_pmu.c             | 104 ++++++++++++++++++++++++++++++
 lib/eal/include/rte_pmu.h         |   3 +
 6 files changed, 155 insertions(+)
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
app/test/test_pmu.c +4 -0
index 9a90aaffdb..e19819c31a 100644
@@ -13,6 +13,10 @@ test_pmu_read(void)
 	int tries = 10;
 	int event = -1;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/eal/arm/include/meson.build +1 -0
index 657bf58569..ab13b0220a 100644
@@ -20,6 +20,7 @@ arch_headers = files(
         'rte_pause_32.h',
         'rte_pause_64.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch_32.h',
         'rte_prefetch_64.h',
lib/eal/arm/include/rte_pmu_pmc.h new file +39 -0
new file mode 100644
index 0000000000..729f3d4dfe
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_ARM_H_
+#define _RTE_PMU_PMC_ARM_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_ARM_H_ */
lib/eal/arm/meson.build +4 -0
index dca1106aae..0c5575b197 100644
@@ -9,3 +9,7 @@ sources += files(
         'rte_hypervisor.c',
         'rte_power_intrinsics.c',
 )
+
+if is_linux
+    sources += files('rte_pmu.c')
+endif
lib/eal/arm/rte_pmu.c new file +104 -0

104 lines not on this page · raw message

lib/eal/include/rte_pmu.h +3 -0
index 6968b35545..9185d05ca3 100644
@@ -20,6 +20,9 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
+#if defined(RTE_ARCH_ARM64)
+#include <rte_pmu_pmc.h>
+#endif
 
 /**
  * @file
Signature · 1 line
2.34.1
#44 lore raw

PATCH v5 3/4 eal/x86: support reading Intel PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel PMU events in runtime.

---
 app/test/test_pmu.c               |  2 ++
 lib/eal/include/rte_pmu.h         |  2 +-
 lib/eal/x86/include/meson.build   |  1 +
 lib/eal/x86/include/rte_pmu_pmc.h | 33 +++++++++++++++++++++++++++++++
 4 files changed, 37 insertions(+), 1 deletion(-)
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h
app/test/test_pmu.c +2 -0
index e19819c31a..79f83a1925 100644
@@ -15,6 +15,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/eal/include/rte_pmu.h +1 -1
index 9185d05ca3..0345746940 100644
@@ -20,7 +20,7 @@ extern "C" {
 #include <rte_branch_prediction.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
-#if defined(RTE_ARCH_ARM64)
+#if defined(RTE_ARCH_ARM64) || defined(RTE_ARCH_X86_64)
 #include <rte_pmu_pmc.h>
 #endif
 
lib/eal/x86/include/meson.build +1 -0
index 52d2f8e969..03d286ed25 100644
@@ -9,6 +9,7 @@ arch_headers = files(
         'rte_io.h',
         'rte_memcpy.h',
         'rte_pause.h',
+        'rte_pmu_pmc.h',
         'rte_power_intrinsics.h',
         'rte_prefetch.h',
         'rte_rtm.h',
lib/eal/x86/include/rte_pmu_pmc.h new file +33 -0
new file mode 100644
index 0000000000..f241b80bc9
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+
+#ifndef _RTE_PMU_PMC_X86_H_
+#define _RTE_PMU_PMC_X86_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t low, high;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return low | (high << 32);
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMU_PMC_X86_H_ */
Signature · 1 line
2.34.1
#45 lore raw

PATCH v5 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               |  4 ++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 +++++++++++
 lib/eal/common/eal_common_trace_points.c |  3 +
 lib/eal/common/rte_pmu.c                 | 73 +++++++++++++++++++++++-
 lib/eal/include/rte_eal_trace.h          | 10 ++++
 lib/eal/version.map                      |  1 +
 7 files changed, 126 insertions(+), 2 deletions(-)
app/test/test_trace_perf.c +4 -0
index 46ae7d8074..4851b6852f 100644
@@ -114,6 +114,8 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +124,7 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+WORKER_DEFINE(READ_PMU)
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +177,7 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index a8b501fe0c..6a53341c6b 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..10d5b99084 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='.*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace_points.c +3 -0
index 0b0b254615..de918ca618 100644
@@ -75,3 +75,6 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
lib/eal/common/rte_pmu.c +71 -2

111 lines not on this page · raw message

lib/eal/include/rte_eal_trace.h +10 -0
index 5ef4398230..9b35af75d5 100644
@@ -17,6 +17,7 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#include <rte_pmu.h>
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +280,15 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+/* PMU */
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(unsigned int index),
+	uint64_t val;
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/version.map +1 -0
index 1717b221b4..d87a867e5b 100644
@@ -442,6 +442,7 @@ EXPERIMENTAL {
 	rte_thread_join;
 
 	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 	rte_pmu; # WINDOWS_NO_EXPORT
 	rte_pmu_add_event; # WINDOWS_NO_EXPORT
 	rte_pmu_read; # WINDOWS_NO_EXPORT
Signature · 1 line
2.34.1
#46 lore raw
In reply to Tomasz Duszynski

hi,

don't interpret this as an objection to the functionality but this looks like a clear example of something that doesn't belong in the EAL. has there been a discussion as to whether or not this should be in a separate library?

a basic test is whether or not an implementation exists or can be reasonably provided for all platforms and that isn't strictly evident here. red flag is to see yet more code being added conditionally compiled for a single platform.

Morten, Bruce comments?

thanks

Quoted 69 lines · Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there v4:
- fix freeing mem detected by debug_autotest v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 +++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 104 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 504 +++++++++++++++++++++++
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  10 +
 lib/eal/include/rte_pmu.h                | 202 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   7 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1054 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.34.1

#47 lore raw
Quoted 57 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Wednesday, 11 January 2023 01.32

Quoted 3 lines · Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs

on

Quoted 4 lines

isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to

be

Quoted 31 lines

stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there v4:
- fix freeing mem detected by debug_autotest v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

Quoted 13 lines
 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 +++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 104 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 504
Quoted 19 lines

+++++++++++++++++++++++

Quoted 18 lines
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  10 +
 lib/eal/include/rte_pmu.h                | 202 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   7 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1054 insertions(+)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h
 create mode 100644 lib/eal/arm/rte_pmu.c
 create mode 100644 lib/eal/common/pmu_private.h
 create mode 100644 lib/eal/common/rte_pmu.c
 create mode 100644 lib/eal/include/rte_pmu.h
 create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.34.1

[Moved Tyler's post down here.]

Quoted 8 lines

hi,

don't interpret this as an objection to the functionality but this looks
like a clear example of something that doesn't belong in the EAL. has there been a discussion as to whether or not this should be in a separate library?

IIRC, there has been no such discussion.

Although I agree that this doesn't belong in EAL, I would point to the trace library as a reference for allowing it into the EAL.

For the records, I also oppose to the trace library being part of the EAL.

On the other hand, it would be interesting to determine if it is *impossible* adding this functionality as any other normal DPDK library, i.e. outside of the EAL, or if there is an unavoidable tie-in to the EAL.

@Tomasz, if this is impossible, please describe the unavoidable tie-in to the EAL. No need for a long report, just a few words. You (and this functionality) shouldn't suffer from our long term ambition to move stuff out of the EAL.

Quoted 5 lines

a basic test is whether or not an implementation exists or can be reasonably provided for all platforms and that isn't strictly evident here. red flag is to see yet more code being added conditionally compiled for a single platform.

Another basic test: Can DPDK applications run without it? If they can, an Environment Abstraction Layer does not need to have it, and thus it does not need to be part of the EAL.

Quoted 4 lines

Morten, Bruce comments?

thanks

#48 lore raw
Quoted 109 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Wednesday, January 11, 2023 10:31 AM
To: Tyler Retzlaff <[email protected]>; Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXT] RE: [PATCH v5 0/4] add support for self monitoring

External Email

----------------------------------------------------------------------

Quoted 58 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Wednesday, 11 January 2023 01.32

Quoted 4 lines · Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs

on

Quoted 4 lines

isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to

be

Quoted 31 lines

stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there v4:
- fix freeing mem detected by debug_autotest v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

Quoted 13 lines
 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 +++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 104 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 504
Quoted 19 lines

+++++++++++++++++++++++

Quoted 18 lines
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  10 +
 lib/eal/include/rte_pmu.h                | 202 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   7 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1054 insertions(+)  create mode 100644

app/test/test_pmu.c create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h create mode 100644 lib/eal/arm/rte_pmu.c create mode 100644 lib/eal/common/pmu_private.h create mode 100644 lib/eal/common/rte_pmu.c create mode 100644 lib/eal/include/rte_pmu.h create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.34.1

[Moved Tyler's post down here.]

Quoted 7 lines

hi,

don't interpret this as an objection to the functionality but this looks like a clear example of something that doesn't belong in the EAL. has there been a discussion as to whether or not this should be in a separate library?

IIRC, there has been no such discussion.

Although I agree that this doesn't belong in EAL, I would point to the trace library as a reference for allowing it into the EAL.

For the records, I also oppose to the trace library being part of the EAL.

Quoted 8 lines

On the other hand, it would be interesting to determine if it is *impossible* adding this functionality as any other normal DPDK library, i.e. outside of the EAL, or if there is an unavoidable tie-in to the EAL.

@Tomasz, if this is impossible, please describe the unavoidable tie-in to the EAL. No need for a long report, just a few words. You (and this functionality) shouldn't suffer from our long term ambition to move stuff out of the EAL.

You can read about rationale here https://lore.kernel.org/dpdk-dev/DM4PR18MB436872EBC5922084C5DAFC1DD2FC9@DM4PR18MB4368.namprd18.prod.outlook.com/#t

As for the NO-NO there isn't any in fact. There are some tradeoffs though.

For example, seems eal cannot depend on other libs so if someone needs to finetune some part of EAL for whatever reason, then relevant part needs to modified each and every time. I.e specific includes and trcepoints need to be added each time.

On the other hand, if this is coupled with eal then adding tracepoints to some parts will be easier. Or they can just be added to specific points and live there.

No strong opinions besides that. I'd like to know what others think.

Quoted 13 lines
Quoted 5 lines

a basic test is whether or not an implementation exists or can be reasonably provided for all platforms and that isn't strictly evident here. red flag is to see yet more code being added conditionally compiled for a single platform.

Another basic test: Can DPDK applications run without it? If they can, an Environment Abstraction Layer does not need to have it, and thus it does not need to be part of the EAL.

Quoted 4 lines

Morten, Bruce comments?

thanks

#49 lore raw
Quoted 118 lines · Tomasz Duszynski
Quoted 109 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Wednesday, January 11, 2023 10:31 AM
To: Tyler Retzlaff <[email protected]>; Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXT] RE: [PATCH v5 0/4] add support for self monitoring

External Email

----------------------------------------------------------------------

Quoted 58 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Wednesday, 11 January 2023 01.32

Quoted 4 lines · Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs

on

Quoted 4 lines

isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to

be

Quoted 31 lines

stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there v4:
- fix freeing mem detected by debug_autotest v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

Quoted 13 lines
 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 +++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 104 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 504
Quoted 19 lines

+++++++++++++++++++++++

Quoted 18 lines
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  10 +
 lib/eal/include/rte_pmu.h                | 202 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   7 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1054 insertions(+)  create mode 100644

app/test/test_pmu.c create mode 100644 lib/eal/arm/include/rte_pmu_pmc.h create mode 100644 lib/eal/arm/rte_pmu.c create mode 100644 lib/eal/common/pmu_private.h create mode 100644 lib/eal/common/rte_pmu.c create mode 100644 lib/eal/include/rte_pmu.h create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.34.1

[Moved Tyler's post down here.]

Quoted 7 lines

hi,

don't interpret this as an objection to the functionality but this looks like a clear example of something that doesn't belong in the EAL. has there been a discussion as to whether or not this should be in a separate library?

IIRC, there has been no such discussion.

Although I agree that this doesn't belong in EAL, I would point to the trace library as a reference for allowing it into the EAL.

For the records, I also oppose to the trace library being part of the EAL.

Quoted 8 lines

On the other hand, it would be interesting to determine if it is *impossible* adding this functionality as any other normal DPDK library, i.e. outside of the EAL, or if there is an unavoidable tie-in to the EAL.

@Tomasz, if this is impossible, please describe the unavoidable tie-in to the EAL. No need for a long report, just a few words. You (and this functionality) shouldn't suffer from our long term ambition to move stuff out of the EAL.

Quoted 8 lines

You can read about rationale here https://lore.kernel.org/dpdk-dev/DM4PR18MB436872EBC5922084C5DAFC1DD2FC9@DM4PR18MB4368.namprd18.prod.outlook.com/#t

As for the NO-NO there isn't any in fact. There are some tradeoffs though.

For example, seems eal cannot depend on other libs so if someone needs to finetune some part of EAL for whatever reason, then relevant part needs to modified each and every time. I.e specific includes and trcepoints need to be added each time.

Well, EAL can depend on other libs, but then those libs cannot in turn directly depend upon DPDK. This is where breaking out first some of the smaller widely used parts of DPDK e.g. logging, would be good, as it would then in turn allow other, potentially bigger parts of EAL to be taken out.

See [1] for a rough first attempt at this, which allows simlification of telemetry as it no longer needs a "dependency injection" style to have logging. Moving out logging would also allow logging from kvargs library too - another lib which is used by EAL rather than depending on it. Similarly for tracing functionality - if that were pulled out of EAL, it could be used by telemetry, kvargs and any other parts removed from EAL.

/Bruce

[1] http://patches.dpdk.org/project/dpdk/list/?series=24453

#50 lore raw
In reply to Tyler Retzlaff

Hi Tyler,

Quoted 8 lines

-----Original Message-----
From: Tyler Retzlaff <[email protected]>
Sent: Wednesday, January 11, 2023 1:32 AM
To: Tomasz Duszynski <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v5 0/4] add support for self monitoring

Quoted 8 lines

External Email

----------------------------------------------------------------------
hi,

don't interpret this as an objection to the functionality but this looks like a clear example of something that doesn't belong in the EAL. has there been a discussion as to whether or not this should be in a separate library?

No, I don't recall anybody having any concerns about the code placement. Rationale behind making this part of eal was based on the fact that tracing itself is a part of eal and since this was meant to be extension to tracing, code placement decision came out naturally.

During development phase idea evolved a bit and what initially was supposed to be solely yet another tracepoint become generic API to read pmu and tracepoint based on that. Which means both can be used independently.

That said, since this code has both platform agnostic and platform specific parts this can either be split into:
1. library + eal platform code
2. all under eal

Either approach seems legit. Thoughts?

Quoted 4 lines

a basic test is whether or not an implementation exists or can be reasonably provided for all platforms and that isn't strictly evident here. red flag is to see yet more code being added conditionally compiled for a single platform.

Even libs are not entirely pristine and have platform specific ifdefs lurking so not sure where this red flag is coming from.

Quoted 74 lines

Morten, Bruce comments?

thanks

Quoted 68 lines · Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there v4:
- fix freeing mem detected by debug_autotest v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
eal: add generic support for reading PMU events eal/arm: support reading ARM PMU events in runtime eal/x86: support reading Intel PMU events in runtime
eal: add PMU support to tracing library

 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  47 +++
 app/test/test_trace_perf.c               |   4 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 lib/eal/arm/include/meson.build          |   1 +
 lib/eal/arm/include/rte_pmu_pmc.h        |  39 ++
 lib/eal/arm/meson.build                  |   4 +
 lib/eal/arm/rte_pmu.c                    | 104 +++++
 lib/eal/common/eal_common_trace_points.c |   3 +
 lib/eal/common/meson.build               |   3 +
 lib/eal/common/pmu_private.h             |  41 ++
 lib/eal/common/rte_pmu.c                 | 504 +++++++++++++++++++++++
 lib/eal/include/meson.build              |   1 +
 lib/eal/include/rte_eal_trace.h          |  10 +
 lib/eal/include/rte_pmu.h                | 202 +++++++++
 lib/eal/linux/eal.c                      |   4 +
 lib/eal/version.map                      |   7 +
 lib/eal/x86/include/meson.build          |   1 +
 lib/eal/x86/include/rte_pmu_pmc.h        |  33 ++
 20 files changed, 1054 insertions(+)
 create mode 100644 app/test/test_pmu.c  create mode 100644

lib/eal/arm/include/rte_pmu_pmc.h create mode 100644 lib/eal/arm/rte_pmu.c create mode 100644 lib/eal/common/pmu_private.h create mode 100644 lib/eal/common/rte_pmu.c create mode 100644 lib/eal/include/rte_pmu.h create mode 100644 lib/eal/x86/include/rte_pmu_pmc.h

--
2.34.1

#51 lore raw
Quoted 34 lines · Tomasz Duszynski

Hi Tyler,

Quoted 8 lines

-----Original Message-----
From: Tyler Retzlaff <[email protected]>
Sent: Wednesday, January 11, 2023 1:32 AM
To: Tomasz Duszynski <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v5 0/4] add support for self monitoring

Quoted 8 lines

External Email

----------------------------------------------------------------------
hi,

don't interpret this as an objection to the functionality but this looks like a clear example of something that doesn't belong in the EAL. has there been a discussion as to whether or not this should be in a separate library?

No, I don't recall anybody having any concerns about the code placement. Rationale behind making this part of eal was based on the fact that tracing itself is a part of eal and since this was meant to be extension to tracing, code placement decision came out naturally.

During development phase idea evolved a bit and what initially was supposed to be solely yet another tracepoint become generic API to read pmu and tracepoint based on that. Which means both can be used independently.

That said, since this code has both platform agnostic and platform specific parts this can either be split into:
1. library + eal platform code
2. all under eal

Either approach seems legit. Thoughts?

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

a basic test is whether or not an implementation exists or can be reasonably provided for all platforms and that isn't strictly evident here. red flag is to see yet more code being added conditionally compiled for a single platform.

Even libs are not entirely pristine and have platform specific ifdefs lurking so not sure where this red flag is coming from.

i think red flag was probably the wrong term to use sorry for that. rather i should say it is an indicator that the api probably doesn't belong in the eal.

fundamentally the purpose of the abstraction library is to relieve the application from having to do conditional compilation and/or execution for the subject apis coming from eal. including and exporting apis that work for only one platform is in direct contradiction.

please explore adding this as a separate library, it is understood that there are tradeoffs involved.

thanks!

#52 lore raw
Quoted 61 lines

-----Original Message-----
From: Tyler Retzlaff <[email protected]>
Sent: Wednesday, January 11, 2023 10:06 PM
To: Tomasz Duszynski <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v5 0/4] add support for self monitoring

Quoted 18 lines · Tomasz Duszynski

Hi Tyler,

Quoted 16 lines

-----Original Message-----
From: Tyler Retzlaff <[email protected]>
Sent: Wednesday, January 11, 2023 1:32 AM
To: Tomasz Duszynski <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v5 0/4] add support for self monitoring

External Email

---------------------------------------------------------------------
-
hi,

Quoted 7 lines
Quoted 3 lines

don't interpret this as an objection to the functionality but this
looks like a clear example of something that doesn't belong in the
EAL. has there been a discussion as to whether or not this should be in a separate library?

No, I don't recall anybody having any concerns about the code
placement. Rationale behind making this part of eal was based on the
fact that tracing itself is a part of eal and since this was meant to be extension to tracing,

code placement decision came out naturally.

Quoted 6 lines

During development phase idea evolved a bit and what initially was supposed to be solely yet another tracepoint become generic API to read pmu and tracepoint based on that. Which means both can be used independently.

That said, since this code has both platform agnostic and platform specific parts this can either

be split into:

Quoted 6 lines

1. library + eal platform code
2. all under eal

Either approach seems legit. Thoughts?

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

a basic test is whether or not an implementation exists or can be
reasonably provided for all platforms and that isn't strictly evident
here. red flag is to see yet more code being added conditionally compiled for a single platform.

Even libs are not entirely pristine and have platform specific ifdefs lurking so not sure where this red flag is coming from.

i think red flag was probably the wrong term to use sorry for that. rather i should say it is an indicator that the api probably doesn't belong in the eal.

Quoted 8 lines

fundamentally the purpose of the abstraction library is to relieve the application from having to do conditional compilation and/or execution for the subject apis coming from eal. including and exporting apis that work for only one platform is in direct contradiction.

please explore adding this as a separate library, it is understood that there are tradeoffs involved.

thanks!

Any ideas how to name the library?

#53 lore raw
Quoted 65 lines · Tomasz Duszynski
Quoted 65 lines

-----Original Message-----
From: Tyler Retzlaff <[email protected]>
Sent: Wednesday, January 11, 2023 10:06 PM
To: Tomasz Duszynski <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v5 0/4] add support for self monitoring

Quoted 18 lines · Tomasz Duszynski

Hi Tyler,

Quoted 16 lines

-----Original Message-----
From: Tyler Retzlaff <[email protected]>
Sent: Wednesday, January 11, 2023 1:32 AM
To: Tomasz Duszynski <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v5 0/4] add support for self monitoring

External Email

---------------------------------------------------------------------
-
hi,

Quoted 7 lines
Quoted 3 lines

don't interpret this as an objection to the functionality but this
looks like a clear example of something that doesn't belong in the
EAL. has there been a discussion as to whether or not this should be in a separate library?

No, I don't recall anybody having any concerns about the code
placement. Rationale behind making this part of eal was based on the
fact that tracing itself is a part of eal and since this was meant to be extension to tracing,

code placement decision came out naturally.

Quoted 6 lines

During development phase idea evolved a bit and what initially was supposed to be solely yet another tracepoint become generic API to read pmu and tracepoint based on that. Which means both can be used independently.

That said, since this code has both platform agnostic and platform specific parts this can either

be split into:

Quoted 6 lines

1. library + eal platform code
2. all under eal

Either approach seems legit. Thoughts?

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

a basic test is whether or not an implementation exists or can be
reasonably provided for all platforms and that isn't strictly evident
here. red flag is to see yet more code being added conditionally compiled for a single platform.

Even libs are not entirely pristine and have platform specific ifdefs lurking so not sure where this red flag is coming from.

i think red flag was probably the wrong term to use sorry for that. rather i should say it is an indicator that the api probably doesn't belong in the eal.

fundamentally the purpose of the abstraction library is to relieve the application from having to do conditional compilation and/or execution for the subject apis coming from eal. including and exporting apis that work for only one platform is in direct contradiction.

Quoted 6 lines
Quoted 4 lines

please explore adding this as a separate library, it is understood that there are tradeoffs involved.

thanks!

Any ideas how to name the library?

naming is always so hard and i'm definitely not authoritative.

it seems like lib/pmu would be the least churn against your existing patch series, here are some other suggestions that might work.

lib/pmu (measuring unit)
lib/pmc (measuring counters)
lib/pcq (counter query)
lib/pmq (measuring query)

#54 lore raw
Quoted 70 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Friday, 13 January 2023 20.22

Quoted 5 lines · Tomasz Duszynski
Quoted 5 lines

From: Tyler Retzlaff <[email protected]>
Sent: Wednesday, January 11, 2023 10:06 PM

On Wed, Jan 11, 2023 at 09:39:35AM +0000, Tomasz Duszynski wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Hi Tyler,

Quoted 6 lines

From: Tyler Retzlaff <[email protected]>
Sent: Wednesday, January 11, 2023 1:32 AM

hi,

don't interpret this as an objection to the functionality but

this

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

looks like a clear example of something that doesn't belong in

the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

EAL. has there been a discussion as to whether or not this should

be in a separate library?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

No, I don't recall anybody having any concerns about the code placement. Rationale behind making this part of eal was based on

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

fact that tracing itself is a part of eal and since this was meant

to be extension to tracing,

Quoted 5 lines
Quoted 5 lines

code placement decision came out naturally.

Quoted 4 lines

During development phase idea evolved a bit and what initially was supposed to be solely yet another tracepoint become generic API to read pmu and tracepoint based on that. Which means both can be

used independently.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

That said, since this code has both platform agnostic and platform

specific parts this can either

Quoted 9 lines
Quoted 9 lines

be split into:

Quoted 8 lines

1. library + eal platform code
2. all under eal

Either approach seems legit. Thoughts?

Quoted 3 lines

a basic test is whether or not an implementation exists or can be reasonably provided for all platforms and that isn't strictly

evident

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

here. red flag is to see yet more code being added conditionally

compiled for a single platform.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Even libs are not entirely pristine and have platform specific

ifdefs

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

lurking so not sure where this red flag is coming from.

i think red flag was probably the wrong term to use sorry for that. rather i should say it is an indicator that the api probably doesn't

belong in the eal.

Quoted 2 lines
Quoted 2 lines

fundamentally the purpose of the abstraction library is to relieve

the application from having to

Quoted 1 line
Quoted 1 line

do conditional compilation and/or execution for the subject apis

coming from eal. including and

Quoted 1 line
Quoted 1 line

exporting apis that work for only one platform is in direct

contradiction.

Quoted 2 lines
Quoted 2 lines

please explore adding this as a separate library, it is understood

that there are tradeoffs

Quoted 2 lines
Quoted 2 lines

involved.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

thanks!

Any ideas how to name the library?

naming is always so hard and i'm definitely not authoritative.

it seems like lib/pmu would be the least churn against your existing patch series, here are some other suggestions that might work.

+1 to lib/pmu

Less work, as Tyler already mentioned. Furthermore:

Both Intel and ARM use the term Performance Monitoring Unit (abbreviated PMU).

Microsoft does too [1].

[1]: https://learn.microsoft.com/en-us/windows-hardware/test/wpt/recording-pmu-events

RISC-V uses the term Hardware Performance Monitor (abbreviated HPM). I haven't checked other CPU vendors.

#55 lore raw

v6 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there v4:
- fix freeing mem detected by debug_autotest v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library

 MAINTAINERS                              |   5 +
 app/test/meson.build                     |   4 +
 app/test/test_pmu.c                      |  48 +++
 app/test/test_trace_perf.c               |  10 +
 doc/api/doxy-api-index.md                |   3 +-
 doc/api/doxy-api.conf.in                 |   1 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 doc/guides/rel_notes/release_23_03.rst   |   7 +
 lib/eal/common/eal_common_trace.c        |  13 +-
 lib/eal/common/eal_common_trace_points.c |   5 +
 lib/eal/include/rte_eal_trace.h          |  13 +
 lib/eal/meson.build                      |   3 +
 lib/eal/version.map                      |   3 +
 lib/meson.build                          |   1 +
 lib/pmu/meson.build                      |  21 +
 lib/pmu/pmu_arm64.c                      |  94 +++++
 lib/pmu/pmu_private.h                    |  29 ++
 lib/pmu/rte_pmu.c                        | 497 +++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 226 +++++++++++
 lib/pmu/rte_pmu_pmc_arm64.h              |  30 ++
 lib/pmu/rte_pmu_pmc_x86_64.h             |  24 ++
 lib/pmu/version.map                      |  20 +
 23 files changed, 1100 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
 create mode 100644 lib/pmu/version.map

--
2.34.1

#56 lore raw

PATCH v6 1/4 lib: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   4 +
 app/test/test_pmu.c                    |  42 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   7 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 436 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 206 ++++++++++++
 lib/pmu/version.map                    |  19 ++
 13 files changed, 773 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
index 9a0f416d2e..9f13eafd95 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build +4 -0
index f34d19e3c3..b2c2a618b1 100644
@@ -360,6 +360,10 @@ if dpdk_conf.has('RTE_LIB_METRICS')
     test_sources += ['test_metrics.c']
     fast_tests += [['metrics_autotest', true, true]]
 endif
+if is_linux
+    test_sources += ['test_pmu.c']
+    fast_tests += [['pmu_autotest', true, true]]
+endif
 if not is_windows and dpdk_conf.has('RTE_LIB_TELEMETRY')
     test_sources += ['test_telemetry_json.c', 'test_telemetry_data.c']
     fast_tests += [['telemetry_json_autotest', true, true]]
app/test/test_pmu.c new file +42 -0

42 lines not on this page · raw message

doc/api/doxy-api-index.md +2 -1
index de488c7abf..7f1938f92f 100644
@@ -222,7 +222,8 @@ The public API headers are grouped by topics:
   [log](@ref rte_log.h),
   [errno](@ref rte_errno.h),
   [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),
+  [pmu](@ref rte_pmu.h)
 
 - **misc**:
   [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
index f0886c3bd1..920e615996 100644
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
                           @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +8 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
doc/guides/rel_notes/release_23_03.rst +7 -0
index 92ec1e4b88..f43bd62376 100644
@@ -57,6 +57,13 @@ New Features
 
 * **Added multi-process support for axgbe PMD.**
 
+* **Added PMU library.**
+
+  Added a new PMU (performance measurement unit) library which allows applications
+  to perform self monitoring activities without depending on external utilities like perf.
+  After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters
+  can be stored in CTF format for further analysis.
+
 
 Removed Items
 -------------
lib/meson.build +1 -0
index a90fee31b7..7132131b5c 100644
@@ -11,6 +11,7 @@
 libraries = [
         'kvargs', # eal depends on kvargs
         'telemetry', # basic info querying
+        'pmu',
         'eal', # everything depends on eal
         'ring',
         'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
new file mode 100644
index 0000000000..a4160b494e
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2023 Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +29 -0
new file mode 100644
index 0000000000..849549b125
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +436 -0

436 lines not on this page · raw message

lib/pmu/rte_pmu.h new file +206 -0

206 lines not on this page · raw message

lib/pmu/version.map new file +19 -0
new file mode 100644
index 0000000000..e15e21156a
@@ -0,0 +1,19 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
+
+INTERNAL {
+	global:
+
+	rte_pmu_enable_group;
+};
Signature · 1 line
2.34.1
#57 lore raw

RE: [PATCH v6 1/4] lib: add generic support for reading PMU events

Quoted 6 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Friday, 20 January 2023 00.39

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

Quoted 6 lines

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---

If you insist on passing lcore_id around as a function parameter, the function description must mention that the lcore_id parameter must be set to rte_lcore_id() for the functions where this is a requirement, including all functions that use those functions.

Alternatively, follow my previous suggestion: Omit the lcore_id function parameter, and use rte_lcore_id() instead.

#58 lore raw

RE: [PATCH v6 1/4] lib: add generic support for reading PMU events

Quoted 22 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Friday, January 20, 2023 10:47 AM
To: Tomasz Duszynski <[email protected]>; [email protected]; Thomas Monjalon <[email protected]>
Cc: Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] RE: [PATCH v6 1/4] lib: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 10 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Friday, 20 January 2023 00.39

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Quoted 7 lines
Quoted 2 lines

---

If you insist on passing lcore_id around as a function parameter, the function description must mention that the lcore_id parameter must be set to rte_lcore_id() for the functions where this is a requirement, including all functions that use those functions.

Not sure why are you insisting so much on removing that rte_lcore_id(). Yes that macro evaluates to integer but if you don't think about internals this resembles a function call.

Then natural pattern is to call it once and reuse results if possible. Passing lcore_id around implies that calls are per l-core, why would that confuse anyone reading that code?

Besides, all functions taking it are internal stuff hence you cannot call it elsewhere.

Quoted 3 lines

Alternatively, follow my previous suggestion: Omit the lcore_id function parameter, and use rte_lcore_id() instead.

#59 lore raw

RE: [PATCH v6 1/4] lib: add generic support for reading PMU events

Quoted 16 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Thursday, 26 January 2023 10.40

Quoted 13 lines

From: Morten Brørup <[email protected]>
Sent: Friday, January 20, 2023 10:47 AM

Quoted 10 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Friday, 20 January 2023 00.39

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

---

If you insist on passing lcore_id around as a function parameter, the

function description must

Quoted 1 line

mention that the lcore_id parameter must be set to rte_lcore_id() for

the functions where this is a

Quoted 1 line

requirement, including all functions that use those functions.

Perhaps I'm stating this wrong, so let me try to rephrase:

As I understand it, some of the setup functions must be called from the EAL thread that executes that function - due to some syscall (SYS_perf_event_open) needing to be called from the thread itself.

Those functions should not take an lcore_id parameter. Otherwise, I would expect to be able to call those functions from e.g. the main thread and pass the lcore_id of any EAL thread as a parameter, which you at the bottom of this email [1] explained is not possible.

[1]: http://inbox.dpdk.org/dev/DM4PR18MB4368461EC42603F77A7DC1BCD2E09@DM4PR18MB4368.namprd18.prod.outlook.com/

Quoted 6 lines
Quoted 1 line

Not sure why are you insisting so much on removing that rte_lcore_id(). Yes that macro evaluates
to integer but if you don't think about internals this resembles a function call.

I agree with this argument. And for that reason, passing lcore_id around could be relevant.

I only wanted to bring your attention to the low cost of fetching it inside the functions, as an alternative to passing it as an argument.

Quoted 2 lines

Then natural pattern is to call it once and reuse results if possible.

Yes, and I would usually agree to using this pattern.

Quoted 3 lines

Passing lcore_id around
implies that calls are per l-core, why would that confuse anyone reading that code?

This is where I disagree: Passing lcore_id as a parameter to a function does NOT imply that the function is running on that lcore!

E.g rte_mempool_default_cache(struct rte_mempool *mp, unsigned lcore_id) [2] takes lcore_id as a parameter, and does not assume that lcore_id==rte_lcore_id().

[2]: https://elixir.bootlin.com/dpdk/latest/source/lib/mempool/rte_mempool.h#L1315

Quoted 3 lines

Besides, all functions taking it are internal stuff hence you cannot call it elsewhere.

OK. I agree that this reduces the risk of incorrect use.

Generally, I think that internal functions should be documented too. Not to the full extent, like public functions, but some documentation is nice.

And if there are special requirements to a function parameter, it should be documented with that function. Requiring that the lcore_id parameter must be == rte_lcore_id() is certainly a special requirement.

It might just be me worrying too much, so... If nobody else complains about this, I can live with it as is. Assuming that none of the public functions have this special requirement (either directly or indirectly, by calling functions with the special requirement).

Quoted 6 lines
Quoted 2 lines

Alternatively, follow my previous suggestion: Omit the lcore_id

function parameter, and use

Quoted 2 lines

rte_lcore_id() instead.

#60 lore raw

Re: [PATCH v6 1/4] lib: add generic support for reading PMU events

Quoted 63 lines · Morten Brørup
Quoted 16 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Thursday, 26 January 2023 10.40

Quoted 13 lines

From: Morten Brørup <[email protected]>
Sent: Friday, January 20, 2023 10:47 AM

Quoted 10 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Friday, 20 January 2023 00.39

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

---

If you insist on passing lcore_id around as a function parameter, the

function description must

Quoted 1 line

mention that the lcore_id parameter must be set to rte_lcore_id() for

the functions where this is a

Quoted 1 line

requirement, including all functions that use those functions.

Perhaps I'm stating this wrong, so let me try to rephrase:

As I understand it, some of the setup functions must be called from the EAL thread that executes that function - due to some syscall (SYS_perf_event_open) needing to be called from the thread itself.

Those functions should not take an lcore_id parameter. Otherwise, I would expect to be able to call those functions from e.g. the main thread and pass the lcore_id of any EAL thread as a parameter, which you at the bottom of this email [1] explained is not possible.

[1]: http://inbox.dpdk.org/dev/DM4PR18MB4368461EC42603F77A7DC1BCD2E09@DM4PR18MB4368.namprd18.prod.outlook.com/

Quoted 6 lines
Quoted 1 line

Not sure why are you insisting so much on removing that rte_lcore_id(). Yes that macro evaluates
to integer but if you don't think about internals this resembles a function call.

I agree with this argument. And for that reason, passing lcore_id around could be relevant.

I only wanted to bring your attention to the low cost of fetching it inside the functions, as an alternative to passing it as an argument.

Quoted 2 lines

Then natural pattern is to call it once and reuse results if possible.

Yes, and I would usually agree to using this pattern.

Quoted 3 lines

Passing lcore_id around
implies that calls are per l-core, why would that confuse anyone reading that code?

This is where I disagree: Passing lcore_id as a parameter to a function does NOT imply that the function is running on that lcore!

E.g rte_mempool_default_cache(struct rte_mempool *mp, unsigned lcore_id) [2] takes lcore_id as a parameter, and does not assume that lcore_id==rte_lcore_id().

[2]: https://elixir.bootlin.com/dpdk/latest/source/lib/mempool/rte_mempool.h#L1315

Quoted 3 lines

Besides, all functions taking it are internal stuff hence you cannot call it elsewhere.

Quoted 8 lines

OK. I agree that this reduces the risk of incorrect use.

Generally, I think that internal functions should be documented too. Not to the full extent, like public functions, but some documentation is nice.

And if there are special requirements to a function parameter, it should be documented with that function. Requiring that the lcore_id parameter must be == rte_lcore_id() is certainly a special requirement.

It might just be me worrying too much, so... If nobody else complains about this, I can live with it as is. Assuming that none of the public functions have this special requirement (either directly or indirectly, by calling functions with the special requirement).

I would tend to agree with you Morten. If the lcore_id parameter to the function must be rte_lcore_id(), then I think it's error prone to have that as an explicit parameter, and that the function should always get the core id itself.

Other possible complication is - how does this work with threads that are not pinned to a particular physical core? Do things work as expected in that case?

/Bruce

#61 lore raw

RE: [EXT] Re: [PATCH v6 1/4] lib: add generic support for reading PMU events

Quoted 110 lines

-----Original Message-----
From: Bruce Richardson <[email protected]>
Sent: Thursday, January 26, 2023 1:59 PM
To: Morten Brørup <[email protected]>
Cc: Tomasz Duszynski <[email protected]>; [email protected]; Thomas Monjalon <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v6 1/4] lib: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 22 lines · Morten Brørup
Quoted 22 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Thursday, 26 January 2023 10.40

Quoted 13 lines

From: Morten Brørup <[email protected]>
Sent: Friday, January 20, 2023 10:47 AM

Quoted 10 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Friday, 20 January 2023 00.39

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Quoted 5 lines
Quoted 2 lines

---

If you insist on passing lcore_id around as a function parameter, the

function description must

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

mention that the lcore_id parameter must be set to rte_lcore_id() for

the functions where this is a

Quoted 1 line

requirement, including all functions that use those functions.

Perhaps I'm stating this wrong, so let me try to rephrase:

As I understand it, some of the setup functions must be called from the EAL thread that executes

that function - due to some syscall (SYS_perf_event_open) needing to be called from the thread itself.

Quoted 2 lines

Those functions should not take an lcore_id parameter. Otherwise, I would expect to be able to

call those functions from e.g. the main thread and pass the lcore_id of any EAL thread as a parameter, which you at the bottom of this email [1] explained is not possible.

Quoted 10 lines

[1]:
https://urldefense.proofpoint.com/v2/url?u=http-3A__inbox.dpdk.org_dev
_DM4PR18MB4368461EC42603F77A7DC1BCD2E09-40DM4PR18MB4368.namprd18.prod. outlook.com_&d=DwIDAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNXgrbjdlXxVEEGYkxIx RndyEUwWU_ad5ce22YI6Is&m=wEvFmuH_S_EhAgRZQTC7z3pQ1Sr_cEsbFAXxgE2Fi2ESd 4sMgg-tgVOVDepp-JYO&s=wU4g1LLV4EHyRYpj2inWOK8MDcUKq7txrZ7RXZhUM2I&e=

Quoted 2 lines
Quoted 1 line
Quoted 8 lines
Quoted 4 lines

Not sure why are you insisting so much on removing that rte_lcore_id(). Yes that macro evaluates
to integer but if you don't think about internals this resembles a function call.

I agree with this argument. And for that reason, passing lcore_id around could be relevant.

I only wanted to bring your attention to the low cost of fetching it inside the functions, as an

alternative to passing it as an argument.

Quoted 11 lines
Quoted 3 lines

Then natural pattern is to call it once and reuse results if possible.

Yes, and I would usually agree to using this pattern.

Quoted 3 lines

Passing lcore_id around
implies that calls are per l-core, why would that confuse anyone reading that code?

This is where I disagree: Passing lcore_id as a parameter to a function does NOT imply that the

function is running on that lcore!

Quoted 2 lines

E.g rte_mempool_default_cache(struct rte_mempool *mp, unsigned lcore_id) [2] takes lcore_id as a

parameter, and does not assume that lcore_id==rte_lcore_id().

Quoted 9 lines

[2]:
https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.co
m_dpdk_latest_source_lib_mempool_rte-5Fmempool.h-23L1315&d=DwIDAw&c=nK jWec2b6R0mOyPaz7xtfQ&r=PZNXgrbjdlXxVEEGYkxIxRndyEUwWU_ad5ce22YI6Is&m=w EvFmuH_S_EhAgRZQTC7z3pQ1Sr_cEsbFAXxgE2Fi2ESd4sMgg-tgVOVDepp-JYO&s=Ayyj pEtATWUHfWnGMn5j2XDLMjgxxJTh5gQV0m77z5Q&e=

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

Besides, all functions taking it are internal stuff hence you cannot call it elsewhere.

OK. I agree that this reduces the risk of incorrect use.

Generally, I think that internal functions should be documented too. Not to the full extent, like

public functions, but some documentation is nice.

Quoted 2 lines

And if there are special requirements to a function parameter, it should be documented with that

function. Requiring that the lcore_id parameter must be == rte_lcore_id() is certainly a special requirement.

Quoted 2 lines

It might just be me worrying too much, so... If nobody else complains about this, I can live with

it as is. Assuming that none of the public functions have this special requirement (either directly or indirectly, by calling functions with the special requirement).

Quoted 1 line
Quoted 7 lines

I would tend to agree with you Morten. If the lcore_id parameter to the function must be rte_lcore_id(), then I think it's error prone to have that as an explicit parameter, and that the function should always get the core id itself.

Other possible complication is - how does this work with threads that are not pinned to a particular physical core? Do things work as expected in that case?

It's assumed that once set of counters is enabled on particular l-core then this thread shouldn't be migrating back and for the obvious reasons.

But, once scheduled elsewhere all should still work as expected.

Quoted 1 line

/Bruce

#62 lore raw

RE: [EXT] Re: [PATCH v6 1/4] lib: add generic support for reading PMU events

Quoted 8 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Thursday, 26 January 2023 16.28

Quoted 4 lines

From: Bruce Richardson <[email protected]>
Sent: Thursday, January 26, 2023 1:59 PM

Other possible complication is - how does this work with threads that

are not pinned to a

Quoted 8 lines
Quoted 2 lines

particular physical core? Do things work as expected in that case?

It's assumed that once set of counters is enabled on particular l-core then this thread shouldn't be migrating
back and for the obvious reasons.

But, once scheduled elsewhere all should still work as expected.

Just to elaborate what Tomasz stated here...

The patch contains this line (code comments are mine):

return syscall(SYS_perf_event_open, &attr, /*pid*/ 0, /*cpu*/ -1, group_fd, 0);

And man 2 perf_event_open [1] says:

pid == 0 and cpu == -1:
This measures the calling process/thread on any CPU.

So it should work just fine, even when a thread is not pinned to a particular physical core.

[1]: https://man7.org/linux/man-pages/man2/perf_event_open.2.html

#63 lore raw

RE: [PATCH v6 1/4] lib: add generic support for reading PMU events

In reply to Morten Brørup
Quoted 74 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Thursday, January 26, 2023 1:30 PM
To: Tomasz Duszynski <[email protected]>; [email protected]; Thomas Monjalon <[email protected]>
Cc: Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] RE: [PATCH v6 1/4] lib: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 16 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Thursday, 26 January 2023 10.40

Quoted 13 lines

From: Morten Brørup <[email protected]>
Sent: Friday, January 20, 2023 10:47 AM

Quoted 10 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Friday, 20 January 2023 00.39

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

---

If you insist on passing lcore_id around as a function parameter, the

function description must

Quoted 1 line

mention that the lcore_id parameter must be set to rte_lcore_id() for

the functions where this is a

Quoted 1 line

requirement, including all functions that use those functions.

Perhaps I'm stating this wrong, so let me try to rephrase:

As I understand it, some of the setup functions must be called from the EAL thread that executes that function - due to some syscall (SYS_perf_event_open) needing to be called from the thread itself.

Those functions should not take an lcore_id parameter. Otherwise, I would expect to be able to call those functions from e.g. the main thread and pass the lcore_id of any EAL thread as a parameter, which you at the bottom of this email [1] explained is not possible.

[1]: https://urldefense.proofpoint.com/v2/url?u=http-
3A__inbox.dpdk.org_dev_DM4PR18MB4368461EC42603F77A7DC1BCD2E09- 40DM4PR18MB4368.namprd18.prod.outlook.com_&d=DwIFAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNXgrbjdlXxVEEGYkxI xRndyEUwWU_ad5ce22YI6Is&m=QkMcmM2epUOCdRd6xI5o3d2nQaqruy0GvOQUgbn75cLlzobEVMwLBUiGXADiuvVz&s=5K9oM8 e7u52C_0_5xtWIKl31aXRHhJDKoQTDQp5EHWY&e=

Quoted 6 lines
Quoted 1 line

Not sure why are you insisting so much on removing that rte_lcore_id(). Yes that macro evaluates
to integer but if you don't think about internals this resembles a function call.

I agree with this argument. And for that reason, passing lcore_id around could be relevant.

I only wanted to bring your attention to the low cost of fetching it inside the functions, as an alternative to passing it as an argument.

Quoted 2 lines

Then natural pattern is to call it once and reuse results if possible.

Yes, and I would usually agree to using this pattern.

Quoted 3 lines

Passing lcore_id around
implies that calls are per l-core, why would that confuse anyone reading that code?

Quoted 6 lines

This is where I disagree: Passing lcore_id as a parameter to a function does NOT imply that the function is running on that lcore!

E.g rte_mempool_default_cache(struct rte_mempool *mp, unsigned lcore_id) [2] takes lcore_id as a parameter, and does not assume that lcore_id==rte_lcore_id().

Oh, now I got your point!

Okay then, if this is going to cause confusion because of misleading self-documenting code I'll change that.

Quoted 29 lines

[2]: https://urldefense.proofpoint.com/v2/url?u=https-
3A__elixir.bootlin.com_dpdk_latest_source_lib_mempool_rte-5Fmempool.h- 23L1315&d=DwIFAw&c=nKjWec2b6R0mOyPaz7xtfQ&r=PZNXgrbjdlXxVEEGYkxIxRndyEUwWU_ad5ce22YI6Is&m=QkMcmM2ep UOCdRd6xI5o3d2nQaqruy0GvOQUgbn75cLlzobEVMwLBUiGXADiuvVz&s=4pnL_TZcVhj476u19ybcn2Rbad6OTb3k2U- nhFvhZ0k&e=

Quoted 3 lines

Besides, all functions taking it are internal stuff hence you cannot call it elsewhere.

OK. I agree that this reduces the risk of incorrect use.

Generally, I think that internal functions should be documented too. Not to the full extent, like public functions, but some documentation is nice.

And if there are special requirements to a function parameter, it should be documented with that function. Requiring that the lcore_id parameter must be == rte_lcore_id() is certainly a special requirement.

It might just be me worrying too much, so... If nobody else complains about this, I can live with it as is. Assuming that none of the public functions have this special requirement (either directly or indirectly, by calling functions with the special requirement).

Quoted 6 lines
Quoted 2 lines

Alternatively, follow my previous suggestion: Omit the lcore_id

function parameter, and use

Quoted 2 lines

rte_lcore_id() instead.

#64 lore raw

Re: [PATCH v6 1/4] lib: add generic support for reading PMU events

In reply to Tomasz Duszynski
Quoted 51 lines · Tomasz Duszynski

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   4 +
 app/test/test_pmu.c                    |  42 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   7 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 436 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 206 ++++++++++++
 lib/pmu/version.map                    |  19 ++
 13 files changed, 773 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
index 9a0f416d2e..9f13eafd95 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build +0 -0
index f34d19e3c3..b2c2a618b1 100644

app/test/meson.build if dpdk_conf.has('RTE_LIB_METRICS') · line 366

Quoted 7 lines
app/test/meson.build +4 -0
@@ -360,6 +360,10 @@ if dpdk_conf.has('RTE_LIB_METRICS')
     test_sources += ['test_metrics.c']
     fast_tests += [['metrics_autotest', true, true]]
 endif
+if is_linux
+    test_sources += ['test_pmu.c']
+    fast_tests += [['pmu_autotest', true, true]]
+endif

traditionally we don't conditionally include tests at the meson.build level, instead we run all tests and have them skip when executed for unsupported exec environments.

you can take a look at test_eventdev.c as an example for a test that is skipped on windows, i'm sure it could be adapted to skip on freebsd if you aren't supporting it.

#65 lore raw

RE: [EXT] Re: [PATCH v6 1/4] lib: add generic support for reading PMU events

Quoted 73 lines

-----Original Message-----
From: Tyler Retzlaff <[email protected]>
Sent: Friday, January 20, 2023 7:30 PM
To: Tomasz Duszynski <[email protected]>
Cc: [email protected]; Thomas Monjalon <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v6 1/4] lib: add generic support for reading PMU events

External Email

----------------------------------------------------------------------

Quoted 49 lines · Tomasz Duszynski

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   4 +
 app/test/test_pmu.c                    |  42 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   7 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 436 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 206 ++++++++++++
 lib/pmu/version.map                    |  19 ++
 13 files changed, 773 insertions(+), 1 deletion(-)  create mode

100644 app/test/test_pmu.c create mode 100644 lib/pmu/meson.build create mode 100644 lib/pmu/pmu_private.h create mode 100644 lib/pmu/rte_pmu.c create mode 100644 lib/pmu/rte_pmu.h create mode 100644 lib/pmu/version.map

MAINTAINERS index 9a0f416d2e..9f13eafd95 renamed from MAINTAINERS +0 -0

100644

MAINTAINERS +5 -0
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build index renamed from app/test/meson.build +0 -0

f34d19e3c3..b2c2a618b1 100644

app/test/meson.build if dpdk_conf.has('RTE_LIB_METRICS') · line 364

Quoted 5 lines
app/test/meson.build +3 -0
@@ -360,6 +360,10 @@ if dpdk_conf.has('RTE_LIB_METRICS')
     test_sources += ['test_metrics.c']
     fast_tests += [['metrics_autotest', true, true]]  endif
+if is_linux
+    test_sources += ['test_pmu.c']
+    fast_tests += [['pmu_autotest', true, true]] endif

traditionally we don't conditionally include tests at the meson.build level, instead we run all tests and have them skip when executed for unsupported exec environments.

Quoted 2 lines

you can take a look at test_eventdev.c as an example for a test that is skipped on windows, i'm sure it could be adapted to skip on freebsd if you aren't supporting it.

Right, this looks better. Thanks for pointing this out.

#66 lore raw

PATCH v6 2/4 pmu: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c         |  4 ++
 lib/pmu/meson.build         |  7 +++
 lib/pmu/pmu_arm64.c         | 94 +++++++++++++++++++++++++++++++++++++
 lib/pmu/rte_pmu.h           |  4 ++
 lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++
 5 files changed, 139 insertions(+)
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
app/test/test_pmu.c +4 -0
index 7c3cf18ed9..4cdc71791e 100644
@@ -15,6 +15,10 @@ test_pmu_read(void)
 	if (rte_pmu_init() < 0)
 		return TEST_FAILED;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/pmu/meson.build +7 -0
index a4160b494e..e857681137 100644
@@ -11,3 +11,10 @@ includes = [global_inc]
 
 sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
+indirect_headers += files(
+        'rte_pmu_pmc_arm64.h',
+)
+
+if dpdk_conf.has('RTE_ARCH_ARM64')
+    sources += files('pmu_arm64.c')
+endif
lib/pmu/pmu_arm64.c new file +94 -0

94 lines not on this page · raw message

lib/pmu/rte_pmu.h +4 -0
index 42c764fa9e..4808d90eb9 100644
@@ -26,6 +26,10 @@ extern "C" {
 #include <rte_compat.h>
 #include <rte_lcore.h>
 
+#if defined(RTE_ARCH_ARM64)
+#include "rte_pmu_pmc_arm64.h"
+#endif
+
 /** Maximum number of events in a group */
 #define MAX_NUM_GROUP_EVENTS 8
 
lib/pmu/rte_pmu_pmc_arm64.h new file +30 -0
new file mode 100644
index 0000000000..10648f0c5f
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_ARM64_H_
+#define _RTE_PMU_PMC_ARM64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_ARM64_H_ */
Signature · 1 line
2.34.1
#67 lore raw

PATCH v6 3/4 pmu: support reading Intel x86_64 PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel x86_64 PMU events in runtime.

---
 app/test/test_pmu.c          |  2 ++
 lib/pmu/meson.build          |  1 +
 lib/pmu/rte_pmu.h            |  2 ++
 lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++++++++++++++++++++++++
 4 files changed, 29 insertions(+)
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
app/test/test_pmu.c +2 -0
index 4cdc71791e..dc7a9cdb27 100644
@@ -17,6 +17,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/pmu/meson.build +1 -0
index e857681137..5b92e5c4e3 100644
@@ -13,6 +13,7 @@ sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
 indirect_headers += files(
         'rte_pmu_pmc_arm64.h',
+        'rte_pmu_pmc_x86_64.h',
 )
 
 if dpdk_conf.has('RTE_ARCH_ARM64')
lib/pmu/rte_pmu.h +2 -0
index 4808d90eb9..617732361c 100644
@@ -28,6 +28,8 @@ extern "C" {
 
 #if defined(RTE_ARCH_ARM64)
 #include "rte_pmu_pmc_arm64.h"
+#elif defined(RTE_ARCH_X86_64)
+#include "rte_pmu_pmc_x86_64.h"
 #endif
 
 /** Maximum number of events in a group */
lib/pmu/rte_pmu_pmc_x86_64.h new file +24 -0
new file mode 100644
index 0000000000..7b67466960
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_X86_64_H_
+#define _RTE_PMU_PMC_X86_64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t low, high;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return low | (high << 32);
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_X86_64_H_ */
Signature · 1 line
2.34.1
#68 lore raw

PATCH v6 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               | 10 ++++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 +++++++++++++
 lib/eal/common/eal_common_trace.c        | 13 ++++-
 lib/eal/common/eal_common_trace_points.c |  5 ++
 lib/eal/include/rte_eal_trace.h          | 13 +++++
 lib/eal/meson.build                      |  3 ++
 lib/eal/version.map                      |  3 ++
 lib/pmu/rte_pmu.c                        | 61 ++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 14 ++++++
 lib/pmu/version.map                      |  1 +
 11 files changed, 159 insertions(+), 1 deletion(-)
app/test/test_trace_perf.c +10 -0
index 46ae7d8074..f1929f2734 100644
@@ -114,6 +114,10 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+#ifdef RTE_EXEC_ENV_LINUX
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
+#endif
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +126,9 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+#ifdef RTE_EXEC_ENV_LINUX
+WORKER_DEFINE(READ_PMU)
+#endif
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +181,9 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+#ifdef RTE_EXEC_ENV_LINUX
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
+#endif
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index a8b501fe0c..6a53341c6b 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..a8e97ee1ec 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86-64 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='.*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace.c +12 -1
index 5caaac8e59..3631d0032b 100644
@@ -11,6 +11,9 @@
 #include <rte_errno.h>
 #include <rte_lcore.h>
 #include <rte_per_lcore.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_string_fns.h>
 
 #include "eal_trace.h"
@@ -71,8 +74,13 @@ eal_trace_init(void)
 		goto free_meta;
 
 	/* Apply global configurations */
-	STAILQ_FOREACH(arg, &trace.args, next)
+	STAILQ_FOREACH(arg, &trace.args, next) {
 		trace_args_apply(arg->val);
+#ifdef RTE_EXEC_ENV_LINUX
+		if (rte_pmu_init() == 0)
+			rte_pmu_add_events_by_pattern(arg->val);
+#endif
+	}
 
 	rte_trace_mode_set(trace.mode);
 
@@ -88,6 +96,9 @@ eal_trace_init(void)
 void
 eal_trace_fini(void)
 {
+#ifdef RTE_EXEC_ENV_LINUX
+	rte_pmu_fini();
+#endif
 	trace_mem_free();
 	trace_metadata_destroy();
 	eal_trace_args_free();
lib/eal/common/eal_common_trace_points.c +5 -0
index 0b0b254615..1e46ce549a 100644
@@ -75,3 +75,8 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
+#endif
lib/eal/include/rte_eal_trace.h +13 -0
index 5ef4398230..afb459b198 100644
@@ -17,6 +17,9 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +282,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(unsigned int index),
+	uint64_t val;
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+#endif
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/meson.build +3 -0
index 056beb9461..f5865dbcd9 100644
@@ -26,6 +26,9 @@ deps += ['kvargs']
 if not is_windows
     deps += ['telemetry']
 endif
+if is_linux
+    deps += ['pmu']
+endif
 if dpdk_conf.has('RTE_USE_LIBBSD')
     ext_deps += libbsd
 endif
lib/eal/version.map +3 -0
index 7ad12a7dc9..eddb45bebf 100644
@@ -440,6 +440,9 @@ EXPERIMENTAL {
 	rte_thread_detach;
 	rte_thread_equal;
 	rte_thread_join;
+
+	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 };
 
 INTERNAL {
lib/pmu/rte_pmu.c +61 -0

67 lines not on this page · raw message

lib/pmu/rte_pmu.h +14 -0
index 617732361c..f642b721e8 100644
@@ -166,6 +166,20 @@ __rte_experimental
 int
 rte_pmu_add_event(const char *name);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add events matching pattern to the group of enabled events.
+ *
+ * @param pattern
+ *   Pattern e=ev1[,ev2,...] matching events, where evX is a placeholder for an event listed under
+ *   /sys/bus/event_source/devices/pmu/events.
+ */
+__rte_experimental
+int
+rte_pmu_add_events_by_pattern(const char *pattern);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice
lib/pmu/version.map +1 -0
index e15e21156a..4646eefd2b 100644
@@ -7,6 +7,7 @@ EXPERIMENTAL {
 
 	rte_pmu;
 	rte_pmu_add_event;
+	rte_pmu_add_events_by_pattern;
 	rte_pmu_fini;
 	rte_pmu_init;
 	rte_pmu_read;
Signature · 1 line
2.34.1
#69 lore raw

v7 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v7:
- use per-lcore event group instead of global table index by lcore-id
- don't add pmu_autotest to fast tests because due to lack of suported on every arch
v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there
v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library

 MAINTAINERS                              |   5 +
 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  61 +++
 app/test/test_trace_perf.c               |  10 +
 doc/api/doxy-api-index.md                |   3 +-
 doc/api/doxy-api.conf.in                 |   1 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 doc/guides/rel_notes/release_23_03.rst   |   7 +
 lib/eal/common/eal_common_trace.c        |  13 +-
 lib/eal/common/eal_common_trace_points.c |   5 +
 lib/eal/include/rte_eal_trace.h          |  13 +
 lib/eal/meson.build                      |   3 +
 lib/eal/version.map                      |   3 +
 lib/meson.build                          |   1 +
 lib/pmu/meson.build                      |  21 +
 lib/pmu/pmu_arm64.c                      |  94 ++++
 lib/pmu/pmu_private.h                    |  29 ++
 lib/pmu/rte_pmu.c                        | 525 +++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 225 ++++++++++
 lib/pmu/rte_pmu_pmc_arm64.h              |  30 ++
 lib/pmu/rte_pmu_pmc_x86_64.h             |  24 ++
 lib/pmu/version.map                      |  21 +
 23 files changed, 1138 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
 create mode 100644 lib/pmu/version.map

--
2.34.1

#70 lore raw

PATCH v7 1/4 lib: add generic support for reading PMU events

Add support for programming PMU counters and reading their values in runtime bypassing kernel completely.

This is especially useful in cases where CPU cores are isolated (nohz_full) i.e run dedicated tasks. In such cases one cannot use standard perf utility without sacrificing latency and performance.

---
 MAINTAINERS                            |   5 +
 app/test/meson.build                   |   1 +
 app/test/test_pmu.c                    |  55 +++
 doc/api/doxy-api-index.md              |   3 +-
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/profile_app.rst  |   8 +
 doc/guides/rel_notes/release_23_03.rst |   7 +
 lib/meson.build                        |   1 +
 lib/pmu/meson.build                    |  13 +
 lib/pmu/pmu_private.h                  |  29 ++
 lib/pmu/rte_pmu.c                      | 464 +++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                      | 205 +++++++++++
 lib/pmu/version.map                    |  20 ++
 13 files changed, 811 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/version.map
MAINTAINERS +5 -0
index 9a0f416d2e..9f13eafd95 100644
@@ -1697,6 +1697,11 @@ M: Nithin Dabilpuram <[email protected]>
 M: Pavan Nikhilesh <[email protected]>
 F: lib/node/
 
+PMU - EXPERIMENTAL
+M: Tomasz Duszynski <[email protected]>
+F: lib/pmu/
+F: app/test/test_pmu*
+
 
 Test Applications
 -----------------
app/test/meson.build +1 -0
index f34d19e3c3..7b6b69dcf1 100644
@@ -111,6 +111,7 @@ test_sources = files(
         'test_reciprocal_division_perf.c',
         'test_red.c',
         'test_pie.c',
+        'test_pmu.c',
         'test_reorder.c',
         'test_rib.c',
         'test_rib6.c',
app/test/test_pmu.c new file +55 -0

55 lines not on this page · raw message

doc/api/doxy-api-index.md +2 -1
index de488c7abf..7f1938f92f 100644
@@ -222,7 +222,8 @@ The public API headers are grouped by topics:
   [log](@ref rte_log.h),
   [errno](@ref rte_errno.h),
   [trace](@ref rte_trace.h),
-  [trace_point](@ref rte_trace_point.h)
+  [trace_point](@ref rte_trace_point.h),
+  [pmu](@ref rte_pmu.h)
 
 - **misc**:
   [EAL config](@ref rte_eal.h),
doc/api/doxy-api.conf.in +1 -0
index f0886c3bd1..920e615996 100644
@@ -63,6 +63,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
+                          @TOPDIR@/lib/pmu \
                           @TOPDIR@/lib/port \
                           @TOPDIR@/lib/power \
                           @TOPDIR@/lib/rawdev \
doc/guides/prog_guide/profile_app.rst +8 -0
index 14292d4c25..a8b501fe0c 100644
@@ -7,6 +7,14 @@ Profile Your Application
 The following sections describe methods of profiling DPDK applications on
 different architectures.
 
+Performance counter based profiling
+-----------------------------------
+
+Majority of architectures support some sort hardware measurement unit which provides a set of
+programmable counters that monitor specific events. There are different tools which can gather
+that information, perf being an example here. Though in some scenarios, eg. when CPU cores are
+isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
+read specific events directly from application via ``rte_pmu_read()``.
 
 Profiling on x86
 ----------------
doc/guides/rel_notes/release_23_03.rst +7 -0
index 84b112a8b1..7e6062022a 100644
@@ -55,6 +55,13 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added PMU library.**
+
+  Added a new PMU (performance measurement unit) library which allows applications
+  to perform self monitoring activities without depending on external utilities like perf.
+  After integration with :doc:`../prog_guide/trace_lib` data gathered from hardware counters
+  can be stored in CTF format for further analysis.
+
 * **Added multi-process support for axgbe PMD.**
 
 * **Updated Corigine nfp driver.**
lib/meson.build +1 -0
index a90fee31b7..7132131b5c 100644
@@ -11,6 +11,7 @@
 libraries = [
         'kvargs', # eal depends on kvargs
         'telemetry', # basic info querying
+        'pmu',
         'eal', # everything depends on eal
         'ring',
         'rcu', # rcu depends on ring
lib/pmu/meson.build new file +13 -0
new file mode 100644
index 0000000000..a4160b494e
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2023 Marvell International Ltd.
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+includes = [global_inc]
+
+sources = files('rte_pmu.c')
+headers = files('rte_pmu.h')
lib/pmu/pmu_private.h new file +29 -0
new file mode 100644
index 0000000000..849549b125
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell
+ */
+
+#ifndef _PMU_PRIVATE_H_
+#define _PMU_PRIVATE_H_
+
+/**
+ * Architecture specific PMU init callback.
+ *
+ * @return
+ *   0 in case of success, negative value otherwise.
+ */
+int
+pmu_arch_init(void);
+
+/**
+ * Architecture specific PMU cleanup callback.
+ */
+void
+pmu_arch_fini(void);
+
+/**
+ * Apply architecture specific settings to config before passing it to syscall.
+ */
+void
+pmu_arch_fixup_config(uint64_t config[3]);
+
+#endif /* _PMU_PRIVATE_H_ */
lib/pmu/rte_pmu.c new file +464 -0

464 lines not on this page · raw message

lib/pmu/rte_pmu.h new file +205 -0

205 lines not on this page · raw message

lib/pmu/version.map new file +20 -0
new file mode 100644
index 0000000000..50fb0f354e
@@ -0,0 +1,20 @@
+DPDK_23 {
+	local: *;
+};
+
+EXPERIMENTAL {
+	global:
+
+	per_lcore__event_group;
+	rte_pmu;
+	rte_pmu_add_event;
+	rte_pmu_fini;
+	rte_pmu_init;
+	rte_pmu_read;
+};
+
+INTERNAL {
+	global:
+
+	rte_pmu_enable_group;
+};
Signature · 1 line
2.34.1
#71 lore raw

PATCH v7 2/4 pmu: support reading ARM PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading ARM PMU events in runtime.

---
 app/test/test_pmu.c         |  4 ++
 lib/pmu/meson.build         |  7 +++
 lib/pmu/pmu_arm64.c         | 94 +++++++++++++++++++++++++++++++++++++
 lib/pmu/rte_pmu.h           |  4 ++
 lib/pmu/rte_pmu_pmc_arm64.h | 30 ++++++++++++
 5 files changed, 139 insertions(+)
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
app/test/test_pmu.c +4 -0
index b30db35724..c53a1bc2f1 100644
@@ -26,6 +26,10 @@ test_pmu_read(void)
 	if (rte_pmu_init() < 0)
 		return TEST_FAILED;
 
+#if defined(RTE_ARCH_ARM64)
+	event = rte_pmu_add_event("cpu_cycles");
+#endif
+
 	while (tries--)
 		val += rte_pmu_read(event);
 
lib/pmu/meson.build +7 -0
index a4160b494e..e857681137 100644
@@ -11,3 +11,10 @@ includes = [global_inc]
 
 sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
+indirect_headers += files(
+        'rte_pmu_pmc_arm64.h',
+)
+
+if dpdk_conf.has('RTE_ARCH_ARM64')
+    sources += files('pmu_arm64.c')
+endif
lib/pmu/pmu_arm64.c new file +94 -0

94 lines not on this page · raw message

lib/pmu/rte_pmu.h +4 -0
index e360375a0c..b18938dab1 100644
@@ -26,6 +26,10 @@ extern "C" {
 #include <rte_compat.h>
 #include <rte_spinlock.h>
 
+#if defined(RTE_ARCH_ARM64)
+#include "rte_pmu_pmc_arm64.h"
+#endif
+
 /** Maximum number of events in a group */
 #define MAX_NUM_GROUP_EVENTS 8
 
lib/pmu/rte_pmu_pmc_arm64.h new file +30 -0
new file mode 100644
index 0000000000..10648f0c5f
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_ARM64_H_
+#define _RTE_PMU_PMC_ARM64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t val;
+
+	if (index == 31) {
+		/* CPU Cycles (0x11) must be read via pmccntr_el0 */
+		asm volatile("mrs %0, pmccntr_el0" : "=r" (val));
+	} else {
+		asm volatile(
+			"msr pmselr_el0, %x0\n"
+			"mrs %0, pmxevcntr_el0\n"
+			: "=r" (val)
+			: "rZ" (index)
+		);
+	}
+
+	return val;
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_ARM64_H_ */
Signature · 1 line
2.34.1
#72 lore raw

PATCH v7 3/4 pmu: support reading Intel x86_64 PMU events in runtime

In reply to Tomasz Duszynski

Add support for reading Intel x86_64 PMU events in runtime.

---
 app/test/test_pmu.c          |  2 ++
 lib/pmu/meson.build          |  1 +
 lib/pmu/rte_pmu.h            |  2 ++
 lib/pmu/rte_pmu_pmc_x86_64.h | 24 ++++++++++++++++++++++++
 4 files changed, 29 insertions(+)
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
app/test/test_pmu.c +2 -0
index c53a1bc2f1..07cdc8f5ec 100644
@@ -28,6 +28,8 @@ test_pmu_read(void)
 
 #if defined(RTE_ARCH_ARM64)
 	event = rte_pmu_add_event("cpu_cycles");
+#elif defined(RTE_ARCH_X86_64)
+	event = rte_pmu_add_event("cpu-cycles");
 #endif
 
 	while (tries--)
lib/pmu/meson.build +1 -0
index e857681137..5b92e5c4e3 100644
@@ -13,6 +13,7 @@ sources = files('rte_pmu.c')
 headers = files('rte_pmu.h')
 indirect_headers += files(
         'rte_pmu_pmc_arm64.h',
+        'rte_pmu_pmc_x86_64.h',
 )
 
 if dpdk_conf.has('RTE_ARCH_ARM64')
lib/pmu/rte_pmu.h +2 -0
index b18938dab1..0f7004c31c 100644
@@ -28,6 +28,8 @@ extern "C" {
 
 #if defined(RTE_ARCH_ARM64)
 #include "rte_pmu_pmc_arm64.h"
+#elif defined(RTE_ARCH_X86_64)
+#include "rte_pmu_pmc_x86_64.h"
 #endif
 
 /** Maximum number of events in a group */
lib/pmu/rte_pmu_pmc_x86_64.h new file +24 -0
new file mode 100644
index 0000000000..7b67466960
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Marvell.
+ */
+#ifndef _RTE_PMU_PMC_X86_64_H_
+#define _RTE_PMU_PMC_X86_64_H_
+
+#include <rte_common.h>
+
+static __rte_always_inline uint64_t
+rte_pmu_pmc_read(int index)
+{
+	uint64_t low, high;
+
+	asm volatile(
+		"rdpmc\n"
+		: "=a" (low), "=d" (high)
+		: "c" (index)
+	);
+
+	return low | (high << 32);
+}
+#define rte_pmu_pmc_read rte_pmu_pmc_read
+
+#endif /* _RTE_PMU_PMC_X86_64_H_ */
Signature · 1 line
2.34.1
#73 lore raw

PATCH v7 4/4 eal: add PMU support to tracing library

In reply to Tomasz Duszynski

In order to profile app one needs to store significant amount of samples somewhere for an analysis latern on. Since trace library supports storing data in a CTF format lets take adventage of that and add a dedicated PMU tracepoint.

---
 app/test/test_trace_perf.c               | 10 ++++
 doc/guides/prog_guide/profile_app.rst    |  5 ++
 doc/guides/prog_guide/trace_lib.rst      | 32 +++++++++++++
 lib/eal/common/eal_common_trace.c        | 13 ++++-
 lib/eal/common/eal_common_trace_points.c |  5 ++
 lib/eal/include/rte_eal_trace.h          | 13 +++++
 lib/eal/meson.build                      |  3 ++
 lib/eal/version.map                      |  3 ++
 lib/pmu/rte_pmu.c                        | 61 ++++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 14 ++++++
 lib/pmu/version.map                      |  1 +
 11 files changed, 159 insertions(+), 1 deletion(-)
app/test/test_trace_perf.c +10 -0
index 46ae7d8074..f1929f2734 100644
@@ -114,6 +114,10 @@ worker_fn_##func(void *arg) \
 #define GENERIC_DOUBLE rte_eal_trace_generic_double(3.66666)
 #define GENERIC_STR rte_eal_trace_generic_str("hello world")
 #define VOID_FP app_dpdk_test_fp()
+#ifdef RTE_EXEC_ENV_LINUX
+/* 0 corresponds first event passed via --trace= */
+#define READ_PMU rte_eal_trace_pmu_read(0)
+#endif
 
 WORKER_DEFINE(GENERIC_VOID)
 WORKER_DEFINE(GENERIC_U64)
@@ -122,6 +126,9 @@ WORKER_DEFINE(GENERIC_FLOAT)
 WORKER_DEFINE(GENERIC_DOUBLE)
 WORKER_DEFINE(GENERIC_STR)
 WORKER_DEFINE(VOID_FP)
+#ifdef RTE_EXEC_ENV_LINUX
+WORKER_DEFINE(READ_PMU)
+#endif
 
 static void
 run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
@@ -174,6 +181,9 @@ test_trace_perf(void)
 	run_test("double", worker_fn_GENERIC_DOUBLE, data, sz);
 	run_test("string", worker_fn_GENERIC_STR, data, sz);
 	run_test("void_fp", worker_fn_VOID_FP, data, sz);
+#ifdef RTE_EXEC_ENV_LINUX
+	run_test("read_pmu", worker_fn_READ_PMU, data, sz);
+#endif
 
 	rte_free(data);
 	return TEST_SUCCESS;
doc/guides/prog_guide/profile_app.rst +5 -0
index a8b501fe0c..6a53341c6b 100644
@@ -16,6 +16,11 @@ that information, perf being an example here. Though in some scenarios, eg. when
 isolated (nohz_full) and run dedicated tasks, using perf is less than ideal. In such cases one can
 read specific events directly from application via ``rte_pmu_read()``.
 
+Alternatively tracing library can be used which offers dedicated tracepoint
+``rte_eal_trace_pmu_event()``.
+
+Refer to :doc:`../prog_guide/trace_lib` for more details.
+
 Profiling on x86
 ----------------
 
doc/guides/prog_guide/trace_lib.rst +32 -0
index 9a8f38073d..a8e97ee1ec 100644
@@ -46,6 +46,7 @@ DPDK tracing library features
   trace format and is compatible with ``LTTng``.
   For detailed information, refer to
   `Common Trace Format <https://diamon.org/ctf/>`_.
+- Support reading PMU events on ARM64 and x86-64 (Intel)
 
 How to add a tracepoint?
 ------------------------
@@ -137,6 +138,37 @@ the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
 the ``enable_trace_fp`` option for meson build.
 
+PMU tracepoint
+--------------
+
+Performance measurement unit (PMU) event values can be read from hardware
+registers using predefined ``rte_pmu_read`` tracepoint.
+
+Tracing is enabled via ``--trace`` EAL option by passing both expression
+matching PMU tracepoint name i.e ``lib.eal.pmu.read`` and expression
+``e=ev1[,ev2,...]`` matching particular events::
+
+    --trace='.*pmu.read\|e=cpu_cycles,l1d_cache'
+
+Event names are available under ``/sys/bus/event_source/devices/PMU/events``
+directory, where ``PMU`` is a placeholder for either a ``cpu`` or a directory
+containing ``cpus``.
+
+In contrary to other tracepoints this does not need any extra variables
+added to source files. Instead, caller passes index which follows the order of
+events specified via ``--trace`` parameter. In the following example index ``0``
+corresponds to ``cpu_cyclces`` while index ``1`` corresponds to ``l1d_cache``.
+
+.. code-block:: c
+
+ ...
+ rte_eal_trace_pmu_read(0);
+ rte_eal_trace_pmu_read(1);
+ ...
+
+PMU tracing support must be explicitly enabled using the ``enable_trace_fp``
+option for meson build.
+
 Event record mode
 -----------------
 
lib/eal/common/eal_common_trace.c +12 -1
index 5caaac8e59..3631d0032b 100644
@@ -11,6 +11,9 @@
 #include <rte_errno.h>
 #include <rte_lcore.h>
 #include <rte_per_lcore.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_string_fns.h>
 
 #include "eal_trace.h"
@@ -71,8 +74,13 @@ eal_trace_init(void)
 		goto free_meta;
 
 	/* Apply global configurations */
-	STAILQ_FOREACH(arg, &trace.args, next)
+	STAILQ_FOREACH(arg, &trace.args, next) {
 		trace_args_apply(arg->val);
+#ifdef RTE_EXEC_ENV_LINUX
+		if (rte_pmu_init() == 0)
+			rte_pmu_add_events_by_pattern(arg->val);
+#endif
+	}
 
 	rte_trace_mode_set(trace.mode);
 
@@ -88,6 +96,9 @@ eal_trace_init(void)
 void
 eal_trace_fini(void)
 {
+#ifdef RTE_EXEC_ENV_LINUX
+	rte_pmu_fini();
+#endif
 	trace_mem_free();
 	trace_metadata_destroy();
 	eal_trace_args_free();
lib/eal/common/eal_common_trace_points.c +5 -0
index 0b0b254615..1e46ce549a 100644
@@ -75,3 +75,8 @@ RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_enable,
 	lib.eal.intr.enable)
 RTE_TRACE_POINT_REGISTER(rte_eal_trace_intr_disable,
 	lib.eal.intr.disable)
+
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_REGISTER(rte_eal_trace_pmu_read,
+	lib.eal.pmu.read)
+#endif
lib/eal/include/rte_eal_trace.h +13 -0
index 5ef4398230..afb459b198 100644
@@ -17,6 +17,9 @@ extern "C" {
 
 #include <rte_alarm.h>
 #include <rte_interrupts.h>
+#ifdef RTE_EXEC_ENV_LINUX
+#include <rte_pmu.h>
+#endif
 #include <rte_trace_point.h>
 
 #include "eal_interrupts.h"
@@ -279,6 +282,16 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(cpuset);
 )
 
+#ifdef RTE_EXEC_ENV_LINUX
+RTE_TRACE_POINT_FP(
+	rte_eal_trace_pmu_read,
+	RTE_TRACE_POINT_ARGS(unsigned int index),
+	uint64_t val;
+	val = rte_pmu_read(index);
+	rte_trace_point_emit_u64(val);
+)
+#endif
+
 #ifdef __cplusplus
 }
 #endif
lib/eal/meson.build +3 -0
index 056beb9461..f5865dbcd9 100644
@@ -26,6 +26,9 @@ deps += ['kvargs']
 if not is_windows
     deps += ['telemetry']
 endif
+if is_linux
+    deps += ['pmu']
+endif
 if dpdk_conf.has('RTE_USE_LIBBSD')
     ext_deps += libbsd
 endif
lib/eal/version.map +3 -0
index 7ad12a7dc9..eddb45bebf 100644
@@ -440,6 +440,9 @@ EXPERIMENTAL {
 	rte_thread_detach;
 	rte_thread_equal;
 	rte_thread_join;
+
+	# added in 23.03
+	__rte_eal_trace_pmu_read; # WINDOWS_NO_EXPORT
 };
 
 INTERNAL {
lib/pmu/rte_pmu.c +61 -0

67 lines not on this page · raw message

lib/pmu/rte_pmu.h +14 -0
index 0f7004c31c..0f6250e81f 100644
@@ -169,6 +169,20 @@ __rte_experimental
 int
 rte_pmu_add_event(const char *name);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add events matching pattern to the group of enabled events.
+ *
+ * @param pattern
+ *   Pattern e=ev1[,ev2,...] matching events, where evX is a placeholder for an event listed under
+ *   /sys/bus/event_source/devices/pmu/events.
+ */
+__rte_experimental
+int
+rte_pmu_add_events_by_pattern(const char *pattern);
+
 /**
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice
lib/pmu/version.map +1 -0
index 50fb0f354e..20a27d085c 100644
@@ -8,6 +8,7 @@ EXPERIMENTAL {
 	per_lcore__event_group;
 	rte_pmu;
 	rte_pmu_add_event;
+	rte_pmu_add_events_by_pattern;
 	rte_pmu_fini;
 	rte_pmu_init;
 	rte_pmu_read;
Signature · 1 line
2.34.1
#74 lore raw
In reply to Tomasz Duszynski
Quoted 16 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 1 February 2023 14.18

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on
isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Quoted 7 lines

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

I like the modifications in v7.

#75 lore raw

Hi Morten,

Quoted 30 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Wednesday, February 1, 2023 2:51 PM
To: Tomasz Duszynski <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXT] RE: [PATCH v7 0/4] add support for self monitoring

External Email

----------------------------------------------------------------------

Quoted 18 lines

From: Tomasz Duszynski [mailto:[email protected]]
Sent: Wednesday, 1 February 2023 14.18

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

Quoted 8 lines
Quoted 4 lines

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

I like the modifications in v7.

Thanks.

#76 lore raw

v8 cover add support for self monitoring

In reply to Tomasz Duszynski

This series adds self monitoring support i.e allows to configure and read performance measurement unit (PMU) counters in runtime without using perf utility. This has certain adventages when application runs on isolated cores with nohz_full kernel parameter.

Events can be read directly using rte_pmu_read() or using dedicated tracepoint rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.

By design, all enabled events are grouped together and the same group is attached to lcores that use self monitoring funtionality.

Events are enabled by names, which need to be read from standard location under sysfs i.e

/sys/bus/event_source/devices/PMU/events

where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.

v8:
- just rebase series
v7:
- use per-lcore event group instead of global table index by lcore-id
- don't add pmu_autotest to fast tests because due to lack of suported on every arch
v6:
- move codebase to the separate library
- address review comments
v5:
- address review comments
- fix sign extension while reading pmu on x86
- fix regex mentioned in doc
- various minor changes/improvements here and there
v4:
- fix freeing mem detected by debug_autotest
v3:
- fix shared build
v2:
- fix problems reported by test build infra

Tomasz Duszynski (4):
lib: add generic support for reading PMU events
pmu: support reading ARM PMU events in runtime
pmu: support reading Intel x86_64 PMU events in runtime
eal: add PMU support to tracing library

 MAINTAINERS                              |   5 +
 app/test/meson.build                     |   1 +
 app/test/test_pmu.c                      |  61 +++
 app/test/test_trace_perf.c               |  10 +
 doc/api/doxy-api-index.md                |   3 +-
 doc/api/doxy-api.conf.in                 |   1 +
 doc/guides/prog_guide/profile_app.rst    |  13 +
 doc/guides/prog_guide/trace_lib.rst      |  32 ++
 doc/guides/rel_notes/release_23_03.rst   |   9 +
 lib/eal/common/eal_common_trace.c        |  13 +-
 lib/eal/common/eal_common_trace_points.c |   5 +
 lib/eal/include/rte_eal_trace.h          |  13 +
 lib/eal/meson.build                      |   3 +
 lib/eal/version.map                      |   1 +
 lib/meson.build                          |   1 +
 lib/pmu/meson.build                      |  21 +
 lib/pmu/pmu_arm64.c                      |  94 ++++
 lib/pmu/pmu_private.h                    |  29 ++
 lib/pmu/rte_pmu.c                        | 525 +++++++++++++++++++++++
 lib/pmu/rte_pmu.h                        | 225 ++++++++++
 lib/pmu/rte_pmu_pmc_arm64.h              |  30 ++
 lib/pmu/rte_pmu_pmc_x86_64.h             |  24 ++
 lib/pmu/version.map                      |  21 +
 23 files changed, 1138 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_pmu.c
 create mode 100644 lib/pmu/meson.build
 create mode 100644 lib/pmu/pmu_arm64.c
 create mode 100644 lib/pmu/pmu_private.h
 create mode 100644 lib/pmu/rte_pmu.c
 create mode 100644 lib/pmu/rte_pmu.h
 create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h
 create mode 100644 lib/pmu/rte_pmu_pmc_x86_64.h
 create mode 100644 lib/pmu/version.map

--
2.34.1