Skip to contentSkip to conversation

perf stat affinity changes

PATCH Superseded Ian Rogers 2 participants 7 messages Jan 8 – Jan 28 perf events

v6 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

v6 cover perf stat affinity changes

Change how affinities work with evlist__for_each_cpu. Move the affinity code into the iterator to simplify setting it up. Detect when affinities will and won't be profitable, for example a tool event and a regular perf event (or read group) may face less delay from a single IPI for the event read than from a call to sched_setaffinity. Add a
--no-affinity flag to perf stat to allow affinities to be disabled.

v6: Drop merged tool event change. Move TPEBS fix into its own patch

    1st.

v5: Drop merged changes. Move tool event reading to first
patch. Change --no-affinity flag to --affinity/--no-affinity flag.
https://lore.kernel.org/lkml/[email protected]/
On v5 there was discussion with Andi Kleen who points out that affinities will work better with real time priorities but using this requires privileges.

v4: Rebase. Add patch to reduce scope of walltime_nsec_stats now that
the legacy metric code is no more. Minor tweak to the ru_stats clean up.
https://lore.kernel.org/lkml/[email protected]/

v3: Add affinity clean ups and read tool events last.
https://lore.kernel.org/lkml/[email protected]/

v2: Fixed an aggregation index issue:
https://lore.kernel.org/lkml/[email protected]/

v1:
https://lore.kernel.org/lkml/[email protected]/

Ian Rogers (3):
perf evlist: Missing TPEBS close in evlist__close perf evlist: Reduce affinity use and move into iterator, fix no
affinity
perf stat: Add no-affinity flag

 tools/perf/Documentation/perf-stat.txt |   4 +
 tools/perf/builtin-stat.c              | 114 ++++++++----------
 tools/perf/util/evlist.c               | 156 +++++++++++++++----------
 tools/perf/util/evlist.h               |  27 +++--
 tools/perf/util/pmu.c                  |  12 ++
 tools/perf/util/pmu.h                  |   1 +
 6 files changed, 183 insertions(+), 131 deletions(-)
Signature · 1 line
2.52.0.457.g6b5491de43-goog
author reviewer
#1 lore raw

PATCH v6 1/3 perf evlist: Missing TPEBS close in evlist__close

The libperf evsel close won't close TPEBS events properly. Add a test to do this. The libperf close routine is used in evlist__close for affinity reasons.

---
 tools/perf/util/evlist.c | 2 ++
 1 file changed, 2 insertions(+)
tools/perf/util/evlist.c +2 -0
index 649519628541..bb042d89e6a0 100644
@@ -1356,6 +1356,8 @@ void evlist__close(struct evlist *evlist)
 		return;
 
 	evlist__for_each_cpu(evlist_cpu_itr, evlist, &affinity) {
+		if (evlist_cpu_itr.cpu_map_idx == 0 && evsel__is_retire_lat(evlist_cpu_itr.evsel))
+			evsel__tpebs_close(evlist_cpu_itr.evsel);
 		perf_evsel__close_cpu(&evlist_cpu_itr.evsel->core,
 				      evlist_cpu_itr.cpu_map_idx);
 	}
Signature · 1 line
2.52.0.457.g6b5491de43-goog
author reviewer
#2 lore raw

PATCH v6 2/3 perf evlist: Reduce affinity use and move into iterator, fix no affinity

In reply to Ian Rogers

The evlist__for_each_cpu iterator will call sched_setaffitinity when moving between CPUs to avoid IPIs. If only 1 IPI is saved then this may be unprofitable as the delay to get scheduled may be considerable. This may be particularly true if reading an event group in perf stat in interval mode.

Move the affinity handling completely into the iterator so that a single evlist__use_affinity can determine whether CPU affinities will be used. For perf record the change is minimal as the dummy event and the real event will always make the use of affinities the thing to do. In perf stat, tool events are ignored and affinities only used if >1 event on the same CPU occur. Determining if affinities are useful is done by evlist__use_affinity which tests per-event whether the event's PMU benefits from affinity use - it is assumed only perf event using PMUs do.

Fix a bug where when there are no affinities that the CPU map iterator may reference a CPU not present in the initial evsel. Fix by making the iterator and non-iterator code common.

---
 tools/perf/builtin-stat.c | 108 +++++++++++---------------
 tools/perf/util/evlist.c  | 158 +++++++++++++++++++++++---------------
 tools/perf/util/evlist.h  |  26 +++++--
 tools/perf/util/pmu.c     |  12 +++
 tools/perf/util/pmu.h     |   1 +
 5 files changed, 174 insertions(+), 131 deletions(-)
tools/perf/builtin-stat.c +44 -64
index ab40d85fb125..bb14268e7393 100644
@@ -369,19 +369,11 @@ static int read_counter_cpu(struct evsel *counter, int cpu_map_idx)
 static int read_counters_with_affinity(void)
 {
 	struct evlist_cpu_iterator evlist_cpu_itr;
-	struct affinity saved_affinity, *affinity;
 
 	if (all_counters_use_bpf)
 		return 0;
 
-	if (!target__has_cpu(&target) || target__has_per_thread(&target))
-		affinity = NULL;
-	else if (affinity__setup(&saved_affinity) < 0)
-		return -1;
-	else
-		affinity = &saved_affinity;
-
-	evlist__for_each_cpu(evlist_cpu_itr, evsel_list, affinity) {
+	evlist__for_each_cpu(evlist_cpu_itr, evsel_list) {
 		struct evsel *counter = evlist_cpu_itr.evsel;
 
 		if (evsel__is_bpf(counter))
@@ -393,8 +385,6 @@ static int read_counters_with_affinity(void)
 		if (!counter->err)
 			counter->err = read_counter_cpu(counter, evlist_cpu_itr.cpu_map_idx);
 	}
-	if (affinity)
-		affinity__cleanup(&saved_affinity);
 
 	return 0;
 }
@@ -793,7 +783,6 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 	const bool forks = (argc > 0);
 	bool is_pipe = STAT_RECORD ? perf_stat.data.is_pipe : false;
 	struct evlist_cpu_iterator evlist_cpu_itr;
-	struct affinity saved_affinity, *affinity = NULL;
 	int err, open_err = 0;
 	bool second_pass = false, has_supported_counters;
 
@@ -805,14 +794,6 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 		child_pid = evsel_list->workload.pid;
 	}
 
-	if (!cpu_map__is_dummy(evsel_list->core.user_requested_cpus)) {
-		if (affinity__setup(&saved_affinity) < 0) {
-			err = -1;
-			goto err_out;
-		}
-		affinity = &saved_affinity;
-	}
-
 	evlist__for_each_entry(evsel_list, counter) {
 		counter->reset_group = false;
 		if (bpf_counter__load(counter, &target)) {
@@ -825,49 +806,48 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 
 	evlist__reset_aggr_stats(evsel_list);
 
-	evlist__for_each_cpu(evlist_cpu_itr, evsel_list, affinity) {
-		counter = evlist_cpu_itr.evsel;
+	/*
+	 * bperf calls evsel__open_per_cpu() in bperf__load(), so
+	 * no need to call it again here.
+	 */
+	if (!target.use_bpf) {
+		evlist__for_each_cpu(evlist_cpu_itr, evsel_list) {
+			counter = evlist_cpu_itr.evsel;
 
-		/*
-		 * bperf calls evsel__open_per_cpu() in bperf__load(), so
-		 * no need to call it again here.
-		 */
-		if (target.use_bpf)
-			break;
+			if (counter->reset_group || !counter->supported)
+				continue;
+			if (evsel__is_bperf(counter))
+				continue;
 
-		if (counter->reset_group || !counter->supported)
-			continue;
-		if (evsel__is_bperf(counter))
-			continue;
+			while (true) {
+				if (create_perf_stat_counter(counter, &stat_config,
+							      evlist_cpu_itr.cpu_map_idx) == 0)
+					break;
 
-		while (true) {
-			if (create_perf_stat_counter(counter, &stat_config,
-						     evlist_cpu_itr.cpu_map_idx) == 0)
-				break;
+				open_err = errno;
+				/*
+				 * Weak group failed. We cannot just undo this
+				 * here because earlier CPUs might be in group
+				 * mode, and the kernel doesn't support mixing
+				 * group and non group reads. Defer it to later.
+				 * Don't close here because we're in the wrong
+				 * affinity.
+				 */
+				if ((open_err == EINVAL || open_err == EBADF) &&
+					evsel__leader(counter) != counter &&
+					counter->weak_group) {
+					evlist__reset_weak_group(evsel_list, counter, false);
+					assert(counter->reset_group);
+					counter->supported = true;
+					second_pass = true;
+					break;
+				}
 
-			open_err = errno;
-			/*
-			 * Weak group failed. We cannot just undo this here
-			 * because earlier CPUs might be in group mode, and the kernel
-			 * doesn't support mixing group and non group reads. Defer
-			 * it to later.
-			 * Don't close here because we're in the wrong affinity.
-			 */
-			if ((open_err == EINVAL || open_err == EBADF) &&
-				evsel__leader(counter) != counter &&
-				counter->weak_group) {
-				evlist__reset_weak_group(evsel_list, counter, false);
-				assert(counter->reset_group);
-				counter->supported = true;
-				second_pass = true;
-				break;
+				if (stat_handle_error(counter, open_err) != COUNTER_RETRY)
+					break;
 			}
-
-			if (stat_handle_error(counter, open_err) != COUNTER_RETRY)
-				break;
 		}
 	}
-
 	if (second_pass) {
 		/*
 		 * Now redo all the weak group after closing them,
@@ -875,7 +855,7 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 		 */
 
 		/* First close errored or weak retry */
-		evlist__for_each_cpu(evlist_cpu_itr, evsel_list, affinity) {
+		evlist__for_each_cpu(evlist_cpu_itr, evsel_list) {
 			counter = evlist_cpu_itr.evsel;
 
 			if (!counter->reset_group && counter->supported)
@@ -884,7 +864,7 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 			perf_evsel__close_cpu(&counter->core, evlist_cpu_itr.cpu_map_idx);
 		}
 		/* Now reopen weak */
-		evlist__for_each_cpu(evlist_cpu_itr, evsel_list, affinity) {
+		evlist__for_each_cpu(evlist_cpu_itr, evsel_list) {
 			counter = evlist_cpu_itr.evsel;
 
 			if (!counter->reset_group)
@@ -893,17 +873,18 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 			while (true) {
 				pr_debug2("reopening weak %s\n", evsel__name(counter));
 				if (create_perf_stat_counter(counter, &stat_config,
-							     evlist_cpu_itr.cpu_map_idx) == 0)
+							     evlist_cpu_itr.cpu_map_idx) == 0) {
+					evlist_cpu_iterator__exit(&evlist_cpu_itr);
 					break;
-
+				}
 				open_err = errno;
-				if (stat_handle_error(counter, open_err) != COUNTER_RETRY)
+				if (stat_handle_error(counter, open_err) != COUNTER_RETRY) {
+					evlist_cpu_iterator__exit(&evlist_cpu_itr);
 					break;
+				}
 			}
 		}
 	}
-	affinity__cleanup(affinity);
-	affinity = NULL;
 
 	has_supported_counters = false;
 	evlist__for_each_entry(evsel_list, counter) {
@@ -1066,7 +1047,6 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 	if (forks)
 		evlist__cancel_workload(evsel_list);
 
-	affinity__cleanup(affinity);
 	return err;
 }
 
tools/perf/util/evlist.c +98 -60

227 lines not on this page · raw message

tools/perf/util/evlist.h +19 -7
index 911834ae7c2a..30dff7484d3c 100644
@@ -10,6 +10,7 @@
 #include <internal/evlist.h>
 #include <internal/evsel.h>
 #include <perf/evlist.h>
+#include "affinity.h"
 #include "events_stats.h"
 #include "evsel.h"
 #include "rblist.h"
@@ -363,6 +364,8 @@ struct evlist_cpu_iterator {
 	struct perf_cpu cpu;
 	/** If present, used to set the affinity when switching between CPUs. */
 	struct affinity *affinity;
+	/** Maybe be used to hold affinity state prior to iterating. */
+	struct affinity saved_affinity;
 };
 
 /**
@@ -370,22 +373,31 @@ struct evlist_cpu_iterator {
  *                        affinity, iterate over all CPUs and then the evlist
  *                        for each evsel on that CPU. When switching between
  *                        CPUs the affinity is set to the CPU to avoid IPIs
- *                        during syscalls.
+ *                        during syscalls. The affinity is set up and removed
+ *                        automatically, if the loop is broken a call to
+ *                        evlist_cpu_iterator__exit is necessary.
  * @evlist_cpu_itr: the iterator instance.
  * @evlist: evlist instance to iterate.
- * @affinity: NULL or used to set the affinity to the current CPU.
  */
-#define evlist__for_each_cpu(evlist_cpu_itr, evlist, affinity)		\
-	for ((evlist_cpu_itr) = evlist__cpu_begin(evlist, affinity);	\
+#define evlist__for_each_cpu(evlist_cpu_itr, evlist)			\
+	for (evlist_cpu_iterator__init(&(evlist_cpu_itr), evlist);	\
 	     !evlist_cpu_iterator__end(&evlist_cpu_itr);		\
 	     evlist_cpu_iterator__next(&evlist_cpu_itr))
 
-/** Returns an iterator set to the first CPU/evsel of evlist. */
-struct evlist_cpu_iterator evlist__cpu_begin(struct evlist *evlist, struct affinity *affinity);
+/** Setup an iterator set to the first CPU/evsel of evlist. */
+void evlist_cpu_iterator__init(struct evlist_cpu_iterator *itr, struct evlist *evlist);
+/**
+ * Cleans up the iterator, automatically done by evlist_cpu_iterator__next when
+ * the end of the list is reached. Multiple calls are safe.
+ */
+void evlist_cpu_iterator__exit(struct evlist_cpu_iterator *itr);
 /** Move to next element in iterator, updating CPU, evsel and the affinity. */
 void evlist_cpu_iterator__next(struct evlist_cpu_iterator *evlist_cpu_itr);
 /** Returns true when iterator is at the end of the CPUs and evlist. */
-bool evlist_cpu_iterator__end(const struct evlist_cpu_iterator *evlist_cpu_itr);
+static inline bool evlist_cpu_iterator__end(const struct evlist_cpu_iterator *evlist_cpu_itr)
+{
+	return evlist_cpu_itr->evlist_cpu_map_idx >= evlist_cpu_itr->evlist_cpu_map_nr;
+}
 
 struct evsel *evlist__get_tracking_event(struct evlist *evlist);
 void evlist__set_tracking_event(struct evlist *evlist, struct evsel *tracking_evsel);
tools/perf/util/pmu.c +12 -0
index 956ea273c2c7..853b8addead6 100644
@@ -2420,6 +2420,18 @@ bool perf_pmu__is_software(const struct perf_pmu *pmu)
 	return false;
 }
 
+bool perf_pmu__benefits_from_affinity(struct perf_pmu *pmu)
+{
+	if (!pmu)
+		return true; /* Assume is core. */
+
+	/*
+	 * All perf event PMUs should benefit from accessing the perf event
+	 * contexts on the local CPU.
+	 */
+	return pmu->type <= PERF_PMU_TYPE_PE_END;
+}
+
 FILE *perf_pmu__open_file(const struct perf_pmu *pmu, const char *name)
 {
 	char path[PATH_MAX];
tools/perf/util/pmu.h +1 -0
index 8f11bfe8ed6d..689542581429 100644
@@ -273,6 +273,7 @@ bool perf_pmu__name_no_suffix_match(const struct perf_pmu *pmu, const char *to_m
  *                        perf_sw_context in the kernel?
  */
 bool perf_pmu__is_software(const struct perf_pmu *pmu);
+bool perf_pmu__benefits_from_affinity(struct perf_pmu *pmu);
 
 FILE *perf_pmu__open_file(const struct perf_pmu *pmu, const char *name);
 FILE *perf_pmu__open_file_at(const struct perf_pmu *pmu, int dirfd, const char *name);
Signature · 1 line
2.52.0.457.g6b5491de43-goog
author reviewer
#3 lore raw

PATCH v6 3/3 perf stat: Add no-affinity flag

In reply to Ian Rogers

Add flag that disables affinity behavior. Using sched_setaffinity to place a perf thread on a CPU can avoid certain interprocessor interrupts but may introduce a delay due to the scheduling, particularly on loaded machines. Add a command line option to disable the behavior. This behavior is less present in other tools like perf record, as it uses a ring buffer and doesn't make repeated system calls.

---
 tools/perf/Documentation/perf-stat.txt | 4 ++++
 tools/perf/builtin-stat.c              | 6 ++++++
 tools/perf/util/evlist.c               | 6 +-----
 tools/perf/util/evlist.h               | 1 +
 4 files changed, 12 insertions(+), 5 deletions(-)
tools/perf/Documentation/perf-stat.txt +4 -0
index 1a766d4a2233..1ffb510606af 100644
@@ -382,6 +382,10 @@ color the metric's computed value.
 Don't print output, warnings or messages. This is useful with perf stat
 record below to only write data to the perf.data file.
 
+--no-affinity::
+Don't change scheduler affinities when iterating over CPUs. Disables
+an optimization aimed at minimizing interprocessor interrupts.
+
 STAT RECORD
 -----------
 Stores stat data into perf data file.
tools/perf/builtin-stat.c +6 -0
index bb14268e7393..ddda0ea62eaf 100644
@@ -2427,6 +2427,7 @@ static int parse_tpebs_mode(const struct option *opt, const char *str,
 int cmd_stat(int argc, const char **argv)
 {
 	struct opt_aggr_mode opt_mode = {};
+	bool affinity = true, affinity_set = false;
 	struct option stat_options[] = {
 		OPT_BOOLEAN('T', "transaction", &transaction_run,
 			"hardware transaction statistics"),
@@ -2555,6 +2556,8 @@ int cmd_stat(int argc, const char **argv)
 			"don't print 'summary' for CSV summary output"),
 		OPT_BOOLEAN(0, "quiet", &quiet,
 			"don't print any output, messages or warnings (useful with record)"),
+		OPT_BOOLEAN_SET(0, "affinity", &affinity, &affinity_set,
+			"don't allow affinity optimizations aimed at reducing IPIs"),
 		OPT_CALLBACK(0, "cputype", &evsel_list, "hybrid cpu type",
 			"Only enable events on applying cpu with this type "
 			"for hybrid platform (e.g. core or atom)",
@@ -2612,6 +2615,9 @@ int cmd_stat(int argc, const char **argv)
 	} else
 		stat_config.csv_sep = DEFAULT_SEPARATOR;
 
+	if (affinity_set)
+		evsel_list->no_affinity = !affinity;
+
 	if (argc && strlen(argv[0]) > 2 && strstarts("record", argv[0])) {
 		argc = __cmd_record(stat_options, &opt_mode, argc, argv);
 		if (argc < 0)
tools/perf/util/evlist.c +1 -5
index d62b8bab8fa4..00fb3cf45bae 100644
@@ -369,11 +369,7 @@ static bool evlist__use_affinity(struct evlist *evlist)
 	struct perf_cpu_map *used_cpus = NULL;
 	bool ret = false;
 
-	/*
-	 * With perf record core.user_requested_cpus is usually NULL.
-	 * Use the old method to handle this for now.
-	 */
-	if (!evlist->core.user_requested_cpus ||
+	if (evlist->no_affinity || !evlist->core.user_requested_cpus ||
 	    cpu_map__is_dummy(evlist->core.user_requested_cpus))
 		return false;
 
tools/perf/util/evlist.h +1 -0
index 30dff7484d3c..d17c3b57a409 100644
@@ -59,6 +59,7 @@ struct event_enable_timer;
 struct evlist {
 	struct perf_evlist core;
 	bool		 enabled;
+	bool		 no_affinity;
 	int		 id_pos;
 	int		 is_pos;
 	int		 nr_br_cntr;
Signature · 1 line
2.52.0.457.g6b5491de43-goog
#4 lore raw

Re: [PATCH v6 3/3] perf stat: Add no-affinity flag

Quoted 7 lines · Ian Rogers

Add flag that disables affinity behavior. Using sched_setaffinity to place a perf thread on a CPU can avoid certain interprocessor interrupts but may introduce a delay due to the scheduling, particularly on loaded machines. Add a command line option to disable the behavior. This behavior is less present in other tools like perf record, as it uses a ring buffer and doesn't make repeated system calls.

I thought we had agreed that this change isn't needed?

-Andi

author reviewer
#5 lore raw

Re: [PATCH v6 3/3] perf stat: Add no-affinity flag

Quoted 11 lines · Andi Kleen
Quoted 7 lines · Ian Rogers

Add flag that disables affinity behavior. Using sched_setaffinity to place a perf thread on a CPU can avoid certain interprocessor interrupts but may introduce a delay due to the scheduling, particularly on loaded machines. Add a command line option to disable the behavior. This behavior is less present in other tools like perf record, as it uses a ring buffer and doesn't make repeated system calls.

I thought we had agreed that this change isn't needed?

This patch or the series? My last feedback was:
https://lore.kernel.org/lkml/CAP-5=fUvsF7RtLAKaMwc28CeSEOJ+j0gVwvQN59moOnUS=kWVg@mail.gmail.com/

So the code as-is is trying to always use setaffinity. For a single syscall on a particular CPU this is unlikely to be profitable on a machine under load, the IPI will happen faster. You mentioned that realtime priorities could address this but that this also required capabilities. I didn't see that as something that contradicted the use of these patches.

The point of the flag in this change is so that the IPI behavior can be had should issues with CPU affinities be experienced. With how the code is refactored we can also make the existing "always use affinities" behavior an option by modifying evlist__use_affinity, but given the cost of an IPI should be less than that of migrating a thread it doesn't seem like this option would be useful - you could also make the code do this by adding a dummy event as >1 event triggers the setaffinity behavior.

Anyway, I think we still want these changes that successfully fixed a customer issue I had. The already merged patches do improve things, and I think we can do yet more, but the real cause of the delay in reading counters was the calls to setaffinity and this being slow on loaded machines. The patches make it so setaffinity doesn't happen if there is just a single IPI being saved or if requested on the command line.

Thanks,
Ian

Quoted 1 line

-Andi

author reviewer
#6 lore raw
In reply to Ian Rogers
Quoted 7 lines · Ian Rogers

Change how affinities work with evlist__for_each_cpu. Move the affinity code into the iterator to simplify setting it up. Detect when affinities will and won't be profitable, for example a tool event and a regular perf event (or read group) may face less delay from a single IPI for the event read than from a call to sched_setaffinity. Add a
--no-affinity flag to perf stat to allow affinities to be disabled.

Ping.

Thanks,
Ian

Quoted 41 lines

v6: Drop merged tool event change. Move TPEBS fix into its own patch

    1st.

v5: Drop merged changes. Move tool event reading to first
patch. Change --no-affinity flag to --affinity/--no-affinity flag.
https://lore.kernel.org/lkml/[email protected]/
On v5 there was discussion with Andi Kleen who points out that affinities will work better with real time priorities but using this requires privileges.

v4: Rebase. Add patch to reduce scope of walltime_nsec_stats now that
the legacy metric code is no more. Minor tweak to the ru_stats clean up.
https://lore.kernel.org/lkml/[email protected]/

v3: Add affinity clean ups and read tool events last.
https://lore.kernel.org/lkml/[email protected]/

v2: Fixed an aggregation index issue:
https://lore.kernel.org/lkml/[email protected]/

v1:
https://lore.kernel.org/lkml/[email protected]/

Ian Rogers (3):
perf evlist: Missing TPEBS close in evlist__close perf evlist: Reduce affinity use and move into iterator, fix no
affinity
perf stat: Add no-affinity flag

 tools/perf/Documentation/perf-stat.txt |   4 +
 tools/perf/builtin-stat.c              | 114 ++++++++----------
 tools/perf/util/evlist.c               | 156 +++++++++++++++----------
 tools/perf/util/evlist.h               |  27 +++--
 tools/perf/util/pmu.c                  |  12 ++
 tools/perf/util/pmu.h                  |   1 +
 6 files changed, 183 insertions(+), 131 deletions(-)

--
2.52.0.457.g6b5491de43-goog