Skip to contentSkip to conversation

perf tools: Add progress diagnostics and a false-sharing workload

PATCH Superseded Arnaldo Carvalho de Melo 3 participants 12 messages Sep 29 perf events

v5 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Arnaldo Carvalho de Melo

author maintainer
#0 lore raw

v5 cover perf tools: Add progress diagnostics and a false-sharing workload

Hi,

This series adds progress and stuck-process diagnostics to perf, and a workload that makes false sharing visible to data type profiling.

The changes are:

- move perf_config__set_variable() to util/config.c and serialize config parser and read-modify-write state, so non-builtin perf code can persist configuration changes safely;

- add 'perf report --progress' for stdio users, showing the current phase,

    percentage, and counts while a session is processed;

- add 'perf report --no-progress', the counterpart above for the TUI and

    GTK browsers, whose progress there is no other way to turn off;

- add the prototype tools/perf/scripts/perf-stuck.sh helper, which samples a running perf process and can invoke the accompanying GDB commands when progress stops;

- add 'perf test -w false_sharing', a synthetic TCP-shaped workload with identity and packet counters sharing a cacheline, and include it in the data type profiling shell test.

The progress backend handles nested phases and completes them on error paths as well as successful paths. The false-sharing workload uses a shared volatile instance so the data type profiler can resolve direct accesses to the individual members.

The perf-stuck helper is intentionally a prototype and requires the usual /proc access; its optional GDB mode additionally requires gdb and suitable debug information. When only one CPU is available, the false-sharing workload warns that it cannot produce cross-CPU traffic but still runs.

Best regards,

- Arnaldo

What changed from v4:

- tools/perf/builtin-report.c: mark --progress PARSE_OPT_NOAUTONEG. parse_long_opt() claims "--no-progress" for it as soon as any option without that flag is tried, before the --no-progress option itself is ever reached, so report.no_progress stayed false and patch 3/5 did not do what it promises: --no-progress --progress printed the phases, and the TUI and GTK progress it exists to turn off stayed on. Checked with the option values printed at the hook, and again with a two-option program linking the same libsubcmd. Sashiko found this reviewing v4, asking whether the option was shadowed and the branch dead code;

- tools/perf/util/config.c: format the path buffer inside the critical section. It is shared by every caller, unchanged by the static storage it got addressing v3, and mkpath() ran before the lock was taken: two callers could still interleave their vsnprintf() there. Nothing calls it from more than one thread today. Sashiko pointed out the window, reviewing v4;

- tools/perf/tests/workloads/false_sharing.c: look at every bit of the affinity mask instead of bounding the walk by sysconf(_SC_NPROCESSORS_CONF), that counts configured CPUs, not the highest ID allowed. A cpuset restricted to high numbered ones is then smaller than the IDs in it, and no allowed CPU is found at all, which unpins every thread: reproduced with a mask holding only CPU 20 and a count of 4, where the old bound finds none and the new one finds it. Sashiko asked about sparse topologies reviewing v4;

What changed from v3:

- tools/perf/util/config.c: keep the buffer perf_config__set_variable() hands to the parser in static storage. The parser publishes that pointer as config_file_name, and every reader currently only stays clear of it by taking config_mutex on the way in, which is too easy to
lose: nothing reachable races today, a getter would have to run outside a parse for that, but the change costs nothing and removes the class. Reported by Sashiko while reviewing v3;

- tools/perf/scripts/perf-stuck.gdb: perf-dso now walks each candidate for the dso until one evaluates, so it resolves the REFCNT_CHECKING proxy of an ASan/LSan build too instead of aborting there with "There is no member named dso", which the previous version could only point at in a comment. Reported by Sashiko while reviewing v3;

- tools/perf/tests/workloads/false_sharing.c: put sum before cpu in struct fs_reader, the implicit padding after cpu pushed the struct to 68 bytes and aligning it rounded that up to 128, twice the cacheline the padding arithmetic was written for. Reported by Sashiko while reviewing v3.

- tools/perf/builtin-report.c: --quiet asks for no messages at all, so --progress leaves the phases uncounted when both are given, documented next to --progress in perf-report.txt. Namhyung Kim asked, reviewing v3, what combining them should do;

- tools/perf/builtin-report.c, tools/perf/ui/progress.c: add --no-progress, installing the no-op ui_progress ops after setup_browser() installed the ones of the TUI or GTK browser. Suggested by Namhyung Kim while reviewing v3: their progress has had no way to be turned off, and the option just added asks for it where there is none. It takes precedence over --progress;

- tools/perf/scripts/perf-stuck.sh: check that gdb is there before watching instead of failing when -g first fires, two samples in, which can be minutes apart. Namhyung Kim pointed it out reviewing v3;

What changed from v2:

- tools/perf/util/config.c: perf_etc_perfconfig() returned NULL when the allocation in system_path() fails, and none of its callers check for that. Reported by Sashiko while reviewing v2: the deref reached from this series was fixed, and since the same unchecked deref is already reachable from the three callers predating it, perf_config_from_file() and 'perf daemon' among them, the opportunity was taken to make the accessor total as well, falling back to the unresolved path, which for the usual absolute ETC_PERFCONFIG is the same string;

- tools/perf/scripts/perf-stuck.gdb: don't dereference map_symbol.sym without checking it for NULL, printing "(no symbol)" instead. Reported by Sashiko while reviewing v2;

- tools/perf/scripts/perf-stuck.gdb: note next to the structure walks that a REFCNT_CHECKING build wraps some structs in a proxy holding the original, where this has to read map->orig->dso->orig->name;

- tools/perf/scripts/perf-stuck.sh: count the samples showing no progress when there is no progress to look at, as an empty or missing progress log reset the counter on every sample and never fired -g;

- tools/perf/scripts/perf-stuck.sh: use -- for pgrep and tail, as a process name starting with a hyphen was taken as an option and had the script attach to an unrelated process, gdb included when -g is used;

- tools/perf/scripts/perf-stuck.sh: bound the gdb run with timeout --signal=INT 30, as the script makes inferior calls and one into a perf wedged in a loop never returns, while SIGINT lets gdb release the inferior instead of leaving perf stopped.

What changed from v1:

- avoid calling CPU_SET() with -1 when false_sharing runs with only one CPU available in its affinity mask.

tools/perf/Documentation/perf-report.txt | 19 ++

  tools/perf/builtin-config.c                   |  70 +----
  tools/perf/builtin-report.c                   |  22 ++
  tools/perf/scripts/perf-stuck.gdb             | 129 +++++++++
  tools/perf/scripts/perf-stuck.sh              | 188 +++++++++++++
  tools/perf/tests/builtin-test.c               |   1 +
  tools/perf/tests/shell/data_type_profiling.sh |   9 +-
  tools/perf/tests/tests.h                      |   1 +
  tools/perf/tests/workloads/Build              |   2 +
  tools/perf/tests/workloads/false_sharing.c    | 250 ++++++++++++++++++
  tools/perf/ui/Build                           |   1 +
  tools/perf/ui/progress.c                      |   6 +
  tools/perf/ui/progress.h                      |   4 +
  tools/perf/ui/stdio/progress.c                | 162 ++++++++++++
  tools/perf/util/config.c                      | 144 +++++++++-
  tools/perf/util/config.h                      |   2 +
  tools/perf/util/ordered-events.c              |  16 +-
  tools/perf/util/session.c                     |  12 +-
  18 files changed, 955 insertions(+), 83 deletions(-)

create mode 100644 tools/perf/scripts/perf-stuck.gdb create mode 100755 tools/perf/scripts/perf-stuck.sh create mode 100644 tools/perf/tests/workloads/false_sharing.c create mode 100644 tools/perf/ui/stdio/progress.c

base-commit: 0ae6fc78c5ce0dfd
v1-head: 45d7917f7e05e8a29828ed5f0bbdc94fc938f79f
v2-head: d4f84e4de8890194924ccd897a8e6773e7d4240b
v3-head: 485532296710225862daf8ffb19802ae327efe2a
v4-head: 38193635508e0a5f04a6ebf70db27534b68b2d5b
--

Assisted-by OpenCode: GPT-5.6 Luna
author maintainer
#1 lore raw

PATCH v5 1/5 perf config: Move perf_config__set_variable() to util/config.c

From: Arnaldo Carvalho de Melo <[email protected]>

Move perf_config__set_variable() out of the 'perf config' builtin so that opt-in features can persist their choice from outside it, e.g. util/debuginfo.c writing core.debuginfod=false when the user disables debuginfod for the rest of the session. The
set_config() body becomes perf_config_set__write(), with the system_config choice as an argument, as the builtin's use_system_config/use_user_config statics are not available outside it.

All config file access shares the static parser state and can now run on more than one thread, with a feature writing the config while perf top's display thread reads it, so serialize parsing and rewriting with a mutex, and the whole read-modify-write of perf_config__set_variable() against itself.

perf_etc_perfconfig() caches what system_path() returns, and that allocates, so on failure every caller dereferenced NULL, the pre-existing ones in perf_config_from_file() and in the daemon included. Pre-existing, not introduced here, but this rewrites the accessor anyway, so make it total: the unresolved path is the same string for the usual absolute ETC_PERFCONFIG.

---
 tools/perf/builtin-config.c |  70 +-----------------
 tools/perf/util/config.c    | 144 ++++++++++++++++++++++++++++++++++--
 tools/perf/util/config.h    |   2 +
 3 files changed, 142 insertions(+), 74 deletions(-)
tools/perf/builtin-config.c +1 -69
index cefd042e4f853466..3b074aca8d344539 100644
@@ -41,37 +41,7 @@ static struct option config_options[] = {
 
 static int set_config(struct perf_config_set *set, const char *file_name)
 {
-	struct perf_config_section *section = NULL;
-	struct perf_config_item *item = NULL;
-	const char *first_line = "# this file is auto-generated.";
-	FILE *fp;
-
-	if (set == NULL)
-		return -1;
-
-	fp = fopen(file_name, "w");
-	if (!fp)
-		return -1;
-
-	fprintf(fp, "%s\n", first_line);
-
-	/* overwrite configvariables */
-	perf_config_items__for_each_entry(&set->sections, section) {
-		if (!use_system_config && section->from_system_config)
-			continue;
-		fprintf(fp, "[%s]\n", section->name);
-
-		perf_config_items__for_each_entry(&section->items, item) {
-			if (!use_system_config && item->from_system_config)
-				continue;
-			if (item->value)
-				fprintf(fp, "\t%s = %s\n",
-					item->name, item->value);
-		}
-	}
-	fclose(fp);
-
-	return 0;
+	return perf_config_set__write(set, file_name, use_system_config);
 }
 
 static int show_spec_config(struct perf_config_set *set, const char *var)
@@ -158,44 +128,6 @@ static int parse_config_arg(char *arg, char **var, char **value)
 	return 0;
 }
 
-int perf_config__set_variable(const char *var, const char *value)
-{
-	char path[PATH_MAX];
-	char *user_config = mkpath(path, sizeof(path), "%s/.perfconfig", getenv("HOME"));
-	const char *config_filename;
-	struct perf_config_set *set;
-	int ret = -1;
-
-	if (use_system_config)
-		config_exclusive_filename = perf_etc_perfconfig();
-	else if (use_user_config)
-		config_exclusive_filename = user_config;
-
-	if (!config_exclusive_filename)
-		config_filename = user_config;
-	else
-		config_filename = config_exclusive_filename;
-
-	set = perf_config_set__new();
-	if (!set)
-		goto out_err;
-
-	if (perf_config_set__collect(set, config_filename, var, value) < 0) {
-		pr_err("Failed to add '%s=%s'\n", var, value);
-		goto out_err;
-	}
-
-	if (set_config(set, config_filename) < 0) {
-		pr_err("Failed to set the configs on %s\n", config_filename);
-		goto out_err;
-	}
-
-	ret = 0;
-out_err:
-	perf_config_set__delete(set);
-	return ret;
-}
-
 int cmd_config(int argc, const char **argv)
 {
 	int i, ret = -1;
tools/perf/util/config.c +139 -5

185 lines not on this page · raw message

tools/perf/util/config.h +2 -0
index 987b47cf54c350ba..9098f8a045850c97 100644
@@ -33,6 +33,8 @@ int perf_config_scan(const char *name, const char *fmt, ...) __scanf(2, 3);
 const char *perf_config_get(const char *name);
 int perf_config_set(struct perf_config_set *set,
 		    config_fn_t fn, void *data);
+int perf_config_set__write(struct perf_config_set *set,
+			   const char *file_name, bool system_config);
 int perf_config_int(int *dest, const char *, const char *);
 int perf_config_u8(u8 *dest, const char *name, const char *value);
 int perf_config_u64(u64 *dest, const char *, const char *);
Signature · 1 line
2.55.0
#2 lore raw

Re: [PATCH 1/5] perf config: Move perf_config__set_variable() to util/config.c

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] perf config: data race on config_file_name causing NULL dereference in bad_config() - [Medium]…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf config: data race on config_file_name causing NULL dereference in bad_config()
- [Medium] perf config: memory leak in perf_home_perfconfig() and perf_config__init() due to concurrent access --

commit 6feab70cc124ed967d7949683aa5b69b49d7b578
Author: Arnaldo Carvalho de Melo <[email protected]>

perf config: Move perf_config__set_variable() to util/config.c

Move perf_config__set_variable() out of the perf config builtin so that opt-in features can persist their choice from outside it. All config file access shares the static parser state and can now run on more than one thread, so parsing and rewriting are serialized with a mutex.

Quoted 5 lines
tools/perf/util/config.c +0 -0
index 8fe43b032e9af..b33964be0c14a 100644

tools/perf/util/config.c int perf_default_config(const char *var, const char *value, · line 560

Quoted 10 lines
tools/perf/util/config.c +7 -0
@@ -549,11 +551,26 @@ int perf_default_config(const char *var, const char *value,
 	return 0;
 }
 
+/*
+ * Serialize config file access: parsing and rewriting share the static
+ * parser state and can run on more than one thread, the debuginfod
+ * fetch writing core.debuginfod=false while perf top reads it.
+ */
+static pthread_mutex_t config_mutex = PTHREAD_MUTEX_INITIALIZER;
+

[Severity: Medium]
Since this patch enables multithreaded access to the configuration state, can concurrent calls to lockless lazy initialization functions leak memory?

For example, in perf_home_perfconfig():

tools/perf/util/config.c:perf_home_perfconfig() {
	static const char *config;
	static bool failed;

	if (failed || config)
		return config;

	config = home_perfconfig();
	...
}

If two threads evaluate config as NULL simultaneously, could they both call home_perfconfig() and overwrite the first thread's allocation, leaking the allocated config path string?

Similarly, in perf_config__init():

tools/perf/util/config.c:perf_config__init() {
	if (config_set == NULL)
		config_set = perf_config_set__new();
	...
}

Could concurrent threads duplicate the parsing and leak the resulting global configuration set structure?

[ ... ]

Quoted 5 lines
tools/perf/util/config.c +1 -0
@@ -783,8 +819,15 @@ static int collect_config(const char *var, const char *value,
 int perf_config_set__collect(struct perf_config_set *set, const char *file_name,
 			     const char *var, const char *value)
 {
+	int ret;

tools/perf/util/config.c static int collect_config(const char *var, const char *value, · line 831

Quoted 10 lines
tools/perf/util/config.c +7 -1
+
+	pthread_mutex_lock(&config_mutex);
 	config_file_name = file_name;
-	return collect_config(var, value, set);
+	ret = collect_config(var, value, set);
+		/* Don't leave the static parser state pointing at the caller's buffer. */
+	config_file_name = NULL;
+	pthread_mutex_unlock(&config_mutex);
+	return ret;
 }

[Severity: Medium]
Does this temporary modification of config_file_name introduce a data race with unlocked readers?

While config_mutex is held here, another thread might execute perf_config() and call bad_config() concurrently:

tools/perf/util/config.c:bad_config() {
	if (config_file_name)
		pr_warning("bad config value for '%s' in %s, ignoring...\n",
			   name, config_file_name);
	...
}

Since bad_config() does not hold config_mutex, if thread A evaluates config_file_name as non-NULL, and thread B immediately sets it to NULL in perf_config_set__collect(), could thread A then pass a NULL pointer to the %s format specifier in pr_warning()?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author maintainer
#3 lore raw

PATCH v5 2/5 perf report: Add --progress option

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

Processing a large session with stdio output gives no feedback about which phase perf is in or how far along it is: ui_progress updates are only shown by the TUI. Add --progress, installing a stdio backend (ui/stdio/progress.c) that prints the phase title, percentage and counts:

Processing events... [42.3%] 317M / 746M

Phases can be nested, so the backend tracks the ones started to complete the right one on ui_progress__finish(); that requires init()/finish() pairs, fixed in ordered-events.c and the pipe and directory event processing.

---
 tools/perf/Documentation/perf-report.txt |  14 ++
 tools/perf/builtin-report.c              |  11 ++
 tools/perf/ui/Build                      |   1 +
 tools/perf/ui/progress.h                 |   2 +
 tools/perf/ui/stdio/progress.c           | 162 +++++++++++++++++++++++
 tools/perf/util/ordered-events.c         |  16 ++-
 tools/perf/util/session.c                |  12 +-
 7 files changed, 211 insertions(+), 7 deletions(-)
 create mode 100644 tools/perf/ui/stdio/progress.c
tools/perf/Documentation/perf-report.txt +14 -0
index 3718ebd297ce0673..a7429a30ec28f903 100644
@@ -29,6 +29,20 @@ OPTIONS
 --quiet::
 	Do not show any warnings or messages.  (Suppress -v)
 
+--progress::
+	Show progress for each of the processing phases, printing the
+	percentage done and the current/total counts: the first phase
+	counts the bytes of the perf.data file processed so far, the
+	merge and sort phases count hist entries, e.g.:
+
+	Processing events... [ 42.3%] 4G / 10G
+	Merging related events... [  7.1%] 1024 / 14387
+	Sorting events for output... [ 98.2%] 14132 / 14387
+
+	It is a no-op when using the TUI or GTK browsers, that already
+	present progress information, or when --quiet is used, that asks
+	for no messages at all.
+
 -n::
 --show-nr-samples::
 	Show the number of samples for each symbol
tools/perf/builtin-report.c +11 -0
index 57225bc87731d074..963808b561e547c3 100644
@@ -87,6 +87,7 @@ struct report {
 	bool			use_gtk;
 #endif
 	bool			use_stdio;
+	bool			progress;
 	bool			show_full_info;
 	bool			show_threads;
 	bool			inverted_callchain;
@@ -1384,6 +1385,8 @@ int cmd_report(int argc, const char **argv)
 		    "Use the stdio interface"),
 	OPT_BOOLEAN(0, "weights", &symbol_conf.annotate_weight,
 			"Show or hide weight columns in annotation. Default show if non-zero."),
+	OPT_BOOLEAN(0, "progress", &report.progress,
+		    "Show progress while processing the perf.data file"),
 	OPT_BOOLEAN(0, "header", &report.header, "Show data header."),
 	OPT_BOOLEAN(0, "header-only", &report.header_only,
 		    "Show only data header."),
@@ -1790,6 +1793,14 @@ int cmd_report(int argc, const char **argv)
 	else
 		use_browser = 0;
 
+	/*
+	 * For the stdio case: print the percentage of the perf.data file
+	 * processed so far for each processing phase.  --quiet asks for no
+	 * messages at all, so it leaves the phases uncounted.
+	 */
+	if (report.progress && !quiet && use_browser == 0)
+		stdio_progress__init();
+
 	if (report.data_type && use_browser == 1) {
 		symbol_conf.annotate_data_member = true;
 		symbol_conf.annotate_data_sample = true;
tools/perf/ui/Build +1 -0
index 6005f813c9e3990c..a7b1740d51c80f23 100644
@@ -4,6 +4,7 @@ perf-ui-y += progress.o
 perf-ui-y += util.o
 perf-ui-y += hist.o
 perf-ui-y += stdio/hist.o
+perf-ui-y += stdio/progress.o
 
 CFLAGS_setup.o += -DLIBDIR="BUILD_STR($(LIBDIR))"
 
tools/perf/ui/progress.h +2 -0
index 4f52c37b2f099a82..03f1a8bb260ba076 100644
@@ -23,6 +23,8 @@ void __ui_progress__init(struct ui_progress *p, u64 total,
 
 void ui_progress__update(struct ui_progress *p, u64 adv);
 
+void stdio_progress__init(void);
+
 struct ui_progress_ops {
 	void (*init)(struct ui_progress *p);
 	void (*update)(struct ui_progress *p);
tools/perf/ui/stdio/progress.c new file +162 -0

162 lines not on this page · raw message

tools/perf/util/ordered-events.c +12 -4
index a5857f9f5af2d3de..54c85663e733be4d 100644
@@ -237,14 +237,16 @@ static int do_flush(struct ordered_events *oe, bool show_progress)
 		ui_progress__init(&prog, oe->nr_events, "Processing time ordered events...");
 
 	list_for_each_entry_safe(iter, tmp, head, list) {
-		if (session_done())
-			return 0;
+		if (session_done()) {
+			ret = 0;
+			goto out_progress;
+		}
 
 		if (iter->timestamp > limit)
 			break;
 		ret = oe->deliver(oe, iter);
 		if (ret < 0)
-			return ret;
+			goto out_progress;
 
 		ordered_events__delete(oe, iter);
 		oe->last_flush = iter->timestamp;
@@ -258,10 +260,16 @@ static int do_flush(struct ordered_events *oe, bool show_progress)
 	else if (last_ts <= limit)
 		oe->last = list_entry(head->prev, struct ordered_event, list);
 
+	ret = 0;
+out_progress:
+	/*
+	 * Always pair ui_progress__init() with ui_progress__finish(), the
+	 * stdio backend tracks the phases on a stack.
+	 */
 	if (show_progress)
 		ui_progress__finish();
 
-	return 0;
+	return ret;
 }
 
 static int __ordered_events__flush(struct ordered_events *oe, enum oe_flush how,
tools/perf/util/session.c +9 -3
index 7fea9e72726c936c..c4b4c7fb3b864589 100644
@@ -3253,8 +3253,12 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 	cur_size = sizeof(union perf_event);
 
 	buf = malloc(cur_size);
-	if (!buf)
-		return -errno;
+	if (!buf) {
+		err = -errno;
+		if (update_prog)
+			ui_progress__finish();
+		return err;
+	}
 	ordered_events__set_copy_on_queue(oe, true);
 more:
 	event = buf;
@@ -3753,8 +3757,10 @@ static int __perf_session__process_dir_events(struct perf_session *session)
 	}
 
 	rd = calloc(nr_readers, sizeof(struct reader));
-	if (!rd)
+	if (!rd) {
+		ui_progress__finish();
 		return -ENOMEM;
+	}
 
 	rd[0] = (struct reader) {
 		.fd		 = perf_data__fd(session->data),
Signature · 1 line
2.55.0
#4 lore raw

Re: [PATCH 2/5] perf report: Add --progress option

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines

From: Arnaldo Carvalho de Melo <[email protected]>

Processing a large session with stdio output gives no feedback about which phase perf is in or how far along it is: ui_progress updates are only shown by the TUI. Add --progress, installing a stdio backend (ui/stdio/progress.c) that prints the phase title, percentage and counts:

Processing events... [42.3%] 317M / 746M

Quoted 7 lines

Phases can be nested, so the backend tracks the ones started to complete the right one on ui_progress__finish(); that requires init()/finish() pairs, fixed in ordered-events.c and the pipe and directory event processing.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author maintainer
#5 lore raw

PATCH v5 3/5 perf report: Add --no-progress option

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

Add the counterpart of --progress: the TUI and GTK browsers present progress of their own and there is no way to turn it off, while the option just added allows asking for it when there is none. Install the no-op ui_progress ops, the ones already used until a backend sets theirs, after setup_browser() installed them. The phases are still counted, nothing is shown for them, and --no-progress takes precedence over --progress.

---
 tools/perf/Documentation/perf-report.txt |  5 +++++
 tools/perf/builtin-report.c              | 17 ++++++++++++++---
 tools/perf/ui/progress.c                 |  6 ++++++
 tools/perf/ui/progress.h                 |  2 ++
 4 files changed, 27 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-report.txt +5 -0
index a7429a30ec28f903..bc0f2031b100f425 100644
@@ -43,6 +43,11 @@ OPTIONS
 	present progress information, or when --quiet is used, that asks
 	for no messages at all.
 
+--no-progress::
+	Do not show progress while processing the perf.data file.  It
+	also turns off the progress the TUI and GTK browsers present,
+	which is their own, and takes precedence over --progress.
+
 -n::
 --show-nr-samples::
 	Show the number of samples for each symbol
tools/perf/builtin-report.c +14 -3
index 963808b561e547c3..8d23c077fab5f874 100644
@@ -88,6 +88,7 @@ struct report {
 #endif
 	bool			use_stdio;
 	bool			progress;
+	bool			no_progress;
 	bool			show_full_info;
 	bool			show_threads;
 	bool			inverted_callchain;
@@ -1385,8 +1386,16 @@ int cmd_report(int argc, const char **argv)
 		    "Use the stdio interface"),
 	OPT_BOOLEAN(0, "weights", &symbol_conf.annotate_weight,
 			"Show or hide weight columns in annotation. Default show if non-zero."),
-	OPT_BOOLEAN(0, "progress", &report.progress,
-		    "Show progress while processing the perf.data file"),
+	/*
+	 * Not auto negatable: parse-options turns --no-progress into the
+	 * negation of this option before it ever reaches the --no-progress
+	 * below, which is the one that also turns off the TUI and GTK progress.
+	 */
+	OPT_BOOLEAN_FLAG(0, "progress", &report.progress,
+			 "Show progress while processing the perf.data file",
+			 PARSE_OPT_NOAUTONEG),
+	OPT_BOOLEAN(0, "no-progress", &report.no_progress,
+		    "Do not show progress while processing the perf.data file"),
 	OPT_BOOLEAN(0, "header", &report.header, "Show data header."),
 	OPT_BOOLEAN(0, "header-only", &report.header_only,
 		    "Show only data header."),
@@ -1798,7 +1807,9 @@ int cmd_report(int argc, const char **argv)
 	 * processed so far for each processing phase.  --quiet asks for no
 	 * messages at all, so it leaves the phases uncounted.
 	 */
-	if (report.progress && !quiet && use_browser == 0)
+	if (report.no_progress)
+		ui_progress__noop_init();
+	else if (report.progress && !quiet && use_browser == 0)
 		stdio_progress__init();
 
 	if (report.data_type && use_browser == 1) {
tools/perf/ui/progress.c +6 -0
index 99d60223c74b2957..362680989ace606a 100644
@@ -13,6 +13,12 @@ static struct ui_progress_ops null_progress__ops =
 
 struct ui_progress_ops *ui_progress__ops = &null_progress__ops;
 
+/* Everything counts but nothing is shown, the way it starts out. */
+void ui_progress__noop_init(void)
+{
+	ui_progress__ops = &null_progress__ops;
+}
+
 void ui_progress__update(struct ui_progress *p, u64 adv)
 {
 	u64 last = p->curr;
tools/perf/ui/progress.h +2 -0
index 03f1a8bb260ba076..e8c4f9f768aaf12b 100644
@@ -25,6 +25,8 @@ void ui_progress__update(struct ui_progress *p, u64 adv);
 
 void stdio_progress__init(void);
 
+void ui_progress__noop_init(void);
+
 struct ui_progress_ops {
 	void (*init)(struct ui_progress *p);
 	void (*update)(struct ui_progress *p);
Signature · 1 line
2.55.0
#6 lore raw

Re: [PATCH 3/5] perf report: Add --no-progress option

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines

From: Arnaldo Carvalho de Melo <[email protected]>

Add the counterpart of --progress: the TUI and GTK browsers present progress of their own and there is no way to turn it off, while the option just added allows asking for it when there is none. Install the no-op ui_progress ops, the ones already used until a backend sets theirs, after setup_browser() installed them. The phases are still counted, nothing is shown for them, and --no-progress takes precedence over --progress.

Quoted 3 lines

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
maintainer
#7 lore raw

Re: [PATCH 3/5] perf report: Add --no-progress option

In reply to Arnaldo Carvalho de Melo
Quoted 9 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

Add the counterpart of --progress: the TUI and GTK browsers present progress of their own and there is no way to turn it off, while the option just added allows asking for it when there is none. Install the no-op ui_progress ops, the ones already used until a backend sets theirs, after setup_browser() installed them. The phases are still counted, nothing is shown for them, and --no-progress takes precedence over --progress.

Hmm.. I mean --progress already gives --no-progress for free. We just need to handle that for TUI and GTK.

Thanks,
Namhyung

Quoted 102 lines
---
 tools/perf/Documentation/perf-report.txt |  5 +++++
 tools/perf/builtin-report.c              | 17 ++++++++++++++---
 tools/perf/ui/progress.c                 |  6 ++++++
 tools/perf/ui/progress.h                 |  2 ++
 4 files changed, 27 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-report.txt +5 -0
index a7429a30ec28f903..bc0f2031b100f425 100644
@@ -43,6 +43,11 @@ OPTIONS
 	present progress information, or when --quiet is used, that asks
 	for no messages at all.
 
+--no-progress::
+	Do not show progress while processing the perf.data file.  It
+	also turns off the progress the TUI and GTK browsers present,
+	which is their own, and takes precedence over --progress.
+
 -n::
 --show-nr-samples::
 	Show the number of samples for each symbol
tools/perf/builtin-report.c +14 -3
index 963808b561e547c3..8d23c077fab5f874 100644
@@ -88,6 +88,7 @@ struct report {
 #endif
 	bool			use_stdio;
 	bool			progress;
+	bool			no_progress;
 	bool			show_full_info;
 	bool			show_threads;
 	bool			inverted_callchain;
@@ -1385,8 +1386,16 @@ int cmd_report(int argc, const char **argv)
 		    "Use the stdio interface"),
 	OPT_BOOLEAN(0, "weights", &symbol_conf.annotate_weight,
 			"Show or hide weight columns in annotation. Default show if non-zero."),
-	OPT_BOOLEAN(0, "progress", &report.progress,
-		    "Show progress while processing the perf.data file"),
+	/*
+	 * Not auto negatable: parse-options turns --no-progress into the
+	 * negation of this option before it ever reaches the --no-progress
+	 * below, which is the one that also turns off the TUI and GTK progress.
+	 */
+	OPT_BOOLEAN_FLAG(0, "progress", &report.progress,
+			 "Show progress while processing the perf.data file",
+			 PARSE_OPT_NOAUTONEG),
+	OPT_BOOLEAN(0, "no-progress", &report.no_progress,
+		    "Do not show progress while processing the perf.data file"),
 	OPT_BOOLEAN(0, "header", &report.header, "Show data header."),
 	OPT_BOOLEAN(0, "header-only", &report.header_only,
 		    "Show only data header."),
@@ -1798,7 +1807,9 @@ int cmd_report(int argc, const char **argv)
 	 * processed so far for each processing phase.  --quiet asks for no
 	 * messages at all, so it leaves the phases uncounted.
 	 */
-	if (report.progress && !quiet && use_browser == 0)
+	if (report.no_progress)
+		ui_progress__noop_init();
+	else if (report.progress && !quiet && use_browser == 0)
 		stdio_progress__init();
 
 	if (report.data_type && use_browser == 1) {
tools/perf/ui/progress.c +6 -0
index 99d60223c74b2957..362680989ace606a 100644
@@ -13,6 +13,12 @@ static struct ui_progress_ops null_progress__ops =
 
 struct ui_progress_ops *ui_progress__ops = &null_progress__ops;
 
+/* Everything counts but nothing is shown, the way it starts out. */
+void ui_progress__noop_init(void)
+{
+	ui_progress__ops = &null_progress__ops;
+}
+
 void ui_progress__update(struct ui_progress *p, u64 adv)
 {
 	u64 last = p->curr;
tools/perf/ui/progress.h +2 -0
index 03f1a8bb260ba076..e8c4f9f768aaf12b 100644
@@ -25,6 +25,8 @@ void ui_progress__update(struct ui_progress *p, u64 adv);
 
 void stdio_progress__init(void);
 
+void ui_progress__noop_init(void);
+
 struct ui_progress_ops {
 	void (*init)(struct ui_progress *p);
 	void (*update)(struct ui_progress *p);
Signature · 1 line
2.55.0
author maintainer
#8 lore raw

PATCH v5 4/5 perf scripts: Add perf-stuck, to tell where a running perf is stuck

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

A perf that takes forever is hard to tell apart from one stuck in a loop, and there is no way to see where without attaching gdb. perf-stuck.sh samples a running process' /proc entries and its progress line at a fixed interval, and with -g runs gdb (perf-stuck.gdb, adding the perf-die-chain command) when no progress is made across two samples, printing the DIE chain a DWARF type chase is stuck in.

It is a prototype: the plan is to turn it into a first class 'perf stuck' command. The process name is resolved with pgrep among the caller's own processes only, as root an unscoped one would attach gdb to the first process of any user with a matching name.

---
 tools/perf/scripts/perf-stuck.gdb | 129 ++++++++++++++++++++
 tools/perf/scripts/perf-stuck.sh  | 188 ++++++++++++++++++++++++++++++
 2 files changed, 317 insertions(+)
 create mode 100644 tools/perf/scripts/perf-stuck.gdb
 create mode 100755 tools/perf/scripts/perf-stuck.sh
tools/perf/scripts/perf-stuck.gdb new file +129 -0
new file mode 100644
index 0000000000000000..be7b9e3fcb5d6d98
@@ -0,0 +1,129 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# gdb commands for a stuck perf, used by perf-stuck.sh -g and usable directly:
+#
+#   gdb -p $(pgrep -x perf) -batch -x perf-stuck.gdb -ex bt
+#
+# PROTOTYPE: part of the perf-stuck.sh stopgap, wants to become a first
+# class 'perf stuck' command printing these DIE chains without gdb.
+#
+# The commands are for the DWARF type chasers in util/dwarf-aux.c:
+#
+#   perf-die-chain <function> <die variable> [iterations]
+#   perf-die-chain-all [iterations]
+#   perf-dso
+#
+# For each iteration of the chasing loop they print the DIE address, the
+# CU, its file offset, tag and name: a cycle shows as the same (addr, cu)
+# pairs repeating, and a CU changing between iterations means the chase
+# hops between a debug file and its dwz common file.
+
+set pagination off
+set confirm off
+set debuginfod enabled off
+set print pretty on
+set height 0
+set width 0
+
+define perf-die-chain
+  if $argc < 2
+    printf "usage: perf-die-chain <function> <die variable> [iterations]\n"
+  else
+    frame function $arg0
+    if $argc == 3
+      set $perf_die_chain_n = $arg2
+    else
+      set $perf_die_chain_n = 10
+    end
+    set $perf_die_chain_head = $pc
+    set $perf_die_chain_i = 0
+    while $perf_die_chain_i < $perf_die_chain_n
+      # Pointer type DIEs have no DW_AT_name, so dwarf_diename() can
+      # return NULL: printf %s of it would error out and abort this
+      # batch script, handle it.
+      set $perf_die_chain_name = (char *) dwarf_diename($arg1)
+      printf "chain[%d] die=%p addr=%p cu=%p off=0x%lx tag=%d name=", $perf_die_chain_i, $arg1, $arg1->addr, $arg1->cu, ((Dwarf_Off) dwarf_dieoffset($arg1)), ((int) dwarf_tag($arg1))
+      if $perf_die_chain_name == 0
+	printf "(null)\n"
+      else
+	printf "%s\n", $perf_die_chain_name
+      end
+      until *$perf_die_chain_head
+      set $perf_die_chain_i = $perf_die_chain_i + 1
+    end
+  end
+end
+
+document perf-die-chain
+Print the DIE chain being walked by a DWARF type chasing loop.
+usage: perf-die-chain <function> <die variable> [iterations]
+  perf-die-chain die_get_pointer_type type_die
+  perf-die-chain __die_get_real_type vr_die
+  perf-die-chain die_get_real_type vr_die
+end
+
+define perf-die-chain-all
+  if $argc == 0
+    set $perf_die_chain_n = 10
+  else
+    set $perf_die_chain_n = $arg0
+  end
+  if $_any_caller_is("die_get_pointer_type", 20)
+    printf "stuck in die_get_pointer_type():\n"
+    perf-die-chain die_get_pointer_type type_die $perf_die_chain_n
+  else
+    if $_any_caller_is("__die_get_real_type", 20)
+      printf "stuck in __die_get_real_type():\n"
+      perf-die-chain __die_get_real_type vr_die $perf_die_chain_n
+    else
+      if $_any_caller_is("die_get_real_type", 20)
+        printf "stuck in die_get_real_type():\n"
+        perf-die-chain die_get_real_type vr_die $perf_die_chain_n
+      else
+        printf "not in a DWARF type chaser, try: bt\n"
+      end
+    end
+  end
+end
+
+document perf-die-chain-all
+Find which DWARF type chaser the process is in and print the DIE chain.
+usage: perf-die-chain-all [iterations]
+end
+
+define perf-dso
+  if $_any_caller_is("find_data_type", 20)
+    frame function find_data_type
+    # REFCNT_CHECKING, implied by an ASan/LSan build, wraps struct map and
+    # struct dso in a proxy keeping the real object in ->orig, the dso being
+    # map->orig->dso->orig there; struct symbol is not wrapped, so sym->name
+    # needs no ->orig.  There is no way to ask gdb which layout it is looking
+    # at, so walk each candidate until one evaluates.  Requiring gdb's Python
+    # here adds nothing: $_any_caller_is() is one of its functions already.
+    python
+import gdb
+
+dso = None
+for expr in ("dloc->ms->map->dso->name",
+             "dloc->ms->map->orig->dso->orig->name"):
+    try:
+        gdb.parse_and_eval(expr)
+        dso = expr
+        break
+    except gdb.error:
+        pass
+
+if dso is None:
+    print("dso=(no such member in struct map)")
+else:
+    gdb.execute('printf "dso=%s ip=0x%lx sym=%s\\n", ' + dso +
+                ', dloc->ip, dloc->ms->sym ? dloc->ms->sym->name : "(no symbol)"')
+    end
+  else
+    printf "not in find_data_type()\n"
+  end
+end
+
+document perf-dso
+Print the dso, ip and symbol of the data location being resolved.
+end
tools/perf/scripts/perf-stuck.sh new file +188 -0

188 lines not on this page · raw message

Signature · 1 line
2.55.0
#9 lore raw

Re: [PATCH 4/5] perf scripts: Add perf-stuck, to tell where a running perf is stuck

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines

From: Arnaldo Carvalho de Melo <[email protected]>

A perf that takes forever is hard to tell apart from one stuck in a loop, and there is no way to see where without attaching gdb. perf-stuck.sh samples a running process' /proc entries and its progress line at a fixed interval, and with -g runs gdb (perf-stuck.gdb, adding the perf-die-chain command) when no progress is made across two samples, printing the DIE chain a DWARF type chase is stuck in.

Quoted 7 lines

It is a prototype: the plan is to turn it into a first class 'perf stuck' command. The process name is resolved with pgrep among the caller's own processes only, as root an unscoped one would attach gdb to the first process of any user with a matching name.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author maintainer
#10 lore raw

PATCH v5 5/5 perf test: Add false_sharing workload exhibiting cross-CPU false sharing

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

Add a 'perf test -w false_sharing' workload that hammers one shared struct from several CPUs, shaped as a TCP connection: a read-mostly identity (five-tuple) shares a cacheline with per-packet rx counters (the false-sharing line), a second line has packet-path private tx and congestion control counters, and a third the connection config.

The packet path runs in the main thread and up to four lookup threads hash the five-tuple and pull the config, reading one volatile shared instance directly so the accesses are PC-relative and resolvable by the data type profiler.

---
 tools/perf/tests/builtin-test.c               |   1 +
 tools/perf/tests/shell/data_type_profiling.sh |   9 +-
 tools/perf/tests/tests.h                      |   1 +
 tools/perf/tests/workloads/Build              |   2 +
 tools/perf/tests/workloads/false_sharing.c    | 250 ++++++++++++++++++
 5 files changed, 261 insertions(+), 2 deletions(-)
 create mode 100644 tools/perf/tests/workloads/false_sharing.c
tools/perf/tests/builtin-test.c +1 -0
index d2f594921e25bda9..98134b1c74cf80f4 100644
@@ -175,6 +175,7 @@ static struct test_workload *workloads[] = {
 	&workload__context_switch_loop,
 	&workload__deterministic,
 	&workload__callchain,
+	&workload__false_sharing,
 
 #ifdef HAVE_RUST_SUPPORT
 	&workload__code_with_type,
tools/perf/tests/shell/data_type_profiling.sh +7 -2
index a916c410274aa888..57203e0e5f857540 100755
@@ -8,8 +8,8 @@ set -e
 # data type profiling manifestation
 
 # Values in testtypes and testprogs should match
-testtypes=("# data-type: struct Buf" "# data-type: struct buf")
-testprogs=("perf test -w code_with_type" "perf test -w datasym")
+testtypes=("# data-type: struct Buf" "# data-type: struct buf" "# data-type: struct net_conn")
+testprogs=("perf test -w code_with_type" "perf test -w datasym" "perf test -w false_sharing")
 
 err=0
 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
@@ -59,6 +59,9 @@ test_basic_annotate() {
 
     "xC")
     index=1 ;;
+
+    "xFS")
+    index=2 ;;
   esac
 
   # Under 'set -e' a bare failing command aborts the script through the EXIT
@@ -114,6 +117,8 @@ test_basic_annotate Basic Rust
 test_basic_annotate Pipe Rust
 test_basic_annotate Basic C
 test_basic_annotate Pipe C
+test_basic_annotate Basic FS
+test_basic_annotate Pipe FS
 
 cleanup
 exit $err
tools/perf/tests/tests.h +1 -0
index 9c96f33483d14356..f379620069d34f3a 100644
@@ -250,6 +250,7 @@ DECLARE_WORKLOAD(jitdump);
 DECLARE_WORKLOAD(context_switch_loop);
 DECLARE_WORKLOAD(deterministic);
 DECLARE_WORKLOAD(callchain);
+DECLARE_WORKLOAD(false_sharing);
 
 #ifdef HAVE_RUST_SUPPORT
 DECLARE_WORKLOAD(code_with_type);
tools/perf/tests/workloads/Build +2 -0
index 048e371eb63e3164..18fceddccb56c013 100644
@@ -14,6 +14,7 @@ perf-test-y += jitdump.o
 perf-test-y += context_switch_loop.o
 perf-test-y += deterministic.o
 perf-test-y += callchain.o
+perf-test-y += false_sharing.o
 
 ifeq ($(CONFIG_RUST_SUPPORT),y)
     perf-test-y += code_with_type.o
@@ -29,3 +30,4 @@ CFLAGS_inlineloop.o       = -g -O2
 CFLAGS_deterministic.o    = -g -O0 -fno-inline -U_FORTIFY_SOURCE
 CFLAGS_named_threads.o    = -g -O0 -fno-inline -U_FORTIFY_SOURCE
 CFLAGS_callchain.o        = -g -O0 -fno-inline -U_FORTIFY_SOURCE
+CFLAGS_false_sharing.o    = -g -O0 -fno-inline -U_FORTIFY_SOURCE
tools/perf/tests/workloads/false_sharing.c new file +250 -0

250 lines not on this page · raw message

Signature · 1 line
2.55.0
#11 lore raw

Re: [PATCH 5/5] perf test: Add false_sharing workload exhibiting cross-CPU false sharing

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 8 lines

From: Arnaldo Carvalho de Melo <[email protected]>

Add a 'perf test -w false_sharing' workload that hammers one shared struct from several CPUs, shaped as a TCP connection: a read-mostly identity (five-tuple) shares a cacheline with per-packet rx counters (the false-sharing line), a second line has packet-path private tx and congestion control counters, and a third the connection config.

Quoted 7 lines

The packet path runs in the main thread and up to four lookup threads hash the five-tuple and pull the config, reading one volatile shared instance directly so the accesses are PC-relative and resolvable by the data type profiler.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5