Skip to contentSkip to conversation

Support metric group constraint

PATCH Applied [email protected] 4 participants 14 messages Feb 24, 2020 – Mar 19, 2020 perf events

lore
Share & tools
lore
#0 lore raw

v2 cover Support metric group constraint

From: Kan Liang <[email protected]>

Changes since V1:
- Remove global static flag violate_nmi_constraint, and add a new function metricgroup___watchdog_constraint_hint() for all watchdog constraint hints in patch 4.
The rest of the patches are not changed.

Some metric groups, e.g. Page_Walks_Utilization, will never count when NMI watchdog is enabled.

$echo 1 > /proc/sys/kernel/nmi_watchdog
$perf stat -M Page_Walks_Utilization

Performance counter stats for 'system wide':

 <not counted>      itlb_misses.walk_pending       (0.00%)
 <not counted>      dtlb_load_misses.walk_pending  (0.00%)
 <not counted>      dtlb_store_misses.walk_pending (0.00%)
 <not counted>      ept.walk_pending               (0.00%)
 <not counted>      cycles                         (0.00%)
       2.343460588 seconds time elapsed

Some events weren't counted. Try disabling the NMI watchdog:
echo 0 > /proc/sys/kernel/nmi_watchdog
perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog
The events in group usually have to be from the same PMU. Try reorganizing the group.

A metric group is a weak group, which relies on group validation code in the kernel to determine whether to be opened as a group or a non-group. However, group validation code may return false-positives, especially when NMI watchdog is enabled. (The metric group is allowed as a group but will never be scheduled.)

The attempt to fix the group validation code has been rejected.
https://lore.kernel.org/lkml/[email protected]/
Because we cannot accurately predict whether the group can be scheduled as a group, only by checking current status.

This patch set provides another solution to mitigate the issue. Add "MetricConstraint" in event list, which provides a hint for perf tool, e.g. "MetricConstraint": "NO_NMI_WATCHDOG". Perf tool can change the metric group to non-group (standalone metrics) if NMI watchdog is enabled.

After applying the patch,

$echo 1 > /proc/sys/kernel/nmi_watchdog
$perf stat -M Page_Walks_Utilization
Splitting metric group Page_Walks_Utilization into standalone metrics. Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:
echo 0 > /proc/sys/kernel/nmi_watchdog
perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog

Performance counter stats for 'system wide':

        18,253,454      itlb_misses.walk_pending  #      0.0
                              Page_Walks_Utilization   (50.55%)
        78,051,525      dtlb_load_misses.walk_pending  (50.55%)
        29,213,063      dtlb_store_misses.walk_pending (50.55%)
                 0      ept.walk_pending               (50.55%)
     2,542,132,364      cycles                         (49.92%)

       1.037095993 seconds time elapsed

Kan Liang (5):
perf jevents: Support metric constraint
perf metricgroup: Factor out metricgroup__add_metric_weak_group() perf util: Factor out sysctl__nmi_watchdog_enabled() perf metricgroup: Support metric constraint
perf vendor events: Add NO_NMI_WATCHDOG metric constraint

 .../arch/x86/cascadelakex/clx-metrics.json         |   3 +-
 .../pmu-events/arch/x86/skylake/skl-metrics.json   |   3 +-
 .../pmu-events/arch/x86/skylakex/skx-metrics.json  |   3 +-
 tools/perf/pmu-events/jevents.c                    |  19 ++--
 tools/perf/pmu-events/jevents.h                    |   2 +-
 tools/perf/pmu-events/pmu-events.h                 |   1 +
 tools/perf/util/metricgroup.c                      | 109 ++++++++++++++++-----
 tools/perf/util/stat-display.c                     |   6 +-
 tools/perf/util/util.c                             |  18 ++++
 tools/perf/util/util.h                             |   2 +
 10 files changed, 128 insertions(+), 38 deletions(-)
Signature · 1 line
2.7.4
#3 lore raw

PATCH v2 2/5 perf metricgroup: Factor out metricgroup__add_metric_weak_group()

From: Kan Liang <[email protected]>

Factor out metricgroup__add_metric_weak_group() which add metrics into a weak group. The change can improve code readability. Because following patch will introduce a function which add standalone metrics.

---
 tools/perf/util/metricgroup.c | 57 +++++++++++++++++++++++++------------------
 1 file changed, 33 insertions(+), 24 deletions(-)
tools/perf/util/metricgroup.c +33 -24
index 02aee94..1cd042c 100644
@@ -399,13 +399,42 @@ void metricgroup__print(bool metrics, bool metricgroups, char *filter,
 	strlist__delete(metriclist);
 }
 
+static void metricgroup__add_metric_weak_group(struct strbuf *events,
+					       const char **ids,
+					       int idnum)
+{
+	bool no_group = false;
+	int i;
+
+	for (i = 0; i < idnum; i++) {
+		pr_debug("found event %s\n", ids[i]);
+		/*
+		 * Duration time maps to a software event and can make
+		 * groups not count. Always use it outside a
+		 * group.
+		 */
+		if (!strcmp(ids[i], "duration_time")) {
+			if (i > 0)
+				strbuf_addf(events, "}:W,");
+			strbuf_addf(events, "duration_time");
+			no_group = true;
+			continue;
+		}
+		strbuf_addf(events, "%s%s",
+			i == 0 || no_group ? "{" : ",",
+			ids[i]);
+		no_group = false;
+	}
+	if (!no_group)
+		strbuf_addf(events, "}:W");
+}
+
 static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 				   struct list_head *group_list)
 {
 	struct pmu_events_map *map = perf_pmu__find_map(NULL);
 	struct pmu_event *pe;
-	int ret = -EINVAL;
-	int i, j;
+	int i, ret = -EINVAL;
 
 	if (!map)
 		return 0;
@@ -422,7 +451,6 @@ static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 			const char **ids;
 			int idnum;
 			struct egroup *eg;
-			bool no_group = false;
 
 			pr_debug("metric expr %s for %s\n", pe->metric_expr, pe->metric_name);
 
@@ -431,27 +459,8 @@ static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 				continue;
 			if (events->len > 0)
 				strbuf_addf(events, ",");
-			for (j = 0; j < idnum; j++) {
-				pr_debug("found event %s\n", ids[j]);
-				/*
-				 * Duration time maps to a software event and can make
-				 * groups not count. Always use it outside a
-				 * group.
-				 */
-				if (!strcmp(ids[j], "duration_time")) {
-					if (j > 0)
-						strbuf_addf(events, "}:W,");
-					strbuf_addf(events, "duration_time");
-					no_group = true;
-					continue;
-				}
-				strbuf_addf(events, "%s%s",
-					j == 0 || no_group ? "{" : ",",
-					ids[j]);
-				no_group = false;
-			}
-			if (!no_group)
-				strbuf_addf(events, "}:W");
+
+			metricgroup__add_metric_weak_group(events, ids, idnum);
 
 			eg = malloc(sizeof(struct egroup));
 			if (!eg) {
Signature · 1 line
2.7.4
#10 lore raw

PATCH v2 5/5 perf vendor events: Add NO_NMI_WATCHDOG metric constraint

In reply to [email protected]

From: Kan Liang <[email protected]>

Add NO_NMI_WATCHDOG metric constraint to Page_Walks_Utilization for Sky Lake and Cascade Lake.

---
 tools/perf/pmu-events/arch/x86/cascadelakex/clx-metrics.json | 3 ++-
 tools/perf/pmu-events/arch/x86/skylake/skl-metrics.json      | 3 ++-
 tools/perf/pmu-events/arch/x86/skylakex/skx-metrics.json     | 3 ++-
 3 files changed, 6 insertions(+), 3 deletions(-)
tools/perf/pmu-events/arch/x86/cascadelakex/clx-metrics.json +2 -1
index f946532..a728c6e 100644
@@ -215,7 +215,8 @@
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
         "MetricExpr": "( ITLB_MISSES.WALK_PENDING + DTLB_LOAD_MISSES.WALK_PENDING + DTLB_STORE_MISSES.WALK_PENDING + EPT.WALK_PENDING ) / ( 2 * cycles )",
         "MetricGroup": "TLB",
-        "MetricName": "Page_Walks_Utilization"
+        "MetricName": "Page_Walks_Utilization",
+        "MetricConstraint": "NO_NMI_WATCHDOG"
     },
     {
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
tools/perf/pmu-events/arch/x86/skylake/skl-metrics.json +2 -1
index e7feb60..f97e831 100644
@@ -215,7 +215,8 @@
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
         "MetricExpr": "( ITLB_MISSES.WALK_PENDING + DTLB_LOAD_MISSES.WALK_PENDING + DTLB_STORE_MISSES.WALK_PENDING + EPT.WALK_PENDING ) / ( 2 * cycles )",
         "MetricGroup": "TLB",
-        "MetricName": "Page_Walks_Utilization"
+        "MetricName": "Page_Walks_Utilization",
+        "MetricConstraint": "NO_NMI_WATCHDOG"
     },
     {
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
tools/perf/pmu-events/arch/x86/skylakex/skx-metrics.json +2 -1
index 21d7a0c..35f5db1 100644
@@ -215,7 +215,8 @@
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
         "MetricExpr": "( ITLB_MISSES.WALK_PENDING + DTLB_LOAD_MISSES.WALK_PENDING + DTLB_STORE_MISSES.WALK_PENDING + EPT.WALK_PENDING ) / ( 2 * cycles )",
         "MetricGroup": "TLB",
-        "MetricName": "Page_Walks_Utilization"
+        "MetricName": "Page_Walks_Utilization",
+        "MetricConstraint": "NO_NMI_WATCHDOG"
     },
     {
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
Signature · 1 line
2.7.4
#8 lore raw

PATCH v2 4/5 perf metricgroup: Support metric constraint

In reply to [email protected]

From: Kan Liang <[email protected]>

Some metric groups have metric constraints. A metric group can be scheduled as a group only when some constraints are applied. For example, Page_Walks_Utilization has a metric constraint, "NO_NMI_WATCHDOG".
When NMI watchdog is disabled, the metric group can be scheduled as a group. Otherwise, splitting the metric group into standalone metrics.

Add a new function, metricgroup__has_constraint(), to check whether all constraints are applied. If not, splitting the metric group into standalone metrics.

Currently, only one constraint, "NO_NMI_WATCHDOG", is checked. Print a warning for the metric group with the constraint, when NMI WATCHDOG is enabled.

---
 tools/perf/util/metricgroup.c | 54 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 1 deletion(-)
tools/perf/util/metricgroup.c +53 -1
index 1cd042c..c3a8c70 100644
@@ -22,6 +22,8 @@
 #include <linux/string.h>
 #include <linux/zalloc.h>
 #include <subcmd/parse-options.h>
+#include <api/fs/fs.h>
+#include "util.h"
 
 struct metric_event *metricgroup__lookup(struct rblist *metric_events,
 					 struct evsel *evsel,
@@ -429,6 +431,49 @@ static void metricgroup__add_metric_weak_group(struct strbuf *events,
 		strbuf_addf(events, "}:W");
 }
 
+static void metricgroup__add_metric_non_group(struct strbuf *events,
+					      const char **ids,
+					      int idnum)
+{
+	int i;
+
+	for (i = 0; i < idnum; i++)
+		strbuf_addf(events, ",%s", ids[i]);
+}
+
+static void metricgroup___watchdog_constraint_hint(const char *name, bool foot)
+{
+	static bool violate_nmi_constraint;
+
+	if (!foot) {
+		pr_warning("Splitting metric group %s into standalone metrics.\n", name);
+		violate_nmi_constraint = true;
+		return;
+	}
+
+	if (!violate_nmi_constraint)
+		return;
+
+	pr_warning("Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:\n"
+		   "    echo 0 > /proc/sys/kernel/nmi_watchdog\n"
+		   "    perf stat ...\n"
+		   "    echo 1 > /proc/sys/kernel/nmi_watchdog\n");
+}
+
+static bool metricgroup__has_constraint(struct pmu_event *pe)
+{
+	if (!pe->metric_constraint)
+		return false;
+
+	if (!strcmp(pe->metric_constraint, "NO_NMI_WATCHDOG") &&
+	    sysctl__nmi_watchdog_enabled()) {
+		metricgroup___watchdog_constraint_hint(pe->metric_name, false);
+		return true;
+	}
+
+	return false;
+}
+
 static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 				   struct list_head *group_list)
 {
@@ -460,7 +505,10 @@ static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 			if (events->len > 0)
 				strbuf_addf(events, ",");
 
-			metricgroup__add_metric_weak_group(events, ids, idnum);
+			if (metricgroup__has_constraint(pe))
+				metricgroup__add_metric_non_group(events, ids, idnum);
+			else
+				metricgroup__add_metric_weak_group(events, ids, idnum);
 
 			eg = malloc(sizeof(struct egroup));
 			if (!eg) {
@@ -502,6 +550,10 @@ static int metricgroup__add_metric_list(const char *list, struct strbuf *events,
 		}
 	}
 	free(nlist);
+
+	if (!ret)
+		metricgroup___watchdog_constraint_hint(NULL, true);
+
 	return ret;
 }
 
Signature · 1 line
2.7.4
#6 lore raw

PATCH v2 3/5 perf util: Factor out sysctl__nmi_watchdog_enabled()

In reply to [email protected]

From: Kan Liang <[email protected]>

NMI watchdog status is required for metric group constraint examination. Factor out sysctl__nmi_watchdog_enabled() to retrieve the NMI watchdog status.

Users may count more than one metric groups each time. If so, the NMI watchdog status may be retrieved several times. To reduce the overhead, cache the NMI watchdog status.

Replace the NMI watchdog status checking in print_footer() by sysctl__nmi_watchdog_enabled().

---
 tools/perf/util/stat-display.c |  6 ++----
 tools/perf/util/util.c         | 18 ++++++++++++++++++
 tools/perf/util/util.h         |  2 ++
 3 files changed, 22 insertions(+), 4 deletions(-)
tools/perf/util/stat-display.c +2 -4
index bc31fcc..16efdba 100644
@@ -16,6 +16,7 @@
 #include <linux/ctype.h>
 #include "cgroup.h"
 #include <api/fs/fs.h>
+#include "util.h"
 
 #define CNTR_NOT_SUPPORTED	"<not supported>"
 #define CNTR_NOT_COUNTED	"<not counted>"
@@ -1097,7 +1098,6 @@ static void print_footer(struct perf_stat_config *config)
 {
 	double avg = avg_stats(config->walltime_nsecs_stats) / NSEC_PER_SEC;
 	FILE *output = config->output;
-	int n;
 
 	if (!config->null_run)
 		fprintf(output, "\n");
@@ -1131,9 +1131,7 @@ static void print_footer(struct perf_stat_config *config)
 	}
 	fprintf(output, "\n\n");
 
-	if (config->print_free_counters_hint &&
-	    sysctl__read_int("kernel/nmi_watchdog", &n) >= 0 &&
-	    n > 0)
+	if (config->print_free_counters_hint && sysctl__nmi_watchdog_enabled())
 		fprintf(output,
 "Some events weren't counted. Try disabling the NMI watchdog:\n"
 "	echo 0 > /proc/sys/kernel/nmi_watchdog\n"
tools/perf/util/util.c +18 -0
index 969ae56..d707c96 100644
@@ -55,6 +55,24 @@ int sysctl__max_stack(void)
 	return sysctl_perf_event_max_stack;
 }
 
+bool sysctl__nmi_watchdog_enabled(void)
+{
+	static bool cached;
+	static bool nmi_watchdog;
+	int value;
+
+	if (cached)
+		return nmi_watchdog;
+
+	if (sysctl__read_int("kernel/nmi_watchdog", &value) < 0)
+		return false;
+
+	nmi_watchdog = (value > 0) ? true : false;
+	cached = true;
+
+	return nmi_watchdog;
+}
+
 bool test_attr__enabled;
 
 bool perf_host  = true;
tools/perf/util/util.h +2 -0
index 9969b8b..f486fdd 100644
@@ -29,6 +29,8 @@ size_t hex_width(u64 v);
 
 int sysctl__max_stack(void);
 
+bool sysctl__nmi_watchdog_enabled(void);
+
 int fetch_kernel_version(unsigned int *puint,
 			 char *str, size_t str_sz);
 #define KVER_VERSION(x)		(((x) >> 16) & 0xff)
Signature · 1 line
2.7.4
#1 lore raw

PATCH v2 1/5 perf jevents: Support metric constraint

In reply to [email protected]

From: Kan Liang <[email protected]>

A new field "MetricConstraint" is introduced in JSON event list.

Extend jevents to parse the field and save the value in metric_constraint.

---
 tools/perf/pmu-events/jevents.c    | 19 +++++++++++++------
 tools/perf/pmu-events/jevents.h    |  2 +-
 tools/perf/pmu-events/pmu-events.h |  1 +
 3 files changed, 15 insertions(+), 7 deletions(-)
tools/perf/pmu-events/jevents.c +13 -6
index 079c77b..6d0f61f 100644
@@ -323,7 +323,7 @@ static int print_events_table_entry(void *data, char *name, char *event,
 				    char *pmu, char *unit, char *perpkg,
 				    char *metric_expr,
 				    char *metric_name, char *metric_group,
-				    char *deprecated)
+				    char *deprecated, char *metric_constraint)
 {
 	struct perf_entry_data *pd = data;
 	FILE *outfp = pd->outfp;
@@ -357,6 +357,8 @@ static int print_events_table_entry(void *data, char *name, char *event,
 		fprintf(outfp, "\t.metric_group = \"%s\",\n", metric_group);
 	if (deprecated)
 		fprintf(outfp, "\t.deprecated = \"%s\",\n", deprecated);
+	if (metric_constraint)
+		fprintf(outfp, "\t.metric_constraint = \"%s\",\n", metric_constraint);
 	fprintf(outfp, "},\n");
 
 	return 0;
@@ -375,6 +377,7 @@ struct event_struct {
 	char *metric_name;
 	char *metric_group;
 	char *deprecated;
+	char *metric_constraint;
 };
 
 #define ADD_EVENT_FIELD(field) do { if (field) {		\
@@ -422,7 +425,7 @@ static int save_arch_std_events(void *data, char *name, char *event,
 				char *desc, char *long_desc, char *pmu,
 				char *unit, char *perpkg, char *metric_expr,
 				char *metric_name, char *metric_group,
-				char *deprecated)
+				char *deprecated, char *metric_constraint)
 {
 	struct event_struct *es;
 
@@ -486,7 +489,7 @@ try_fixup(const char *fn, char *arch_std, char **event, char **desc,
 	  char **name, char **long_desc, char **pmu, char **filter,
 	  char **perpkg, char **unit, char **metric_expr, char **metric_name,
 	  char **metric_group, unsigned long long eventcode,
-	  char **deprecated)
+	  char **deprecated, char **metric_constraint)
 {
 	/* try to find matching event from arch standard values */
 	struct event_struct *es;
@@ -515,7 +518,7 @@ int json_events(const char *fn,
 		      char *pmu, char *unit, char *perpkg,
 		      char *metric_expr,
 		      char *metric_name, char *metric_group,
-		      char *deprecated),
+		      char *deprecated, char *metric_constraint),
 	  void *data)
 {
 	int err;
@@ -545,6 +548,7 @@ int json_events(const char *fn,
 		char *metric_name = NULL;
 		char *metric_group = NULL;
 		char *deprecated = NULL;
+		char *metric_constraint = NULL;
 		char *arch_std = NULL;
 		unsigned long long eventcode = 0;
 		struct msrmap *msr = NULL;
@@ -629,6 +633,8 @@ int json_events(const char *fn,
 				addfield(map, &metric_name, "", "", val);
 			} else if (json_streq(map, field, "MetricGroup")) {
 				addfield(map, &metric_group, "", "", val);
+			} else if (json_streq(map, field, "MetricConstraint")) {
+				addfield(map, &metric_constraint, "", "", val);
 			} else if (json_streq(map, field, "MetricExpr")) {
 				addfield(map, &metric_expr, "", "", val);
 				for (s = metric_expr; *s; s++)
@@ -670,13 +676,13 @@ int json_events(const char *fn,
 					&long_desc, &pmu, &filter, &perpkg,
 					&unit, &metric_expr, &metric_name,
 					&metric_group, eventcode,
-					&deprecated);
+					&deprecated, &metric_constraint);
 			if (err)
 				goto free_strings;
 		}
 		err = func(data, name, real_event(name, event), desc, long_desc,
 			   pmu, unit, perpkg, metric_expr, metric_name,
-			   metric_group, deprecated);
+			   metric_group, deprecated, metric_constraint);
 free_strings:
 		free(event);
 		free(desc);
@@ -691,6 +697,7 @@ int json_events(const char *fn,
 		free(metric_expr);
 		free(metric_name);
 		free(metric_group);
+		free(metric_constraint);
 		free(arch_std);
 
 		if (err)
tools/perf/pmu-events/jevents.h +1 -1
index 5cda49a4..2afc830 100644
@@ -8,7 +8,7 @@ int json_events(const char *fn,
 				char *pmu,
 				char *unit, char *perpkg, char *metric_expr,
 				char *metric_name, char *metric_group,
-				char *deprecated),
+				char *deprecated, char *metric_constraint),
 		void *data);
 char *get_cpu_str(void);
 
tools/perf/pmu-events/pmu-events.h +1 -0
index caeb577..53e76d5 100644
@@ -18,6 +18,7 @@ struct pmu_event {
 	const char *metric_name;
 	const char *metric_group;
 	const char *deprecated;
+	const char *metric_constraint;
 };
 
 /*
Signature · 1 line
2.7.4
Acked-by on the v2 series
#12 lore raw
In reply to [email protected]
Quoted 7 lines · On Mon, Feb 24, 2020 at 01:59:19PM -0800, [email protected] wrote:

From: Kan Liang <[email protected]>

Changes since V1:
- Remove global static flag violate_nmi_constraint, and add a new function metricgroup___watchdog_constraint_hint() for all watchdog constraint hints in patch 4.
The rest of the patches are not changed.

thanks,
jirka

Quoted 83 lines

Some metric groups, e.g. Page_Walks_Utilization, will never count when NMI watchdog is enabled.

$echo 1 > /proc/sys/kernel/nmi_watchdog
$perf stat -M Page_Walks_Utilization

Performance counter stats for 'system wide':

 <not counted>      itlb_misses.walk_pending       (0.00%)
 <not counted>      dtlb_load_misses.walk_pending  (0.00%)
 <not counted>      dtlb_store_misses.walk_pending (0.00%)
 <not counted>      ept.walk_pending               (0.00%)
 <not counted>      cycles                         (0.00%)
       2.343460588 seconds time elapsed

Some events weren't counted. Try disabling the NMI watchdog:
echo 0 > /proc/sys/kernel/nmi_watchdog
perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog
The events in group usually have to be from the same PMU. Try reorganizing the group.

A metric group is a weak group, which relies on group validation code in the kernel to determine whether to be opened as a group or a non-group. However, group validation code may return false-positives, especially when NMI watchdog is enabled. (The metric group is allowed as a group but will never be scheduled.)

The attempt to fix the group validation code has been rejected.
https://lore.kernel.org/lkml/[email protected]/
Because we cannot accurately predict whether the group can be scheduled as a group, only by checking current status.

This patch set provides another solution to mitigate the issue. Add "MetricConstraint" in event list, which provides a hint for perf tool, e.g. "MetricConstraint": "NO_NMI_WATCHDOG". Perf tool can change the metric group to non-group (standalone metrics) if NMI watchdog is enabled.

After applying the patch,

$echo 1 > /proc/sys/kernel/nmi_watchdog
$perf stat -M Page_Walks_Utilization
Splitting metric group Page_Walks_Utilization into standalone metrics. Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:
echo 0 > /proc/sys/kernel/nmi_watchdog
perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog

Performance counter stats for 'system wide':

        18,253,454      itlb_misses.walk_pending  #      0.0
                              Page_Walks_Utilization   (50.55%)
        78,051,525      dtlb_load_misses.walk_pending  (50.55%)
        29,213,063      dtlb_store_misses.walk_pending (50.55%)
                 0      ept.walk_pending               (50.55%)
     2,542,132,364      cycles                         (49.92%)

       1.037095993 seconds time elapsed

Kan Liang (5):
perf jevents: Support metric constraint
perf metricgroup: Factor out metricgroup__add_metric_weak_group() perf util: Factor out sysctl__nmi_watchdog_enabled() perf metricgroup: Support metric constraint
perf vendor events: Add NO_NMI_WATCHDOG metric constraint

 .../arch/x86/cascadelakex/clx-metrics.json         |   3 +-
 .../pmu-events/arch/x86/skylake/skl-metrics.json   |   3 +-
 .../pmu-events/arch/x86/skylakex/skx-metrics.json  |   3 +-
 tools/perf/pmu-events/jevents.c                    |  19 ++--
 tools/perf/pmu-events/jevents.h                    |   2 +-
 tools/perf/pmu-events/pmu-events.h                 |   1 +
 tools/perf/util/metricgroup.c                      | 109 ++++++++++++++++-----
 tools/perf/util/stat-display.c                     |   6 +-
 tools/perf/util/util.c                             |  18 ++++
 tools/perf/util/util.h                             |   2 +
 10 files changed, 128 insertions(+), 38 deletions(-)
Signature · 1 line
2.7.4
#4 lore raw

Re: [PATCH V2 2/5] perf metricgroup: Factor out metricgroup__add_metric_weak_group()

In reply to [email protected]

Em Mon, Feb 24, 2020 at 01:59:21PM -0800, [email protected] escreveu:

Quoted 27 lines

From: Kan Liang <[email protected]>

Factor out metricgroup__add_metric_weak_group() which add metrics into a weak group. The change can improve code readability. Because following patch will introduce a function which add standalone metrics.

---
 tools/perf/util/metricgroup.c | 57 +++++++++++++++++++++++++------------------
 1 file changed, 33 insertions(+), 24 deletions(-)
tools/perf/util/metricgroup.c +8 -0
index 02aee94..1cd042c 100644
@@ -399,13 +399,42 @@ void metricgroup__print(bool metrics, bool metricgroups, char *filter,
 	strlist__delete(metriclist);
 }
 
+static void metricgroup__add_metric_weak_group(struct strbuf *events,
+					       const char **ids,
+					       int idnum)
+{
+	bool no_group = false;
+	int i;
+
+	for (i = 0; i < idnum; i++) {

tools/perf/util/metricgroup.c void metricgroup__print(bool metrics, bool metricgroups, char *filter, · line 419

Quoted 10 lines
tools/perf/util/metricgroup.c +10 -0
+		pr_debug("found event %s\n", ids[i]);
+		/*
+		 * Duration time maps to a software event and can make
+		 * groups not count. Always use it outside a
+		 * group.
+		 */
+		if (!strcmp(ids[i], "duration_time")) {
+			if (i > 0)
+				strbuf_addf(events, "}:W,");
+			strbuf_addf(events, "duration_time");

At some point we need to do some error checking on these strbuf calls, but since this was the state in this file, lets not block this patchkit on such grounds,

- Arnaldo

Quoted 64 lines
tools/perf/util/metricgroup.c +15 -24
+			no_group = true;
+			continue;
+		}
+		strbuf_addf(events, "%s%s",
+			i == 0 || no_group ? "{" : ",",
+			ids[i]);
+		no_group = false;
+	}
+	if (!no_group)
+		strbuf_addf(events, "}:W");
+}
+
 static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 				   struct list_head *group_list)
 {
 	struct pmu_events_map *map = perf_pmu__find_map(NULL);
 	struct pmu_event *pe;
-	int ret = -EINVAL;
-	int i, j;
+	int i, ret = -EINVAL;
 
 	if (!map)
 		return 0;
@@ -422,7 +451,6 @@ static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 			const char **ids;
 			int idnum;
 			struct egroup *eg;
-			bool no_group = false;
 
 			pr_debug("metric expr %s for %s\n", pe->metric_expr, pe->metric_name);
 
@@ -431,27 +459,8 @@ static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 				continue;
 			if (events->len > 0)
 				strbuf_addf(events, ",");
-			for (j = 0; j < idnum; j++) {
-				pr_debug("found event %s\n", ids[j]);
-				/*
-				 * Duration time maps to a software event and can make
-				 * groups not count. Always use it outside a
-				 * group.
-				 */
-				if (!strcmp(ids[j], "duration_time")) {
-					if (j > 0)
-						strbuf_addf(events, "}:W,");
-					strbuf_addf(events, "duration_time");
-					no_group = true;
-					continue;
-				}
-				strbuf_addf(events, "%s%s",
-					j == 0 || no_group ? "{" : ",",
-					ids[j]);
-				no_group = false;
-			}
-			if (!no_group)
-				strbuf_addf(events, "}:W");
+
+			metricgroup__add_metric_weak_group(events, ids, idnum);
 
 			eg = malloc(sizeof(struct egroup));
 			if (!eg) {
Signature · 1 line
2.7.4
Signature · 1 line
- Arnaldo
#13 lore raw
In reply to Jiri Olsa

Em Wed, Feb 26, 2020 at 04:35:49PM +0100, Jiri Olsa escreveu:

Quoted 10 lines · On Mon, Feb 24, 2020 at 01:59:19PM -0800, [email protected] wrote:
Quoted 7 lines

From: Kan Liang <[email protected]>

Changes since V1:
- Remove global static flag violate_nmi_constraint, and add a new function metricgroup___watchdog_constraint_hint() for all watchdog constraint hints in patch 4.
The rest of the patches are not changed.

Thanks, tested, applied,

- Arnaldo

Quoted 87 lines

thanks,
jirka

Quoted 83 lines

Some metric groups, e.g. Page_Walks_Utilization, will never count when NMI watchdog is enabled.

$echo 1 > /proc/sys/kernel/nmi_watchdog
$perf stat -M Page_Walks_Utilization

Performance counter stats for 'system wide':

 <not counted>      itlb_misses.walk_pending       (0.00%)
 <not counted>      dtlb_load_misses.walk_pending  (0.00%)
 <not counted>      dtlb_store_misses.walk_pending (0.00%)
 <not counted>      ept.walk_pending               (0.00%)
 <not counted>      cycles                         (0.00%)
       2.343460588 seconds time elapsed

Some events weren't counted. Try disabling the NMI watchdog:
echo 0 > /proc/sys/kernel/nmi_watchdog
perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog
The events in group usually have to be from the same PMU. Try reorganizing the group.

A metric group is a weak group, which relies on group validation code in the kernel to determine whether to be opened as a group or a non-group. However, group validation code may return false-positives, especially when NMI watchdog is enabled. (The metric group is allowed as a group but will never be scheduled.)

The attempt to fix the group validation code has been rejected.
https://lore.kernel.org/lkml/[email protected]/
Because we cannot accurately predict whether the group can be scheduled as a group, only by checking current status.

This patch set provides another solution to mitigate the issue. Add "MetricConstraint" in event list, which provides a hint for perf tool, e.g. "MetricConstraint": "NO_NMI_WATCHDOG". Perf tool can change the metric group to non-group (standalone metrics) if NMI watchdog is enabled.

After applying the patch,

$echo 1 > /proc/sys/kernel/nmi_watchdog
$perf stat -M Page_Walks_Utilization
Splitting metric group Page_Walks_Utilization into standalone metrics. Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:
echo 0 > /proc/sys/kernel/nmi_watchdog
perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog

Performance counter stats for 'system wide':

        18,253,454      itlb_misses.walk_pending  #      0.0
                              Page_Walks_Utilization   (50.55%)
        78,051,525      dtlb_load_misses.walk_pending  (50.55%)
        29,213,063      dtlb_store_misses.walk_pending (50.55%)
                 0      ept.walk_pending               (50.55%)
     2,542,132,364      cycles                         (49.92%)

       1.037095993 seconds time elapsed

Kan Liang (5):
perf jevents: Support metric constraint
perf metricgroup: Factor out metricgroup__add_metric_weak_group() perf util: Factor out sysctl__nmi_watchdog_enabled() perf metricgroup: Support metric constraint
perf vendor events: Add NO_NMI_WATCHDOG metric constraint

 .../arch/x86/cascadelakex/clx-metrics.json         |   3 +-
 .../pmu-events/arch/x86/skylake/skl-metrics.json   |   3 +-
 .../pmu-events/arch/x86/skylakex/skx-metrics.json  |   3 +-
 tools/perf/pmu-events/jevents.c                    |  19 ++--
 tools/perf/pmu-events/jevents.h                    |   2 +-
 tools/perf/pmu-events/pmu-events.h                 |   1 +
 tools/perf/util/metricgroup.c                      | 109 ++++++++++++++++-----
 tools/perf/util/stat-display.c                     |   6 +-
 tools/perf/util/util.c                             |  18 ++++
 tools/perf/util/util.h                             |   2 +
 10 files changed, 128 insertions(+), 38 deletions(-)
Signature · 1 line
2.7.4
Signature · 1 line
- Arnaldo
merge bot Applied
#2 lore raw

[tip: perf/core] perf jevents: Support metric constraint

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     03fe02b113888576dc90c3e918d8e1a76b1ceb63
Gitweb:        https://git.kernel.org/tip/03fe02b113888576dc90c3e918d8e1a76b1ceb63
Author:        Kan Liang <[email protected]>
AuthorDate:    Mon, 24 Feb 2020 13:59:20 -08:00
Committer:     Arnaldo Carvalho de Melo <[email protected]>
CommitterDate: Tue, 10 Mar 2020 14:43:05 -03:00

perf jevents: Support metric constraint

A new field "MetricConstraint" is introduced in JSON event list.

Extend jevents to parse the field and save the value in metric_constraint.

---
 tools/perf/pmu-events/jevents.c    | 19 +++++++++++++------
 tools/perf/pmu-events/jevents.h    |  2 +-
 tools/perf/pmu-events/pmu-events.h |  1 +
 3 files changed, 15 insertions(+), 7 deletions(-)
tools/perf/pmu-events/jevents.c +13 -6
index 079c77b..6d0f61f 100644
@@ -323,7 +323,7 @@ static int print_events_table_entry(void *data, char *name, char *event,
 				    char *pmu, char *unit, char *perpkg,
 				    char *metric_expr,
 				    char *metric_name, char *metric_group,
-				    char *deprecated)
+				    char *deprecated, char *metric_constraint)
 {
 	struct perf_entry_data *pd = data;
 	FILE *outfp = pd->outfp;
@@ -357,6 +357,8 @@ static int print_events_table_entry(void *data, char *name, char *event,
 		fprintf(outfp, "\t.metric_group = \"%s\",\n", metric_group);
 	if (deprecated)
 		fprintf(outfp, "\t.deprecated = \"%s\",\n", deprecated);
+	if (metric_constraint)
+		fprintf(outfp, "\t.metric_constraint = \"%s\",\n", metric_constraint);
 	fprintf(outfp, "},\n");
 
 	return 0;
@@ -375,6 +377,7 @@ struct event_struct {
 	char *metric_name;
 	char *metric_group;
 	char *deprecated;
+	char *metric_constraint;
 };
 
 #define ADD_EVENT_FIELD(field) do { if (field) {		\
@@ -422,7 +425,7 @@ static int save_arch_std_events(void *data, char *name, char *event,
 				char *desc, char *long_desc, char *pmu,
 				char *unit, char *perpkg, char *metric_expr,
 				char *metric_name, char *metric_group,
-				char *deprecated)
+				char *deprecated, char *metric_constraint)
 {
 	struct event_struct *es;
 
@@ -486,7 +489,7 @@ try_fixup(const char *fn, char *arch_std, char **event, char **desc,
 	  char **name, char **long_desc, char **pmu, char **filter,
 	  char **perpkg, char **unit, char **metric_expr, char **metric_name,
 	  char **metric_group, unsigned long long eventcode,
-	  char **deprecated)
+	  char **deprecated, char **metric_constraint)
 {
 	/* try to find matching event from arch standard values */
 	struct event_struct *es;
@@ -515,7 +518,7 @@ int json_events(const char *fn,
 		      char *pmu, char *unit, char *perpkg,
 		      char *metric_expr,
 		      char *metric_name, char *metric_group,
-		      char *deprecated),
+		      char *deprecated, char *metric_constraint),
 	  void *data)
 {
 	int err;
@@ -545,6 +548,7 @@ int json_events(const char *fn,
 		char *metric_name = NULL;
 		char *metric_group = NULL;
 		char *deprecated = NULL;
+		char *metric_constraint = NULL;
 		char *arch_std = NULL;
 		unsigned long long eventcode = 0;
 		struct msrmap *msr = NULL;
@@ -629,6 +633,8 @@ int json_events(const char *fn,
 				addfield(map, &metric_name, "", "", val);
 			} else if (json_streq(map, field, "MetricGroup")) {
 				addfield(map, &metric_group, "", "", val);
+			} else if (json_streq(map, field, "MetricConstraint")) {
+				addfield(map, &metric_constraint, "", "", val);
 			} else if (json_streq(map, field, "MetricExpr")) {
 				addfield(map, &metric_expr, "", "", val);
 				for (s = metric_expr; *s; s++)
@@ -670,13 +676,13 @@ int json_events(const char *fn,
 					&long_desc, &pmu, &filter, &perpkg,
 					&unit, &metric_expr, &metric_name,
 					&metric_group, eventcode,
-					&deprecated);
+					&deprecated, &metric_constraint);
 			if (err)
 				goto free_strings;
 		}
 		err = func(data, name, real_event(name, event), desc, long_desc,
 			   pmu, unit, perpkg, metric_expr, metric_name,
-			   metric_group, deprecated);
+			   metric_group, deprecated, metric_constraint);
 free_strings:
 		free(event);
 		free(desc);
@@ -691,6 +697,7 @@ free_strings:
 		free(metric_expr);
 		free(metric_name);
 		free(metric_group);
+		free(metric_constraint);
 		free(arch_std);
 
 		if (err)
tools/perf/pmu-events/jevents.h +1 -1
index 5cda49a..2afc830 100644
@@ -8,7 +8,7 @@ int json_events(const char *fn,
 				char *pmu,
 				char *unit, char *perpkg, char *metric_expr,
 				char *metric_name, char *metric_group,
-				char *deprecated),
+				char *deprecated, char *metric_constraint),
 		void *data);
 char *get_cpu_str(void);
 
tools/perf/pmu-events/pmu-events.h +1 -0
index caeb577..53e76d5 100644
@@ -18,6 +18,7 @@ struct pmu_event {
 	const char *metric_name;
 	const char *metric_group;
 	const char *deprecated;
+	const char *metric_constraint;
 };
 
 /*
merge bot Applied
#9 lore raw

[tip: perf/core] perf metricgroup: Support metric constraint

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     ab483d8bc8acb83f0103bc38ef8f2c27d98ffd1b
Gitweb:        https://git.kernel.org/tip/ab483d8bc8acb83f0103bc38ef8f2c27d98ffd1b
Author:        Kan Liang <[email protected]>
AuthorDate:    Mon, 24 Feb 2020 13:59:23 -08:00
Committer:     Arnaldo Carvalho de Melo <[email protected]>
CommitterDate: Tue, 10 Mar 2020 14:47:50 -03:00

perf metricgroup: Support metric constraint

Some metric groups have metric constraints. A metric group can be scheduled as a group only when some constraints are applied. For example, Page_Walks_Utilization has a metric constraint, "NO_NMI_WATCHDOG".

When NMI watchdog is disabled, the metric group can be scheduled as a group. Otherwise, splitting the metric group into standalone metrics.

Add a new function, metricgroup__has_constraint(), to check whether all constraints are applied. If not, splitting the metric group into standalone metrics.

Currently, only one constraint, "NO_NMI_WATCHDOG", is checked. Print a warning for the metric group with the constraint, when NMI WATCHDOG is enabled.

---
 tools/perf/util/metricgroup.c | 54 +++++++++++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 1 deletion(-)
tools/perf/util/metricgroup.c +53 -1
index 1cd042c..c3a8c70 100644
@@ -22,6 +22,8 @@
 #include <linux/string.h>
 #include <linux/zalloc.h>
 #include <subcmd/parse-options.h>
+#include <api/fs/fs.h>
+#include "util.h"
 
 struct metric_event *metricgroup__lookup(struct rblist *metric_events,
 					 struct evsel *evsel,
@@ -429,6 +431,49 @@ static void metricgroup__add_metric_weak_group(struct strbuf *events,
 		strbuf_addf(events, "}:W");
 }
 
+static void metricgroup__add_metric_non_group(struct strbuf *events,
+					      const char **ids,
+					      int idnum)
+{
+	int i;
+
+	for (i = 0; i < idnum; i++)
+		strbuf_addf(events, ",%s", ids[i]);
+}
+
+static void metricgroup___watchdog_constraint_hint(const char *name, bool foot)
+{
+	static bool violate_nmi_constraint;
+
+	if (!foot) {
+		pr_warning("Splitting metric group %s into standalone metrics.\n", name);
+		violate_nmi_constraint = true;
+		return;
+	}
+
+	if (!violate_nmi_constraint)
+		return;
+
+	pr_warning("Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:\n"
+		   "    echo 0 > /proc/sys/kernel/nmi_watchdog\n"
+		   "    perf stat ...\n"
+		   "    echo 1 > /proc/sys/kernel/nmi_watchdog\n");
+}
+
+static bool metricgroup__has_constraint(struct pmu_event *pe)
+{
+	if (!pe->metric_constraint)
+		return false;
+
+	if (!strcmp(pe->metric_constraint, "NO_NMI_WATCHDOG") &&
+	    sysctl__nmi_watchdog_enabled()) {
+		metricgroup___watchdog_constraint_hint(pe->metric_name, false);
+		return true;
+	}
+
+	return false;
+}
+
 static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 				   struct list_head *group_list)
 {
@@ -460,7 +505,10 @@ static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 			if (events->len > 0)
 				strbuf_addf(events, ",");
 
-			metricgroup__add_metric_weak_group(events, ids, idnum);
+			if (metricgroup__has_constraint(pe))
+				metricgroup__add_metric_non_group(events, ids, idnum);
+			else
+				metricgroup__add_metric_weak_group(events, ids, idnum);
 
 			eg = malloc(sizeof(struct egroup));
 			if (!eg) {
@@ -502,6 +550,10 @@ static int metricgroup__add_metric_list(const char *list, struct strbuf *events,
 		}
 	}
 	free(nlist);
+
+	if (!ret)
+		metricgroup___watchdog_constraint_hint(NULL, true);
+
 	return ret;
 }
 
merge bot Applied
#5 lore raw

[tip: perf/core] perf metricgroup: Factor out metricgroup__add_metric_weak_group()

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     f742634ab47f59160a85ddc502418556b21953c2
Gitweb:        https://git.kernel.org/tip/f742634ab47f59160a85ddc502418556b21953c2
Author:        Kan Liang <[email protected]>
AuthorDate:    Mon, 24 Feb 2020 13:59:21 -08:00
Committer:     Arnaldo Carvalho de Melo <[email protected]>
CommitterDate: Tue, 10 Mar 2020 14:44:36 -03:00

perf metricgroup: Factor out metricgroup__add_metric_weak_group()

Factor out metricgroup__add_metric_weak_group() which add metrics into a weak group. The change can improve code readability. Because following patch will introduce a function which add standalone metrics.

---
 tools/perf/util/metricgroup.c | 57 +++++++++++++++++++---------------
 1 file changed, 33 insertions(+), 24 deletions(-)
tools/perf/util/metricgroup.c +33 -24
index 02aee94..1cd042c 100644
@@ -399,13 +399,42 @@ void metricgroup__print(bool metrics, bool metricgroups, char *filter,
 	strlist__delete(metriclist);
 }
 
+static void metricgroup__add_metric_weak_group(struct strbuf *events,
+					       const char **ids,
+					       int idnum)
+{
+	bool no_group = false;
+	int i;
+
+	for (i = 0; i < idnum; i++) {
+		pr_debug("found event %s\n", ids[i]);
+		/*
+		 * Duration time maps to a software event and can make
+		 * groups not count. Always use it outside a
+		 * group.
+		 */
+		if (!strcmp(ids[i], "duration_time")) {
+			if (i > 0)
+				strbuf_addf(events, "}:W,");
+			strbuf_addf(events, "duration_time");
+			no_group = true;
+			continue;
+		}
+		strbuf_addf(events, "%s%s",
+			i == 0 || no_group ? "{" : ",",
+			ids[i]);
+		no_group = false;
+	}
+	if (!no_group)
+		strbuf_addf(events, "}:W");
+}
+
 static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 				   struct list_head *group_list)
 {
 	struct pmu_events_map *map = perf_pmu__find_map(NULL);
 	struct pmu_event *pe;
-	int ret = -EINVAL;
-	int i, j;
+	int i, ret = -EINVAL;
 
 	if (!map)
 		return 0;
@@ -422,7 +451,6 @@ static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 			const char **ids;
 			int idnum;
 			struct egroup *eg;
-			bool no_group = false;
 
 			pr_debug("metric expr %s for %s\n", pe->metric_expr, pe->metric_name);
 
@@ -431,27 +459,8 @@ static int metricgroup__add_metric(const char *metric, struct strbuf *events,
 				continue;
 			if (events->len > 0)
 				strbuf_addf(events, ",");
-			for (j = 0; j < idnum; j++) {
-				pr_debug("found event %s\n", ids[j]);
-				/*
-				 * Duration time maps to a software event and can make
-				 * groups not count. Always use it outside a
-				 * group.
-				 */
-				if (!strcmp(ids[j], "duration_time")) {
-					if (j > 0)
-						strbuf_addf(events, "}:W,");
-					strbuf_addf(events, "duration_time");
-					no_group = true;
-					continue;
-				}
-				strbuf_addf(events, "%s%s",
-					j == 0 || no_group ? "{" : ",",
-					ids[j]);
-				no_group = false;
-			}
-			if (!no_group)
-				strbuf_addf(events, "}:W");
+
+			metricgroup__add_metric_weak_group(events, ids, idnum);
 
 			eg = malloc(sizeof(struct egroup));
 			if (!eg) {
merge bot Applied
#7 lore raw

[tip: perf/core] perf util: Factor out sysctl__nmi_watchdog_enabled()

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     2a14c1bf017f48a17c8c0ba26a22625363e77cc7
Gitweb:        https://git.kernel.org/tip/2a14c1bf017f48a17c8c0ba26a22625363e77cc7
Author:        Kan Liang <[email protected]>
AuthorDate:    Mon, 24 Feb 2020 13:59:22 -08:00
Committer:     Arnaldo Carvalho de Melo <[email protected]>
CommitterDate: Tue, 10 Mar 2020 14:46:19 -03:00

perf util: Factor out sysctl__nmi_watchdog_enabled()

The NMI watchdog status is required for metric group constraint examination. Factor out sysctl__nmi_watchdog_enabled() to retrieve the NMI watchdog status.

Users may count more than one metric group each time. If so, the NMI watchdog status may be retrieved several times. To reduce the overhead, cache the NMI watchdog status.

Replace the NMI watchdog status checking in print_footer() by sysctl__nmi_watchdog_enabled().

---
 tools/perf/util/stat-display.c |  6 ++----
 tools/perf/util/util.c         | 18 ++++++++++++++++++
 tools/perf/util/util.h         |  2 ++
 3 files changed, 22 insertions(+), 4 deletions(-)
tools/perf/util/stat-display.c +2 -4
index d89cb0d..76c6052 100644
@@ -16,6 +16,7 @@
 #include <linux/ctype.h>
 #include "cgroup.h"
 #include <api/fs/fs.h>
+#include "util.h"
 
 #define CNTR_NOT_SUPPORTED	"<not supported>"
 #define CNTR_NOT_COUNTED	"<not counted>"
@@ -1097,7 +1098,6 @@ static void print_footer(struct perf_stat_config *config)
 {
 	double avg = avg_stats(config->walltime_nsecs_stats) / NSEC_PER_SEC;
 	FILE *output = config->output;
-	int n;
 
 	if (!config->null_run)
 		fprintf(output, "\n");
@@ -1131,9 +1131,7 @@ static void print_footer(struct perf_stat_config *config)
 	}
 	fprintf(output, "\n\n");
 
-	if (config->print_free_counters_hint &&
-	    sysctl__read_int("kernel/nmi_watchdog", &n) >= 0 &&
-	    n > 0)
+	if (config->print_free_counters_hint && sysctl__nmi_watchdog_enabled())
 		fprintf(output,
 "Some events weren't counted. Try disabling the NMI watchdog:\n"
 "	echo 0 > /proc/sys/kernel/nmi_watchdog\n"
tools/perf/util/util.c +18 -0
index 969ae56..d707c96 100644
@@ -55,6 +55,24 @@ int sysctl__max_stack(void)
 	return sysctl_perf_event_max_stack;
 }
 
+bool sysctl__nmi_watchdog_enabled(void)
+{
+	static bool cached;
+	static bool nmi_watchdog;
+	int value;
+
+	if (cached)
+		return nmi_watchdog;
+
+	if (sysctl__read_int("kernel/nmi_watchdog", &value) < 0)
+		return false;
+
+	nmi_watchdog = (value > 0) ? true : false;
+	cached = true;
+
+	return nmi_watchdog;
+}
+
 bool test_attr__enabled;
 
 bool perf_host  = true;
tools/perf/util/util.h +2 -0
index 9969b8b..f486fdd 100644
@@ -29,6 +29,8 @@ size_t hex_width(u64 v);
 
 int sysctl__max_stack(void);
 
+bool sysctl__nmi_watchdog_enabled(void);
+
 int fetch_kernel_version(unsigned int *puint,
 			 char *str, size_t str_sz);
 #define KVER_VERSION(x)		(((x) >> 16) & 0xff)
merge bot Applied
#11 lore raw

[tip: perf/core] perf vendor events intel: Add NO_NMI_WATCHDOG metric constraint

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     b95fcd2c1c25bd14f55d5d6ab268b3ab00b8a774
Gitweb:        https://git.kernel.org/tip/b95fcd2c1c25bd14f55d5d6ab268b3ab00b8a774
Author:        Kan Liang <[email protected]>
AuthorDate:    Mon, 24 Feb 2020 13:59:24 -08:00
Committer:     Arnaldo Carvalho de Melo <[email protected]>
CommitterDate: Tue, 10 Mar 2020 14:56:46 -03:00

perf vendor events intel: Add NO_NMI_WATCHDOG metric constraint

Add NO_NMI_WATCHDOG metric constraint to Page_Walks_Utilization for Sky Lake and Cascade Lake.

Committer testing:

On a Lenovo T480S, Intel(R) Core(TM) i7-8650U Kaby Lake, that looking at x86's mapfile.csv file is a:

  $ grep -w skylake tools/perf/pmu-events/arch/x86/mapfile.csv
  GenuineIntel-6-[4589]E,v24,skylake,core
  $

So uses the constraint added in this patch in this file:

tools/perf/pmu-events/arch/x86/skylake/skl-metrics.json

Before:

  # perf stat -a -M Page_Walks_Utilization sleep 2

Performance counter stats for 'system wide':

       <not counted>      itlb_misses.walk_pending                                      (0.00%)
       <not counted>      dtlb_load_misses.walk_pending                                     (0.00%)
       <not counted>      dtlb_store_misses.walk_pending                                     (0.00%)
       <not counted>      ept.walk_pending                                              (0.00%)
       <not counted>      cycles                                                        (0.00%)
         2.001750514 seconds time elapsed

Some events weren't counted. Try disabling the NMI watchdog:
echo 0 > /proc/sys/kernel/nmi_watchdog
perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog
The events in group usually have to be from the same PMU. Try reorganizing the group. #

After:

  # perf stat -a -M Page_Walks_Utilization sleep 2
  Splitting metric group Page_Walks_Utilization into standalone metrics.
  Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:
      echo 0 > /proc/sys/kernel/nmi_watchdog
      perf stat ...
      echo 1 > /proc/sys/kernel/nmi_watchdog
  ,
   Performance counter stats for 'system wide':
          36,883,102      itlb_misses.walk_pending  #      0.1 Page_Walks_Utilization   (79.99%)
         123,104,146      dtlb_load_misses.walk_pending                                     (80.02%)
          13,720,795      dtlb_store_misses.walk_pending                                     (79.99%)
                   0      ept.walk_pending                                              (79.99%)
       1,519,948,400      cycles                                                        (80.01%)
         2.002170780 seconds time elapsed

#

Before and after, if we disable the nmi_watchdog we get:

  # echo 0 > /proc/sys/kernel/nmi_watchdog
  # perf stat -a -M Page_Walks_Utilization sleep 2

Performance counter stats for 'system wide':

          33,721,658      itlb_misses.walk_pending  #      0.1 Page_Walks_Utilization
          84,070,996      dtlb_load_misses.walk_pending
           9,816,071      dtlb_store_misses.walk_pending
                   0      ept.walk_pending
         704,920,899      cycles
         2.002331670 seconds time elapsed

#

More information about the metric expressions:

  # perf stat -v -a -M Page_Walks_Utilization sleep 2
  Using CPUID GenuineIntel-6-8E-A
  metric expr ( itlb_misses.walk_pending + dtlb_load_misses.walk_pending + dtlb_store_misses.walk_pending + ept.walk_pending ) / ( 2 * cycles ) for Page_Walks_Utilization
  found event itlb_misses.walk_pending
  found event dtlb_load_misses.walk_pending
  found event dtlb_store_misses.walk_pending
  found event ept.walk_pending
  found event cycles
  adding {itlb_misses.walk_pending,dtlb_load_misses.walk_pending,dtlb_store_misses.walk_pending,ept.walk_pending,cycles}:W
   -> cpu/umask=0x10,(null)=0x186a3,event=0x85/
   -> cpu/umask=0x10,(null)=0x1e8483,event=0x8/
   -> cpu/umask=0x10,(null)=0x1e8483,event=0x49/
   -> cpu/umask=0x10,(null)=0x1e8483,event=0x4f/
  itlb_misses.walk_pending: 8085772 16010162799 16010162799
  dtlb_load_misses.walk_pending: 28134579 16010162799 16010162799
  dtlb_store_misses.walk_pending: 7276535 16010162799 16010162799
  ept.walk_pending: 2 16010162799 16010162799
  cycles: 315140605 16010162799 16010162799

Performance counter stats for 'system wide':

           8,085,772      itlb_misses.walk_pending  #      0.1 Page_Walks_Utilization
          28,134,579      dtlb_load_misses.walk_pending
           7,276,535      dtlb_store_misses.walk_pending
                   2      ept.walk_pending
         315,140,605      cycles
         2.002333181 seconds time elapsed

#

---
 tools/perf/pmu-events/arch/x86/cascadelakex/clx-metrics.json | 3 ++-
 tools/perf/pmu-events/arch/x86/skylake/skl-metrics.json      | 3 ++-
 tools/perf/pmu-events/arch/x86/skylakex/skx-metrics.json     | 3 ++-
 3 files changed, 6 insertions(+), 3 deletions(-)
tools/perf/pmu-events/arch/x86/cascadelakex/clx-metrics.json +2 -1
index f946532..a728c6e 100644
@@ -215,7 +215,8 @@
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
         "MetricExpr": "( ITLB_MISSES.WALK_PENDING + DTLB_LOAD_MISSES.WALK_PENDING + DTLB_STORE_MISSES.WALK_PENDING + EPT.WALK_PENDING ) / ( 2 * cycles )",
         "MetricGroup": "TLB",
-        "MetricName": "Page_Walks_Utilization"
+        "MetricName": "Page_Walks_Utilization",
+        "MetricConstraint": "NO_NMI_WATCHDOG"
     },
     {
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
tools/perf/pmu-events/arch/x86/skylake/skl-metrics.json +2 -1
index e7feb60..f97e831 100644
@@ -215,7 +215,8 @@
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
         "MetricExpr": "( ITLB_MISSES.WALK_PENDING + DTLB_LOAD_MISSES.WALK_PENDING + DTLB_STORE_MISSES.WALK_PENDING + EPT.WALK_PENDING ) / ( 2 * cycles )",
         "MetricGroup": "TLB",
-        "MetricName": "Page_Walks_Utilization"
+        "MetricName": "Page_Walks_Utilization",
+        "MetricConstraint": "NO_NMI_WATCHDOG"
     },
     {
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
tools/perf/pmu-events/arch/x86/skylakex/skx-metrics.json +2 -1
index 21d7a0c..35f5db1 100644
@@ -215,7 +215,8 @@
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",
         "MetricExpr": "( ITLB_MISSES.WALK_PENDING + DTLB_LOAD_MISSES.WALK_PENDING + DTLB_STORE_MISSES.WALK_PENDING + EPT.WALK_PENDING ) / ( 2 * cycles )",
         "MetricGroup": "TLB",
-        "MetricName": "Page_Walks_Utilization"
+        "MetricName": "Page_Walks_Utilization",
+        "MetricConstraint": "NO_NMI_WATCHDOG"
     },
     {
         "BriefDescription": "Utilization of the core's Page Walker(s) serving STLB misses triggered by instruction/Load/Store accesses",