Skip to contentSkip to conversation

perf expr: Add flex scanner

PATCH Superseded Jiri Olsa 4 participants 9 messages Feb 24, 2020 – Feb 27, 2020 perf events

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

v2 cover perf expr: Add flex scanner

hi,
while preparing changes for user defined metric expressions I also moved the expression manual parser to flex.

The reason is to have an easy and reasonable way to support and parse multiple user-defined metric expressions from command line or file.

I was posponing the change, but I just saw another update to the expr manual scanner (from Kajol Jain), so cherry picked just the expr flex code changes to get it out.

Kajol Jain,
I think it should ease up your change for unknown values marked by '?'. Would you consider rebasing your changes on top of this?

v2 changes:
- handle special chars properly
- fix return value for expr__parse

Available also in:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/metric_flex

thanks,
jirka

---
Jiri Olsa (5):
perf expr: Add expr.c object
perf expr: Move expr lexer to flex
perf expr: Increase EXPR_MAX_OTHER
perf expr: Straighten expr__parse/expr__find_other interface perf expr: Make expr__parse return -1 on error

 tools/perf/tests/expr.c       |  10 +++---
 tools/perf/util/Build         |  11 ++++++-
 tools/perf/util/expr.c        | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/expr.h        |   8 ++---
 tools/perf/util/expr.l        |  87 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/expr.y        | 208 ++++++++++++++++++++++++++++++++-----------------------------------------------------------------------------------------
 tools/perf/util/stat-shadow.c |   4 +--
 7 files changed, 272 insertions(+), 168 deletions(-)
 create mode 100644 tools/perf/util/expr.c
 create mode 100644 tools/perf/util/expr.l
author reviewer
#1 lore raw

PATCH v2 1/5 perf expr: Add expr.c object

Add generic expr code into new expr.c object.

The expr.c object will be mainly used in following change that will get rid of the manual flex code,

---
 tools/perf/util/Build  |  1 +
 tools/perf/util/expr.c | 19 +++++++++++++++++++
 tools/perf/util/expr.y | 16 ----------------
 3 files changed, 20 insertions(+), 16 deletions(-)
 create mode 100644 tools/perf/util/expr.c
tools/perf/util/Build +1 -0
index 07da6c790b63..6fdf073093a6 100644
@@ -122,6 +122,7 @@ perf-y += vsprintf.o
 perf-y += units.o
 perf-y += time-utils.o
 perf-y += expr-bison.o
+perf-y += expr.o
 perf-y += branch.o
 perf-y += mem2node.o
 
tools/perf/util/expr.c new file +19 -0
new file mode 100644
index 000000000000..816b23b2068a
@@ -0,0 +1,19 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <assert.h>
+#include "expr.h"
+
+/* Caller must make sure id is allocated */
+void expr__add_id(struct parse_ctx *ctx, const char *name, double val)
+{
+	int idx;
+
+	assert(ctx->num_ids < MAX_PARSE_ID);
+	idx = ctx->num_ids++;
+	ctx->ids[idx].name = name;
+	ctx->ids[idx].val = val;
+}
+
+void expr__ctx_init(struct parse_ctx *ctx)
+{
+	ctx->num_ids = 0;
+}
tools/perf/util/expr.y +0 -16
index 7d226241f1d7..7cea8b7c3a5c 100644
@@ -6,7 +6,6 @@
 #define IN_EXPR_Y 1
 #include "expr.h"
 #include "smt.h"
-#include <assert.h>
 #include <string.h>
 
 #define MAXIDLEN 256
@@ -169,21 +168,6 @@ static int expr__lex(YYSTYPE *res, const char **pp)
 	return tok;
 }
 
-/* Caller must make sure id is allocated */
-void expr__add_id(struct parse_ctx *ctx, const char *name, double val)
-{
-	int idx;
-	assert(ctx->num_ids < MAX_PARSE_ID);
-	idx = ctx->num_ids++;
-	ctx->ids[idx].name = name;
-	ctx->ids[idx].val = val;
-}
-
-void expr__ctx_init(struct parse_ctx *ctx)
-{
-	ctx->num_ids = 0;
-}
-
 static bool already_seen(const char *val, const char *one, const char **other,
 			 int num_other)
 {
Signature · 1 line
2.24.1
author reviewer
#2 lore raw

PATCH v2 2/5 perf expr: Move expr lexer to flex

In reply to Jiri Olsa

Adding expr flex code instead of the manual parser code. So it's easily extensible in upcoming changes.

The new flex code is in flex.l object and gets compiled like all the other flexers we use. It's defined as flex reentrant parser.

It's used by both expr__parse and expr__find_other interfaces by separating the starting point.

There's no intended change of functionality ;-) the test expr is passing.

---
 tools/perf/util/Build  |  10 ++-
 tools/perf/util/expr.c |  93 ++++++++++++++++++++
 tools/perf/util/expr.h |   2 -
 tools/perf/util/expr.l |  87 +++++++++++++++++++
 tools/perf/util/expr.y | 192 ++++++++++++-----------------------------
 5 files changed, 243 insertions(+), 141 deletions(-)
 create mode 100644 tools/perf/util/expr.l
tools/perf/util/Build +9 -1
index 6fdf073093a6..c0cf8dff694e 100644
@@ -121,6 +121,7 @@ perf-y += mem-events.o
 perf-y += vsprintf.o
 perf-y += units.o
 perf-y += time-utils.o
+perf-y += expr-flex.o
 perf-y += expr-bison.o
 perf-y += expr.o
 perf-y += branch.o
@@ -190,9 +191,13 @@ $(OUTPUT)util/parse-events-bison.c: util/parse-events.y
 	$(call rule_mkdir)
 	$(Q)$(call echo-cmd,bison)$(BISON) -v util/parse-events.y -d $(PARSER_DEBUG_BISON) -o $@ -p parse_events_
 
+$(OUTPUT)util/expr-flex.c: util/expr.l $(OUTPUT)util/expr-bison.c
+	$(call rule_mkdir)
+	$(Q)$(call echo-cmd,flex)$(FLEX) -o $@ --header-file=$(OUTPUT)util/expr-flex.h $(PARSER_DEBUG_FLEX) util/expr.l
+
 $(OUTPUT)util/expr-bison.c: util/expr.y
 	$(call rule_mkdir)
-	$(Q)$(call echo-cmd,bison)$(BISON) -v util/expr.y -d $(PARSER_DEBUG_BISON) -o $@ -p expr__
+	$(Q)$(call echo-cmd,bison)$(BISON) -v util/expr.y -d $(PARSER_DEBUG_BISON) -o $@ -p expr_
 
 $(OUTPUT)util/pmu-flex.c: util/pmu.l $(OUTPUT)util/pmu-bison.c
 	$(call rule_mkdir)
@@ -204,12 +209,14 @@ $(OUTPUT)util/pmu-bison.c: util/pmu.y
 
 CFLAGS_parse-events-flex.o  += -w
 CFLAGS_pmu-flex.o           += -w
+CFLAGS_expr-flex.o          += -w
 CFLAGS_parse-events-bison.o += -DYYENABLE_NLS=0 -w
 CFLAGS_pmu-bison.o          += -DYYENABLE_NLS=0 -DYYLTYPE_IS_TRIVIAL=0 -w
 CFLAGS_expr-bison.o         += -DYYENABLE_NLS=0 -DYYLTYPE_IS_TRIVIAL=0 -w
 
 $(OUTPUT)util/parse-events.o: $(OUTPUT)util/parse-events-flex.c $(OUTPUT)util/parse-events-bison.c
 $(OUTPUT)util/pmu.o: $(OUTPUT)util/pmu-flex.c $(OUTPUT)util/pmu-bison.c
+$(OUTPUT)util/expr.o: $(OUTPUT)util/expr-flex.c $(OUTPUT)util/expr-bison.c
 
 CFLAGS_bitmap.o        += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
 CFLAGS_find_bit.o      += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
@@ -217,6 +224,7 @@ CFLAGS_rbtree.o        += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ET
 CFLAGS_libstring.o     += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
 CFLAGS_hweight.o       += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
 CFLAGS_parse-events.o  += -Wno-redundant-decls
+CFLAGS_expr.o          += -Wno-redundant-decls
 CFLAGS_header.o        += -include $(OUTPUT)PERF-VERSION-FILE
 
 $(OUTPUT)util/kallsyms.o: ../lib/symbol/kallsyms.c FORCE
tools/perf/util/expr.c +93 -0

102 lines not on this page · raw message

tools/perf/util/expr.h +0 -2
index 046160831f90..9332796e6649 100644
@@ -17,9 +17,7 @@ struct parse_ctx {
 
 void expr__ctx_init(struct parse_ctx *ctx);
 void expr__add_id(struct parse_ctx *ctx, const char *id, double val);
-#ifndef IN_EXPR_Y
 int expr__parse(double *final_val, struct parse_ctx *ctx, const char **pp);
-#endif
 int expr__find_other(const char *p, const char *one, const char ***other,
 		int *num_other);
 
tools/perf/util/expr.l new file +87 -0

87 lines not on this page · raw message

tools/perf/util/expr.y +54 -138

233 lines not on this page · raw message

Signature · 1 line
2.24.1
author reviewer
#3 lore raw

PATCH v2 3/5 perf expr: Increase EXPR_MAX_OTHER

In reply to Jiri Olsa

There's no need to be greedy on allowed variables, also when some of the metrics define more than 15 variables, like Branch_Misprediction_Cost.

Increasing the maximum to 100.

---
 tools/perf/util/expr.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/util/expr.h +1 -1
index 9332796e6649..01e1529bdfd9 100644
@@ -2,7 +2,7 @@
 #ifndef PARSE_CTX_H
 #define PARSE_CTX_H 1
 
-#define EXPR_MAX_OTHER 15
+#define EXPR_MAX_OTHER 100
 #define MAX_PARSE_ID EXPR_MAX_OTHER
 
 struct parse_id {
Signature · 1 line
2.24.1
#4 lore raw

Re: [PATCH 3/5] perf expr: Increase EXPR_MAX_OTHER

Quoted 5 lines · Jiri Olsa

There's no need to be greedy on allowed variables, also when some of the metrics define more than 15 variables, like Branch_Misprediction_Cost.

Increasing the maximum to 100.

FWIW some of the algorithms (e.g. already_seen) have O(n^2) complexity.

If you really want that many probably would need to add some hash tables.

-Andi

author reviewer
#5 lore raw

PATCH v2 4/5 perf expr: Straighten expr__parse/expr__find_other interface

In reply to Jiri Olsa

Now with flex parser we don't need to update parsed string pointer, so the interface can just passed the pointer to the expression instead of pointer to pointer.

---
 tools/perf/tests/expr.c       | 6 +++---
 tools/perf/util/expr.c        | 8 ++++----
 tools/perf/util/expr.h        | 4 ++--
 tools/perf/util/stat-shadow.c | 4 +---
 4 files changed, 10 insertions(+), 12 deletions(-)
tools/perf/tests/expr.c +3 -3
index 87843af4c118..755d73c86c68 100644
@@ -10,7 +10,7 @@ static int test(struct parse_ctx *ctx, const char *e, double val2)
 {
 	double val;
 
-	if (expr__parse(&val, ctx, &e))
+	if (expr__parse(&val, ctx, e))
 		TEST_ASSERT_VAL("parse test failed", 0);
 	TEST_ASSERT_VAL("unexpected value", val == val2);
 	return 0;
@@ -44,11 +44,11 @@ int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
 		return ret;
 
 	p = "FOO/0";
-	ret = expr__parse(&val, &ctx, &p);
+	ret = expr__parse(&val, &ctx, p);
 	TEST_ASSERT_VAL("division by zero", ret == 1);
 
 	p = "BAR/";
-	ret = expr__parse(&val, &ctx, &p);
+	ret = expr__parse(&val, &ctx, p);
 	TEST_ASSERT_VAL("missing operand", ret == 1);
 
 	TEST_ASSERT_VAL("find other",
tools/perf/util/expr.c +4 -4
index b39fd39f10ec..45b25530db5b 100644
@@ -52,9 +52,9 @@ __expr__parse(double *val, struct parse_ctx *ctx, const char *expr,
 	return ret;
 }
 
-int expr__parse(double *final_val, struct parse_ctx *ctx, const char **pp)
+int expr__parse(double *final_val, struct parse_ctx *ctx, const char *expr)
 {
-	return __expr__parse(final_val, ctx, *pp, EXPR_PARSE);
+	return __expr__parse(final_val, ctx, expr, EXPR_PARSE);
 }
 
 static bool
@@ -71,14 +71,14 @@ already_seen(const char *val, const char *one, const char **other,
 	return false;
 }
 
-int expr__find_other(const char *p, const char *one, const char ***other,
+int expr__find_other(const char *expr, const char *one, const char ***other,
 		     int *num_other)
 {
 	int err, i = 0, j = 0;
 	struct parse_ctx ctx;
 
 	expr__ctx_init(&ctx);
-	err = __expr__parse(NULL, &ctx, p, EXPR_OTHER);
+	err = __expr__parse(NULL, &ctx, expr, EXPR_OTHER);
 	if (err)
 		return -1;
 
tools/perf/util/expr.h +2 -2
index 01e1529bdfd9..28a0a2dfbd8c 100644
@@ -17,8 +17,8 @@ struct parse_ctx {
 
 void expr__ctx_init(struct parse_ctx *ctx);
 void expr__add_id(struct parse_ctx *ctx, const char *id, double val);
-int expr__parse(double *final_val, struct parse_ctx *ctx, const char **pp);
-int expr__find_other(const char *p, const char *one, const char ***other,
+int expr__parse(double *final_val, struct parse_ctx *ctx, const char *expr);
+int expr__find_other(const char *expr, const char *one, const char ***other,
 		int *num_other);
 
 #endif
tools/perf/util/stat-shadow.c +1 -3
index 2c41d47f6f83..854d6abf2993 100644
@@ -779,9 +779,7 @@ static void generic_metric(struct perf_stat_config *config,
 	}
 
 	if (!metric_events[i]) {
-		const char *p = metric_expr;
-
-		if (expr__parse(&ratio, &pctx, &p) == 0) {
+		if (expr__parse(&ratio, &pctx, metric_expr) == 0) {
 			char *unit;
 			char metric_bf[64];
 
Signature · 1 line
2.24.1
author reviewer
#6 lore raw

PATCH v2 5/5 perf expr: Make expr__parse return -1 on error

In reply to Jiri Olsa

To match the error value of the expr__find_other function, so all exported expr functions return the same values:
0 on success, -1 on error.

---
 tools/perf/tests/expr.c | 4 ++--
 tools/perf/util/expr.c  | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
tools/perf/tests/expr.c +2 -2
index 755d73c86c68..28313e59d6f6 100644
@@ -45,11 +45,11 @@ int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
 
 	p = "FOO/0";
 	ret = expr__parse(&val, &ctx, p);
-	TEST_ASSERT_VAL("division by zero", ret == 1);
+	TEST_ASSERT_VAL("division by zero", ret == -1);
 
 	p = "BAR/";
 	ret = expr__parse(&val, &ctx, p);
-	TEST_ASSERT_VAL("missing operand", ret == 1);
+	TEST_ASSERT_VAL("missing operand", ret == -1);
 
 	TEST_ASSERT_VAL("find other",
 			expr__find_other("FOO + BAR + BAZ + BOZO", "FOO", &other, &num_other) == 0);
tools/perf/util/expr.c +1 -1
index 45b25530db5b..fd192ddf93c1 100644
@@ -54,7 +54,7 @@ __expr__parse(double *val, struct parse_ctx *ctx, const char *expr,
 
 int expr__parse(double *final_val, struct parse_ctx *ctx, const char *expr)
 {
-	return __expr__parse(final_val, ctx, expr, EXPR_PARSE);
+	return __expr__parse(final_val, ctx, expr, EXPR_PARSE) ? -1 : 0;
 }
 
 static bool
Signature · 1 line
2.24.1
#7 lore raw
In reply to Jiri Olsa
Quoted 8 lines · Jiri Olsa

hi,
while preparing changes for user defined metric expressions I also moved the expression manual parser to flex.

The reason is to have an easy and reasonable way to support and parse multiple user-defined metric expressions from command line or file.

Quoted 9 lines

I was posponing the change, but I just saw another update to the expr manual scanner (from Kajol Jain), so cherry picked just the expr flex code changes to get it out.

Kajol Jain,
I think it should ease up your change for unknown values marked by '?'. Would you consider rebasing your changes on top of this?

kajoljain found and issue in this one, I'll send v3 as soon as he confirms the fix

jirka

Quoted 31 lines

v2 changes:
- handle special chars properly
- fix return value for expr__parse

Available also in:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/metric_flex

thanks,
jirka

---
Jiri Olsa (5):
perf expr: Add expr.c object
perf expr: Move expr lexer to flex
perf expr: Increase EXPR_MAX_OTHER
perf expr: Straighten expr__parse/expr__find_other interface perf expr: Make expr__parse return -1 on error

 tools/perf/tests/expr.c       |  10 +++---
 tools/perf/util/Build         |  11 ++++++-
 tools/perf/util/expr.c        | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/expr.h        |   8 ++---
 tools/perf/util/expr.l        |  87 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/expr.y        | 208 ++++++++++++++++++++++++++++++++-----------------------------------------------------------------------------------------
 tools/perf/util/stat-shadow.c |   4 +--
 7 files changed, 272 insertions(+), 168 deletions(-)
 create mode 100644 tools/perf/util/expr.c
 create mode 100644 tools/perf/util/expr.l
#8 lore raw

Em Thu, Feb 27, 2020 at 01:10:00PM +0100, Jiri Olsa escreveu:

Quoted 13 lines · Jiri Olsa
Quoted 12 lines

hi,
while preparing changes for user defined metric expressions I also moved the expression manual parser to flex.

The reason is to have an easy and reasonable way to support and parse multiple user-defined metric expressions from command line or file.

I was posponing the change, but I just saw another update to the expr manual scanner (from Kajol Jain), so cherry picked just the expr flex code changes to get it out.

Quoted 8 lines
Quoted 5 lines

Kajol Jain,
I think it should ease up your change for unknown values marked by '?'. Would you consider rebasing your changes on top of this?

kajoljain found and issue in this one, I'll send v3 as soon as he confirms the fix

Ok, I'll hold off processing those patches then, thanks!

- Arnaldo

Quoted 35 lines

jirka

Quoted 31 lines

v2 changes:
- handle special chars properly
- fix return value for expr__parse

Available also in:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/metric_flex

thanks,
jirka

---
Jiri Olsa (5):
perf expr: Add expr.c object
perf expr: Move expr lexer to flex
perf expr: Increase EXPR_MAX_OTHER
perf expr: Straighten expr__parse/expr__find_other interface perf expr: Make expr__parse return -1 on error

 tools/perf/tests/expr.c       |  10 +++---
 tools/perf/util/Build         |  11 ++++++-
 tools/perf/util/expr.c        | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/expr.h        |   8 ++---
 tools/perf/util/expr.l        |  87 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/expr.y        | 208 ++++++++++++++++++++++++++++++++-----------------------------------------------------------------------------------------
 tools/perf/util/stat-shadow.c |   4 +--
 7 files changed, 272 insertions(+), 168 deletions(-)
 create mode 100644 tools/perf/util/expr.c
 create mode 100644 tools/perf/util/expr.l
Signature · 1 line
- Arnaldo