perf expr: Add flex scanner
Jiri Olsa
reviewerv2 cover perf expr: Add flex scanner
hi,
while preparing changes for user defined metric expressions I also moved the expression manual parser to flex.
The reason is to have an easy and reasonable way to support and parse multiple user-defined metric expressions from command line or file.
I was posponing the change, but I just saw another update to the expr manual scanner (from Kajol Jain), so cherry picked just the expr flex code changes to get it out.
Kajol Jain,
I think it should ease up your change for unknown values marked by '?'. Would you consider rebasing your changes on top of this?
v2 changes:
- handle special chars properly
- fix return value for expr__parse
Available also in:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/metric_flex
thanks,
jirka
---
Jiri Olsa (5):
perf expr: Add expr.c object
perf expr: Move expr lexer to flex
perf expr: Increase EXPR_MAX_OTHER
perf expr: Straighten expr__parse/expr__find_other interface perf expr: Make expr__parse return -1 on error
tools/perf/tests/expr.c | 10 +++--- tools/perf/util/Build | 11 ++++++- tools/perf/util/expr.c | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/expr.h | 8 ++--- tools/perf/util/expr.l | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/expr.y | 208 ++++++++++++++++++++++++++++++++----------------------------------------------------------------------------------------- tools/perf/util/stat-shadow.c | 4 +-- 7 files changed, 272 insertions(+), 168 deletions(-) create mode 100644 tools/perf/util/expr.c create mode 100644 tools/perf/util/expr.l
Jiri Olsa
reviewerPATCH v2 1/5 perf expr: Add expr.c object
Add generic expr code into new expr.c object.
The expr.c object will be mainly used in following change that will get rid of the manual flex code,
--- tools/perf/util/Build | 1 + tools/perf/util/expr.c | 19 +++++++++++++++++++ tools/perf/util/expr.y | 16 ---------------- 3 files changed, 20 insertions(+), 16 deletions(-) create mode 100644 tools/perf/util/expr.c
tools/perf/util/Build +1 -0
@@ -122,6 +122,7 @@ perf-y += vsprintf.o perf-y += units.o perf-y += time-utils.o perf-y += expr-bison.o +perf-y += expr.o perf-y += branch.o perf-y += mem2node.o
tools/perf/util/expr.c new file +19 -0
@@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <assert.h> +#include "expr.h" + +/* Caller must make sure id is allocated */ +void expr__add_id (struct parse_ctx *ctx,const char *name,double val) +{ +int idx; + +assert (ctx->num_ids <MAX_PARSE_ID ); + idx = ctx->num_ids++; + ctx->ids[idx].name = name; + ctx->ids[idx].val = val; +} + +void expr__ctx_init (struct parse_ctx *ctx) +{ + ctx->num_ids =0 ; +}
tools/perf/util/expr.y +0 -16
@@ -6,7 +6,6 @@ #define IN_EXPR_Y 1 #include "expr.h" #include "smt.h" -#include <assert.h> #include <string.h> #define MAXIDLEN 256
@@ -169,21 +168,6 @@ static int expr__lex(YYSTYPE *res, const char **pp) return tok; } -/* Caller must make sure id is allocated */ -void expr__add_id(struct parse_ctx *ctx, const char *name, double val) -{ - int idx; - assert(ctx->num_ids < MAX_PARSE_ID); - idx = ctx->num_ids++; - ctx->ids[idx].name = name; - ctx->ids[idx].val = val; -} - -void expr__ctx_init(struct parse_ctx *ctx) -{ - ctx->num_ids = 0; -} - static bool already_seen(const char *val, const char *one, const char **other, int num_other) {
Signature · 1 line
2.24.1
Jiri Olsa
reviewerPATCH v2 2/5 perf expr: Move expr lexer to flex
In reply to Jiri OlsaAdding expr flex code instead of the manual parser code. So it's easily extensible in upcoming changes.
The new flex code is in flex.l object and gets compiled like all the other flexers we use. It's defined as flex reentrant parser.
It's used by both expr__parse and expr__find_other interfaces by separating the starting point.
There's no intended change of functionality ;-) the test expr is passing.
--- tools/perf/util/Build | 10 ++- tools/perf/util/expr.c | 93 ++++++++++++++++++++ tools/perf/util/expr.h | 2 - tools/perf/util/expr.l | 87 +++++++++++++++++++ tools/perf/util/expr.y | 192 ++++++++++++----------------------------- 5 files changed, 243 insertions(+), 141 deletions(-) create mode 100644 tools/perf/util/expr.l
tools/perf/util/Build +9 -1
@@ -121,6 +121,7 @@ perf-y += mem-events.o perf-y += vsprintf.o perf-y += units.o perf-y += time-utils.o +perf-y += expr-flex.o perf-y += expr-bison.o perf-y += expr.o perf-y += branch.o
@@ -190,9 +191,13 @@ $(OUTPUT)util/parse-events-bison.c: util/parse-events.y $(call rule_mkdir) $(Q)$(call echo-cmd,bison)$(BISON) -v util/parse-events.y -d $(PARSER_DEBUG_BISON) -o $@ -p parse_events_ +$(OUTPUT)util/expr-flex.c: util/expr.l $(OUTPUT)util/expr-bison.c + $(call rule_mkdir) + $(Q)$(call echo-cmd,flex)$(FLEX) -o $@ --header-file=$(OUTPUT)util/expr-flex.h $(PARSER_DEBUG_FLEX) util/expr.l + $(OUTPUT)util/expr-bison.c: util/expr.y $(call rule_mkdir) - $(Q)$(call echo-cmd,bison)$(BISON) -v util/expr.y -d $(PARSER_DEBUG_BISON) -o $@ -p expr__ + $(Q)$(call echo-cmd,bison)$(BISON) -v util/expr.y -d $(PARSER_DEBUG_BISON) -o $@ -p expr_ $(OUTPUT)util/pmu-flex.c: util/pmu.l $(OUTPUT)util/pmu-bison.c $(call rule_mkdir)
@@ -204,12 +209,14 @@ $(OUTPUT)util/pmu-bison.c: util/pmu.y CFLAGS_parse-events-flex.o += -w CFLAGS_pmu-flex.o += -w +CFLAGS_expr-flex.o += -w CFLAGS_parse-events-bison.o += -DYYENABLE_NLS=0 -w CFLAGS_pmu-bison.o += -DYYENABLE_NLS=0 -DYYLTYPE_IS_TRIVIAL=0 -w CFLAGS_expr-bison.o += -DYYENABLE_NLS=0 -DYYLTYPE_IS_TRIVIAL=0 -w $(OUTPUT)util/parse-events.o: $(OUTPUT)util/parse-events-flex.c $(OUTPUT)util/parse-events-bison.c $(OUTPUT)util/pmu.o: $(OUTPUT)util/pmu-flex.c $(OUTPUT)util/pmu-bison.c +$(OUTPUT)util/expr.o: $(OUTPUT)util/expr-flex.c $(OUTPUT)util/expr-bison.c CFLAGS_bitmap.o += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))" CFLAGS_find_bit.o += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
@@ -217,6 +224,7 @@ CFLAGS_rbtree.o += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ET CFLAGS_libstring.o += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))" CFLAGS_hweight.o += -Wno-unused-parameter -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))" CFLAGS_parse-events.o += -Wno-redundant-decls +CFLAGS_expr.o += -Wno-redundant-decls CFLAGS_header.o += -include $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)util/kallsyms.o: ../lib/symbol/kallsyms.c FORCE
tools/perf/util/expr.c +93 -0
102 lines not on this page · raw message
tools/perf/util/expr.h +0 -2
@@ -17,9 +17,7 @@ struct parse_ctx {void expr__ctx_init (struct parse_ctx *ctx);void expr__add_id (struct parse_ctx *ctx,const char *id,double val); -#ifndef IN_EXPR_Y int expr__parse (double *final_val,struct parse_ctx *ctx,const char **pp); -#endif int expr__find_other (const char *p,const char *one,const char ***other,int *num_other);
tools/perf/util/expr.l new file +87 -0
87 lines not on this page · raw message
tools/perf/util/expr.y +54 -138
233 lines not on this page · raw message
Signature · 1 line
2.24.1
Jiri Olsa
reviewerPATCH v2 3/5 perf expr: Increase EXPR_MAX_OTHER
In reply to Jiri OlsaThere's no need to be greedy on allowed variables, also when some of the metrics define more than 15 variables, like Branch_Misprediction_Cost.
Increasing the maximum to 100.
--- tools/perf/util/expr.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/util/expr.h +1 -1
@@ -2,7 +2,7 @@#ifndef PARSE_CTX_H #define PARSE_CTX_H 1 -#define EXPR_MAX_OTHER 15 +#define EXPR_MAX_OTHER 100#define MAX_PARSE_ID EXPR_MAX_OTHER struct parse_id {
Signature · 1 line
2.24.1
Re: [PATCH 3/5] perf expr: Increase EXPR_MAX_OTHER
Quoted 5 lines · Jiri Olsa
There's no need to be greedy on allowed variables, also when some of the metrics define more than 15 variables, like
Branch_Misprediction_Cost.Increasing the maximum to 100.
FWIW some of the algorithms (e.g. already_seen) have O(n^2) complexity.
If you really want that many probably would need to add some hash tables.
-Andi
Jiri Olsa
reviewerPATCH v2 4/5 perf expr: Straighten expr__parse/expr__find_other interface
In reply to Jiri OlsaNow with flex parser we don't need to update parsed string pointer, so the interface can just passed the pointer to the expression instead of pointer to pointer.
--- tools/perf/tests/expr.c | 6 +++--- tools/perf/util/expr.c | 8 ++++---- tools/perf/util/expr.h | 4 ++-- tools/perf/util/stat-shadow.c | 4 +--- 4 files changed, 10 insertions(+), 12 deletions(-)
tools/perf/tests/expr.c +3 -3
@@ -10,7 +10,7 @@ static int test(struct parse_ctx *ctx, const char *e, double val2) {double val; -if (expr__parse (&val, ctx, &e)) +if (expr__parse (&val, ctx, e))TEST_ASSERT_VAL ("parse test failed" ,0 );TEST_ASSERT_VAL ("unexpected value" , val == val2);return 0 ;
@@ -44,11 +44,11 @@ int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)return ret; p ="FOO/0" ; - ret =expr__parse (&val, &ctx, &p); + ret =expr__parse (&val, &ctx, p);TEST_ASSERT_VAL ("division by zero" , ret ==1 ); p ="BAR/" ; - ret =expr__parse (&val, &ctx, &p); + ret =expr__parse (&val, &ctx, p);TEST_ASSERT_VAL ("missing operand" , ret ==1 ); TEST_ASSERT_VAL("find other",
tools/perf/util/expr.c +4 -4
@@ -52,9 +52,9 @@ __expr__parse(double *val, struct parse_ctx *ctx, const char *expr,return ret; } -int expr__parse (double *final_val,struct parse_ctx *ctx,const char **pp) +int expr__parse (double *final_val,struct parse_ctx *ctx,const char *expr) { -return __expr__parse (final_val, ctx, *pp,EXPR_PARSE ); +return __expr__parse (final_val, ctx, expr,EXPR_PARSE ); } static bool
@@ -71,14 +71,14 @@ already_seen(const char *val, const char *one, const char **other,return false; } -int expr__find_other (const char *p,const char *one,const char ***other, +int expr__find_other (const char *expr,const char *one,const char ***other,int *num_other) {int err, i =0 , j =0 ;struct parse_ctx ctx;expr__ctx_init (&ctx); - err =__expr__parse (NULL , &ctx, p,EXPR_OTHER ); + err =__expr__parse (NULL , &ctx, expr,EXPR_OTHER );if (err)return -1 ;
tools/perf/util/expr.h +2 -2
@@ -17,8 +17,8 @@ struct parse_ctx {void expr__ctx_init (struct parse_ctx *ctx);void expr__add_id (struct parse_ctx *ctx,const char *id,double val); -int expr__parse (double *final_val,struct parse_ctx *ctx,const char **pp); -int expr__find_other (const char *p,const char *one,const char ***other, +int expr__parse (double *final_val,struct parse_ctx *ctx,const char *expr); +int expr__find_other (const char *expr,const char *one,const char ***other,int *num_other);#endif
tools/perf/util/stat-shadow.c +1 -3
@@ -779,9 +779,7 @@ static void generic_metric(struct perf_stat_config *config, }if (!metric_events[i]) { -const char *p = metric_expr; - -if (expr__parse (&ratio, &pctx, &p) ==0 ) { +if (expr__parse (&ratio, &pctx, metric_expr) ==0 ) {char *unit;char metric_bf[64 ];
Signature · 1 line
2.24.1
Jiri Olsa
reviewerPATCH v2 5/5 perf expr: Make expr__parse return -1 on error
In reply to Jiri OlsaTo match the error value of the expr__find_other function, so all exported expr functions return the same values:
0 on success, -1 on error.
--- tools/perf/tests/expr.c | 4 ++-- tools/perf/util/expr.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
tools/perf/tests/expr.c +2 -2
@@ -45,11 +45,11 @@ int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused) p ="FOO/0" ; ret =expr__parse (&val, &ctx, p); -TEST_ASSERT_VAL ("division by zero" , ret ==1 ); +TEST_ASSERT_VAL ("division by zero" , ret ==-1 ); p ="BAR/" ; ret =expr__parse (&val, &ctx, p); -TEST_ASSERT_VAL ("missing operand" , ret ==1 ); +TEST_ASSERT_VAL ("missing operand" , ret ==-1 );TEST_ASSERT_VAL ("find other" ,expr__find_other ("FOO + BAR + BAZ + BOZO" ,"FOO" , &other, &num_other) ==0 );
tools/perf/util/expr.c +1 -1
@@ -54,7 +54,7 @@ __expr__parse(double *val, struct parse_ctx *ctx, const char *expr,int expr__parse (double *final_val,struct parse_ctx *ctx,const char *expr) { -return __expr__parse (final_val, ctx, expr,EXPR_PARSE ); +return __expr__parse (final_val, ctx, expr,EXPR_PARSE ) ?-1 :0 ; } static bool
Signature · 1 line
2.24.1
Quoted 8 lines · Jiri Olsa
hi,
while preparing changes for user defined metric expressions I also moved the expression manual parser to flex.The reason is to have an easy and reasonable way to support and parse multiple user-defined metric expressions from command line or file.
Quoted 9 lines
I was posponing the change, but I just saw another update to the expr manual scanner (from Kajol Jain), so cherry picked just the expr flex code changes to get it out.
Kajol Jain,
I think it should ease up your change for unknown values marked by '?'. Would you consider rebasing your changes on top of this?
kajoljain found and issue in this one, I'll send v3 as soon as he confirms the fix
jirka
Quoted 31 lines
v2 changes:
- handle special chars properly
- fix return value forexpr__parseAvailable also in:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/metric_flexthanks,
jirka---
Jiri Olsa (5):
perf expr: Addexpr.cobject
perf expr: Move expr lexer to flex
perf expr: IncreaseEXPR_MAX_OTHER
perf expr: Straighten expr__parse/expr__find_other interface perf expr: Makeexpr__parsereturn -1 on errortools/perf/tests/expr.c | 10 +++--- tools/perf/util/Build | 11 ++++++- tools/perf/util/expr.c | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/expr.h | 8 ++--- tools/perf/util/expr.l | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/expr.y | 208 ++++++++++++++++++++++++++++++++----------------------------------------------------------------------------------------- tools/perf/util/stat-shadow.c | 4 +-- 7 files changed, 272 insertions(+), 168 deletions(-) create mode 100644 tools/perf/util/expr.c create mode 100644 tools/perf/util/expr.l
Em Thu, Feb 27, 2020 at 01:10:00PM +0100, Jiri Olsa escreveu:
Quoted 13 lines · Jiri Olsa
Quoted 12 lines
hi,
while preparing changes for user defined metric expressions I also moved the expression manual parser to flex.The reason is to have an easy and reasonable way to support and parse multiple user-defined metric expressions from command line or file.
I was posponing the change, but I just saw another update to the expr manual scanner (from Kajol Jain), so cherry picked just the expr flex code changes to get it out.
Quoted 8 lines
Quoted 5 lines
Kajol Jain,
I think it should ease up your change for unknown values marked by '?'. Would you consider rebasing your changes on top of this?kajoljain found and issue in this one, I'll send v3 as soon as he confirms the fix
Ok, I'll hold off processing those patches then, thanks!
- Arnaldo
Quoted 35 lines
jirka
Quoted 31 lines
v2 changes:
- handle special chars properly
- fix return value forexpr__parseAvailable also in:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/metric_flexthanks,
jirka---
Jiri Olsa (5):
perf expr: Addexpr.cobject
perf expr: Move expr lexer to flex
perf expr: IncreaseEXPR_MAX_OTHER
perf expr: Straighten expr__parse/expr__find_other interface perf expr: Makeexpr__parsereturn -1 on errortools/perf/tests/expr.c | 10 +++--- tools/perf/util/Build | 11 ++++++- tools/perf/util/expr.c | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/expr.h | 8 ++--- tools/perf/util/expr.l | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/expr.y | 208 ++++++++++++++++++++++++++++++++----------------------------------------------------------------------------------------- tools/perf/util/stat-shadow.c | 4 +-- 7 files changed, 272 insertions(+), 168 deletions(-) create mode 100644 tools/perf/util/expr.c create mode 100644 tools/perf/util/expr.l
Signature · 1 line
- Arnaldo