perf tools: Including pre-generated flex files
Hi Linus,
We now use lex and bison for the events and PMU parser:
[acme@sandy linux]$ wc -l tools/perf/util/*.[yl]
126 tools/perf/util/parse-events.l
229 tools/perf/util/parse-events.y
43 tools/perf/util/pmu.l
93 tools/perf/util/pmu.y
491 total
Jiri and Peter agreed on having pre-generated files:
[acme@sandy linux]$ wc -l tools/perf/util/*-{bison,flex}.[ch]
1917 tools/perf/util/parse-events-bison.c
81 tools/perf/util/parse-events-bison.h
1663 tools/perf/util/pmu-bison.c
73 tools/perf/util/pmu-bison.h
2272 tools/perf/util/parse-events-flex.c
316 tools/perf/util/parse-events-flex.h
1821 tools/perf/util/pmu-flex.c
316 tools/perf/util/pmu-flex.h
8459 total
so that we don't require even more tools to build the kernel, following the precedent of:
[acme@sandy linux]$ wc -l scripts/*/*.[ch]_shipped
1976 scripts/dtc/dtc-lexer.lex.c_shipped 1946 scripts/dtc/dtc-parser.tab.c_shipped
86 scripts/dtc/dtc-parser.tab.h_shipped
220 scripts/genksyms/keywords.hash.c_shipped
2245 scripts/genksyms/lex.lex.c_shipped 2399 scripts/genksyms/parse.tab.c_shipped
95 scripts/genksyms/parse.tab.h_shipped
286 scripts/kconfig/zconf.hash.c_shipped
2420 scripts/kconfig/zconf.lex.c_shipped 2504 scripts/kconfig/zconf.tab.c_shipped
14177 total
[acme@sandy linux]$
But Ingo argues that flex and bison are just a yum/whatever
install away, so we shouldn't clutter the kernel git history with things we can generate at build time.
What is your take on this?
- Arnaldo
Quoted 7 lines · Arnaldo Carvalho de Melo
[...]
But Ingo argues that flex and bison are just a
yum/whatever install away, so we shouldn't clutter the kernel git history with things we can generate at build time.What is your take on this?
I.e. we could just autogenerate automatically via something like the patch below.
My main worry was that Linus would (somewhat rightfully) object to the large diffstats that these things introduce:
tools/perf/util/parse-events-bison.c | 1917 +++++++++++++++++ tools/perf/util/parse-events-bison.h | 81 + tools/perf/util/parse-events-flex.c | 2272 ++++++++++++++++++++ tools/perf/util/parse-events-flex.h | 316 +++ tools/perf/util/parse-events.c | 603 +++--- tools/perf/util/parse-events.h | 49 + tools/perf/util/parse-events.l | 126 ++ tools/perf/util/parse-events.y | 229 ++ tools/perf/util/pmu-bison.c | 1663 ++++++++++++++ tools/perf/util/pmu-bison.h | 73 + tools/perf/util/pmu-flex.c | 1821 ++++++++++++++++ tools/perf/util/pmu-flex.h | 316 +++
I mean, since auto-generating them is in the Makefile *already*, the only argument in favor of it is that whether we want to make the perf build depend on bison and flex.
Anyway, it would be nice if we could hear an official opinion on this. I didn't want to send the latest perf/urgent with this diffstat until this is settled one way or another.
Thanks,
Ingo
------------->
Quoted 1 line
From
3cfa99c3a09224cc270ed1e1260cc4fe1b79cebcMon Sep 17 00:00:00 2001
From: Ingo Molnar <[email protected]>
Date: Tue, 27 Mar 2012 14:07:50 +0200
Subject: [PATCH] perf tools: Remove auto-generated bison/flex files
These should not be in the Git history - they are auto-generated.
Extend the Makefile rules of the parser files to include the generation run.
--- tools/perf/Makefile | 25 +- tools/perf/util/parse-events-bison.c | 1917 ---------------------------- tools/perf/util/parse-events-bison.h | 81 -- tools/perf/util/parse-events-flex.c | 2272 ---------------------------------- tools/perf/util/parse-events-flex.h | 316 ----- tools/perf/util/pmu-bison.c | 1663 ------------------------- tools/perf/util/pmu-bison.h | 73 -- tools/perf/util/pmu-flex.c | 1821 --------------------------- tools/perf/util/pmu-flex.h | 316 ----- 9 files changed, 15 insertions(+), 8469 deletions(-)
tools/perf/Makefile +15 -10
@@ -61,8 +61,6 @@ ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/i386/ -e s/sun4u/sparc64/ \ CC = $(CROSS_COMPILE)gcc AR = $(CROSS_COMPILE)ar -FLEX = $(CROSS_COMPILE)flex -BISON= $(CROSS_COMPILE)bison # Additional ARCH settings for x86 ifeq ($(ARCH),i386)
@@ -236,6 +234,21 @@ endif export PERL_PATH +FLEX = $(CROSS_COMPILE)flex +BISON= $(CROSS_COMPILE)bison + +event-parser: + $(QUIET_BISON)$(BISON) -v util/parse-events.y -d -o util/parse-events-bison.c + $(QUIET_FLEX)$(FLEX) --header-file=util/parse-events-flex.h -t util/parse-events.l > util/parse-events-flex.c + +util/parse-events-flex.c: event-parser + +pmu-parser: + $(QUIET_BISON)$(BISON) -v util/pmu.y -d -o util/pmu-bison.c + $(QUIET_FLEX)$(FLEX) --header-file=util/pmu-flex.h -t util/pmu.l > util/pmu-flex.c + +util/pmu-flex.c: pmu-parser + LIB_FILE=$(OUTPUT)libperf.a LIB_H += ../../include/linux/perf_event.h
@@ -883,14 +896,6 @@ cscope: $(RM) cscope* $(FIND) . -name '*.[hcS]' -print | xargs cscope -b -event-parser: - $(QUIET_BISON)$(BISON) -v util/parse-events.y -d -o util/parse-events-bison.c - $(QUIET_FLEX)$(FLEX) --header-file=util/parse-events-flex.h -t util/parse-events.l > util/parse-events-flex.c - -pmu-parser: - $(QUIET_BISON)$(BISON) -v util/pmu.y -d -o util/pmu-bison.c - $(QUIET_FLEX)$(FLEX) --header-file=util/pmu-flex.h -t util/pmu.l > util/pmu-flex.c - ### Detect prefix changes TRACK_CFLAGS = $(subst ','\'',$(ALL_CFLAGS)):\ $(bindir_SQ):$(perfexecdir_SQ):$(template_dir_SQ):$(prefix_SQ)
[...]
Jiri Olsa
Tested-byQuoted 12 lines · Ingo Molnar
Quoted 7 lines · Arnaldo Carvalho de Melo
[...]
But Ingo argues that flex and bison are just a
yum/whatever install away, so we shouldn't clutter the kernel git history with things we can generate at build time.What is your take on this?
I.e. we could just autogenerate automatically via something like
SNIP
Quoted 7 lines
From
3cfa99c3a09224cc270ed1e1260cc4fe1b79cebcMon Sep 17 00:00:00 2001
From: Ingo Molnar <[email protected]>
Date: Tue, 27 Mar 2012 14:07:50 +0200
Subject: [PATCH] perf tools: Remove auto-generated bison/flex filesThese should not be in the Git history - they are auto-generated.
Quoted 8 lines
Extend the
Makefilerules of the parser files to include the generation run.Cc Peter Zijlstra <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Signed-off-by Ingo Molnar <[email protected]>---
thanks,
jirka
Quoted 62 lines
tools/perf/Makefile | 25 +- tools/perf/util/parse-events-bison.c | 1917 ---------------------------- tools/perf/util/parse-events-bison.h | 81 -- tools/perf/util/parse-events-flex.c | 2272 ---------------------------------- tools/perf/util/parse-events-flex.h | 316 ----- tools/perf/util/pmu-bison.c | 1663 ------------------------- tools/perf/util/pmu-bison.h | 73 -- tools/perf/util/pmu-flex.c | 1821 --------------------------- tools/perf/util/pmu-flex.h | 316 ----- 9 files changed, 15 insertions(+), 8469 deletions(-)tools/perf/Makefile +15
-10@@ -61,8 +61,6 @@ ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/i386/ -e s/sun4u/sparc64/ \ CC = $(CROSS_COMPILE)gcc AR = $(CROSS_COMPILE)ar -FLEX = $(CROSS_COMPILE)flex -BISON= $(CROSS_COMPILE)bison # Additional ARCH settings for x86 ifeq ($(ARCH),i386)@@ -236,6 +234,21 @@ endif export PERL_PATH +FLEX = $(CROSS_COMPILE)flex +BISON= $(CROSS_COMPILE)bison + +event-parser: + $(QUIET_BISON)$(BISON) -v util/parse-events.y -d -o util/parse-events-bison.c + $(QUIET_FLEX)$(FLEX) --header-file=util/parse-events-flex.h -t util/parse-events.l > util/parse-events-flex.c + +util/parse-events-flex.c: event-parser + +pmu-parser: + $(QUIET_BISON)$(BISON) -v util/pmu.y -d -o util/pmu-bison.c + $(QUIET_FLEX)$(FLEX) --header-file=util/pmu-flex.h -t util/pmu.l > util/pmu-flex.c + +util/pmu-flex.c: pmu-parser + LIB_FILE=$(OUTPUT)libperf.a LIB_H += ../../include/linux/perf_event.h@@ -883,14 +896,6 @@ cscope: $(RM) cscope* $(FIND) . -name '*.[hcS]' -print | xargs cscope -b -event-parser: - $(QUIET_BISON)$(BISON) -v util/parse-events.y -d -o util/parse-events-bison.c - $(QUIET_FLEX)$(FLEX) --header-file=util/parse-events-flex.h -t util/parse-events.l > util/parse-events-flex.c - -pmu-parser: - $(QUIET_BISON)$(BISON) -v util/pmu.y -d -o util/pmu-bison.c - $(QUIET_FLEX)$(FLEX) --header-file=util/pmu-flex.h -t util/pmu.l > util/pmu-flex.c - ### Detect prefix changes TRACK_CFLAGS = $(subst ','\'',$(ALL_CFLAGS)):\ $(bindir_SQ):$(perfexecdir_SQ):$(template_dir_SQ):$(prefix_SQ)
[tip:perf/urgent] perf tools: Remove auto-generated bison/ flex files
In reply to Ingo MolnarCommit-ID: 65f3e56e0c81d1f581c4bdef7646ae5a8d51f494 Gitweb: http://git.kernel.org/tip/65f3e56e0c81d1f581c4bdef7646ae5a8d51f494 Author: Ingo Molnar <[email protected]> AuthorDate: Thu, 29 Mar 2012 13:08:38 -0300 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Fri, 30 Mar 2012 12:43:17 -0300
perf tools: Remove auto-generated bison/flex files
These should not be in the Git history - they are auto-generated.
Extend the Makefile rules of the parser files to include the generation run.
--- tools/perf/Makefile | 47 +- tools/perf/util/parse-events-bison.c | 1917 ---------------------------- tools/perf/util/parse-events-bison.h | 81 -- tools/perf/util/parse-events-flex.c | 2272 ---------------------------------- tools/perf/util/parse-events-flex.h | 316 ----- tools/perf/util/pmu-bison.c | 1663 ------------------------- tools/perf/util/pmu-bison.h | 73 -- tools/perf/util/pmu-flex.c | 1821 --------------------------- tools/perf/util/pmu-flex.h | 316 ----- 9 files changed, 30 insertions(+), 8476 deletions(-)
tools/perf/Makefile +30 -17
@@ -61,8 +61,6 @@ ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/i386/ -e s/sun4u/sparc64/ \ CC = $(CROSS_COMPILE)gcc AR = $(CROSS_COMPILE)ar -FLEX = $(CROSS_COMPILE)flex -BISON= $(CROSS_COMPILE)bison # Additional ARCH settings for x86 ifeq ($(ARCH),i386)
@@ -184,7 +182,7 @@ endif ### --- END CONFIGURATION SECTION --- -BASIC_CFLAGS = -Iutil/include -Iarch/$(ARCH)/include -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE +BASIC_CFLAGS = -Iutil/include -Iarch/$(ARCH)/include -I$(OUTPUT)/util -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE BASIC_LDFLAGS = # Guard against environment variables
@@ -236,6 +234,25 @@ endif export PERL_PATH +FLEX = $(CROSS_COMPILE)flex +BISON= $(CROSS_COMPILE)bison + +event-parser: + $(QUIET_BISON)$(BISON) -v util/parse-events.y -d -o $(OUTPUT)util/parse-events-bison.c + $(QUIET_FLEX)$(FLEX) --header-file=$(OUTPUT)util/parse-events-flex.h -t util/parse-events.l > $(OUTPUT)util/parse-events-flex.c + +$(OUTPUT)util/parse-events-flex.c: event-parser +$(OUTPUT)util/parse-events-bison.c: event-parser + +pmu-parser: + $(QUIET_BISON)$(BISON) -v util/pmu.y -d -o $(OUTPUT)util/pmu-bison.c + $(QUIET_FLEX)$(FLEX) --header-file=$(OUTPUT)util/pmu-flex.h -t util/pmu.l > $(OUTPUT)util/pmu-flex.c + +$(OUTPUT)util/pmu-flex.c: pmu-parser +$(OUTPUT)util/pmu-bison.c: pmu-parser + +$(OUTPUT)util/parse-events.o: event-parser pmu-parser + LIB_FILE=$(OUTPUT)libperf.a LIB_H += ../../include/linux/perf_event.h
@@ -754,6 +771,15 @@ $(OUTPUT)perf.o perf.spec \ .SUFFIXES: .SUFFIXES: .o .c .S .s +# These two need to be here so that when O= is not used they take precedence +# over the general rule for .o + +$(OUTPUT)util/%-flex.o: $(OUTPUT)util/%-flex.c $(OUTPUT)PERF-CFLAGS + $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -Iutil/ -Wno-redundant-decls -Wno-switch-default -Wno-unused-function $< + +$(OUTPUT)util/%-bison.o: $(OUTPUT)util/%-bison.c $(OUTPUT)PERF-CFLAGS + $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -DYYENABLE_NLS=0 -DYYLTYPE_IS_TRIVIAL=0 -Iutil/ -Wno-redundant-decls -Wno-switch-default -Wno-unused-function $< + $(OUTPUT)%.o: %.c $(OUTPUT)PERF-CFLAGS $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) $< $(OUTPUT)%.i: %.c $(OUTPUT)PERF-CFLAGS
@@ -790,12 +816,6 @@ $(OUTPUT)util/ui/browsers/map.o: util/ui/browsers/map.c $(OUTPUT)PERF-CFLAGS $(OUTPUT)util/rbtree.o: ../../lib/rbtree.c $(OUTPUT)PERF-CFLAGS $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -DETC_PERFCONFIG='"$(ETC_PERFCONFIG_SQ)"' $< -$(OUTPUT)util/parse-events-flex.o: util/parse-events-flex.c $(OUTPUT)PERF-CFLAGS - $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -Wno-redundant-decls -Wno-switch-default -Wno-unused-function $< - -$(OUTPUT)util/pmu-flex.o: util/pmu-flex.c $(OUTPUT)PERF-CFLAGS - $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -Wno-redundant-decls -Wno-switch-default -Wno-unused-function $< - $(OUTPUT)util/scripting-engines/trace-event-perl.o: util/scripting-engines/trace-event-perl.c $(OUTPUT)PERF-CFLAGS $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) $(PERL_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-shadow $<
@@ -883,14 +903,6 @@ cscope: $(RM) cscope* $(FIND) . -name '*.[hcS]' -print | xargs cscope -b -event-parser: - $(QUIET_BISON)$(BISON) -v util/parse-events.y -d -o util/parse-events-bison.c - $(QUIET_FLEX)$(FLEX) --header-file=util/parse-events-flex.h -t util/parse-events.l > util/parse-events-flex.c - -pmu-parser: - $(QUIET_BISON)$(BISON) -v util/pmu.y -d -o util/pmu-bison.c - $(QUIET_FLEX)$(FLEX) --header-file=util/pmu-flex.h -t util/pmu.l > util/pmu-flex.c - ### Detect prefix changes TRACK_CFLAGS = $(subst ','\'',$(ALL_CFLAGS)):\ $(bindir_SQ):$(perfexecdir_SQ):$(template_dir_SQ):$(prefix_SQ)
@@ -978,6 +990,7 @@ clean: $(RM) *.spec *.pyc *.pyo */*.pyc */*.pyo $(OUTPUT)common-cmds.h TAGS tags cscope* $(MAKE) -C Documentation/ clean $(RM) $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)PERF-CFLAGS + $(RM) $(OUTPUT)util/*-{bison,flex}* $(python-clean) .PHONY: all install clean strip
tools/perf/util/parse-events-bison.c deleted +0 -1917
1,917 lines not on this page · raw message
tools/perf/util/parse-events-bison.h deleted +0 -81
81 lines not on this page · raw message
tools/perf/util/parse-events-flex.c deleted +0 -2272
913 lines not on this page · raw message
tools/perf/util/parse-events-flex.h deleted +0 -316
316 lines not on this page · raw message
tools/perf/util/pmu-bison.c deleted +0 -1663
1,663 lines not on this page · raw message
tools/perf/util/pmu-bison.h deleted +0 -73
73 lines not on this page · raw message
tools/perf/util/pmu-flex.c deleted +0 -1821
1,821 lines not on this page · raw message
tools/perf/util/pmu-flex.h deleted +0 -316
316 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Quoted 6 lines · Arnaldo Carvalho de Melo
But Ingo argues that flex and bison are just a yum/whatever install away, so we shouldn't clutter the kernel git history with things we can generate at build time.
What is your take on this?
I think I agree.
Building the *basic* kernel is different - I want people to be able to build the kernel and install it and test it out with a fairly minimal development environment.
But the perf tools
(a) are mainly useful for developers who can easily install these
things *anyway* (ie we're not talking the same kind of usage cases where we want to encourage random people off the street to build and test)
(b) already require things that are *much* less common than flex (ie
the whole libelf-devel stuff etc)
so I think having pre-generated lex/yacc files is just silly - just install the tool.
Linus
Quoted 25 lines · Linus Torvalds
Quoted 6 lines · Arnaldo Carvalho de Melo
But Ingo argues that flex and bison are just a yum/whatever install away, so we shouldn't clutter the kernel git history with things we can generate at build time.
What is your take on this?
I think I agree.
Building the *basic* kernel is different - I want people to be able to build the kernel and install it and test it out with a fairly minimal development environment.
But the perf tools
(a) are mainly useful for developers who can easily install these
things *anyway* (ie we're not talking the same kind of usage cases where we want to encourage random people off the street to build and test)
(b) already require things that are *much* less common than flex (ie
the whole libelf-devel stuff etc)
so I think having pre-generated lex/yacc files is just silly - just install the tool.
Quoted 1 line
Linus
I just tried to generate the parser on RHEL6 and got:
CC util/parse-events-bison.outil/parse-events-bison.c:222:6: error: "YYENABLE_NLS" is not defined util/parse-events-bison.c:664:6: error: "YYLTYPE_IS_TRIVIAL" is not defined
looks like bison bug, that RHEL6 did not picked up fix for yet.
I agree with not including the parser sources in git, but we'll probably get some screaming from ppl having similar issue like above.
Also I'm not fully sure we could even use generated bison/flex files cross architectures.. need to check :)
jirka
Em Thu, Mar 29, 2012 at 02:22:08PM +0200, Jiri Olsa escreveu:
Quoted 7 lines
I just tried to generate the parser on RHEL6 and got:
CC
util/parse-events-bison.outil/parse-events-bison.c:222:6: error: "YYENABLE_NLS" is not definedutil/parse-events-bison.c:664:6: error: "YYLTYPE_IS_TRIVIAL" is not defined
Quoted 7 lines
looks like bison bug, that RHEL6 did not picked up fix for yet.
I agree with not including the parser sources in git, but we'll probably get some screaming from ppl having similar issue like above.
Also I'm not fully sure we could even use generated bison/flex files cross architectures.. need to check :)
I got yet another error here:
[acme@sandy linux]$ rm -rf ../build/perf/
[acme@sandy linux]$ mkdir ../build/perf/
[acme@sandy linux]$ make -j8 -C tools/perf/ O=/home/git/build/perf install
make: Entering directory `/home/git/linux/tools/perf'
MKDIR /home/git/build/perf/arch/x86/util
<SNIP>
MKDIR /home/git/build/perf/util/ui/browsersPERF_VERSION = perf.urgent.for.mingo.6.g7e025ca
make: Leaving directory `/home/git/linux/tools/perf'
make: Entering directory `/home/git/linux/tools/perf'
GEN /home/git/build/perf/common-cmds.h
* new build flags or prefix
CC /home/git/build/perf/bench/mem-memcpy-x86-64-asm.o
<SNIP>
CC /home/git/build/perf/util/trace-event-parse.o
BISON event-parser
CC /home/git/build/perf/util/parse-events-bison.o
FLEX event-parser
BISON pmu-parserutil/parse-events-bison.c:150:10: error: missing terminating " character util/parse-events-bison.c:150:10: error: ""util/par" is not a valid filenameutil/parse-events-bison.c:145:1: error: unterminated #if util/parse-events-bison.c:147: error: expected specifier-qualifier-list at end of input
make: *** [/home/git/build/perf/util/parse-events-bison.o] Error 1
make: *** Waiting for unfinished jobs....
FLEX pmu-parser
make: Leaving directory `/home/git/linux/tools/perf' [acme@sandy linux]$
- Arnaldo
Quoted 6 lines · Jiri Olsa
I just tried to generate the parser on RHEL6 and got:
CC
util/parse-events-bison.outil/parse-events-bison.c:222:6: error: "YYENABLE_NLS" is not definedutil/parse-events-bison.c:664:6: error: "YYLTYPE_IS_TRIVIAL" is not defined
Have you first deleted all the autogenerated files via:
rm -f util/pmu-* and util/parse-events-*
? ['make clean' should also do this - it doesn't, right now.]
Otherwise you could be mixing different versions of the parser files.
Thanks,
Ingo
Em Thu, Mar 29, 2012 at 04:38:06PM +0200, Ingo Molnar escreveu:
Quoted 8 lines · Jiri Olsa
Quoted 6 lines
I just tried to generate the parser on RHEL6 and got:
CC
util/parse-events-bison.outil/parse-events-bison.c:222:6: error: "YYENABLE_NLS" is not definedutil/parse-events-bison.c:664:6: error: "YYLTYPE_IS_TRIVIAL" is not defined
Quoted 8 lines
Have you first deleted all the autogenerated files via:
rm -f util/pmu-* and util/parse-events-*
? ['make clean' should also do this - it doesn't, right now.]
Otherwise you could be mixing different versions of the parser files.
I don't think so, I discussed this with him and I'm fixing it.
There are several problems, the generate rules weren't obeying O=, etc, I think I got it fixed, testing now.
- arnaldo