xen: rework compat headers generation
v4 cover xen: rework compat headers generation
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.build-system-xen-include-rework-v4
v4:
- new patch removing get_typedefs() from the existing script (and thus also removing it from the new script)
- Adding some optimisation, mainly pre-compile regex It's slightly faster.
v3:
- Rewrite script into python instead of perl.
(last patch of the series)
v2:
- new patch [1/4] to fix issue with command line that can be way too long
- other small changes, and reorder patches
Hi,
This patch series is about 2 improvement. First one is to use $(if_changed, ) in "include/Makefile" to make the generation of the compat headers less verbose and to have the command line part of the decision to rebuild the headers. Second one is to replace one slow script by a much faster one, and save time when generating the headers.
Thanks.
Anthony PERARD (3):
build: include/compat, remove typedefs handling
build: replace get-fields.sh by a python script
build: compat-xlat-header.py: optimisation to search for just '{'
instead of [{}]xen/include/Makefile | 6 +- xen/tools/compat-xlat-header.py | 432 ++++++++++++++++++++++++++ xen/tools/get-fields.sh | 528 -------------------------------- 3 files changed, 434 insertions(+), 532 deletions(-) create mode 100644 xen/tools/compat-xlat-header.py delete mode 100644 xen/tools/get-fields.sh
Signature · 1 line
Anthony PERARD
PATCH v4 1/3 build: include/compat, remove typedefs handling
Partial revert of c93bd0e6ea2a ("tmem: fix 32-on-64 support") Since c492e19fdd05 ("xen: remove tmem from hypervisor"), this code isn't used anymore.
---
Notes:
v4:
- new patch
xen/tools/get-fields.sh | 57 +---------------------------------------- 1 file changed, 1 insertion(+), 56 deletions(-)
xen/tools/get-fields.sh +1 -56
@@ -41,34 +41,6 @@ get_fields () done } -get_typedefs () -{ - local level=1 state= - for token in $1 - do - case "$token" in - typedef) - test $level != 1 || state=1 - ;; - COMPAT_HANDLE\(*\)) - test $level != 1 -o "$state" != 1 || state=2 - ;; - [\{\[]) - level=$(expr $level + 1) - ;; - [\}\]]) - level=$(expr $level - 1) - ;; - ";") - test $level != 1 || state= - ;; - [a-zA-Z_]*) - test $level != 1 -o "$state" != 2 || echo "$token" - ;; - esac - done -} - build_enums () { local level=1 kind= fields= members= named= id= token
@@ -201,21 +173,7 @@ for line in sys.stdin.readlines(): fi ;; [a-zA-Z]*) - if [ -z "$id" -a -z "$type" -a -z "$array_type" ] - then - for id in $typedefs - do - test $id != "$token" || type=$id - done - if [ -z "$type" ] - then - id=$token - else - id= - fi - else - id=$token - fi + id=$token ;; [\,\;]) if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
@@ -330,18 +288,6 @@ build_body () if [ -n "$array" ] then array="$array $token" - elif [ -z "$id" -a -z "$type" -a -z "$array_type" ] - then - for id in $typedefs - do - test $id != "$token" || type=$id - done - if [ -z "$type" ] - then - id=$token - else - id= - fi else id=$token fi
@@ -514,7 +460,6 @@ name=${2#compat_} name=${name#xen} case "$1" in "!") - typedefs="$(get_typedefs "$list")" build_enums $name "$fields" build_body $name "$fields" ;;
Signature · 1 line
Anthony PERARD
Re: [XEN PATCH v4 1/3] build: include/compat, remove typedefs handling
Quoted 5 lines · Anthony PERARD
Partial revert of
c93bd0e6ea2a("tmem: fix 32-on-64 support") Sincec492e19fdd05("xen: remove tmem from hypervisor"), this code isn't used anymore.Signed-off-by Anthony PERARD <[email protected]>
PATCH v4 2/3 build: replace get-fields.sh by a python script
In reply to Anthony PERARDThe get-fields.sh which generate all the include/compat/.xlat/*.h headers is quite slow. It takes for example nearly 3 seconds to generate platform.h on a recent machine, or 2.3 seconds for memory.h.
Rewriting the mix of shell/sed/python into a single python script make the generation of those file a lot faster.
No functional change, the headers generated are identical.
---
Notes:
To test the header generation, I've submit a branch to gitlab ci, where all the headers where generated, and for each one both the shell script and the python script where run and the result of both compared.
v4:
- Use "token in ()" instead of "token in []"
- Pre-compile regex
- removed get_typedefs() as it's been removed from get-fields.sh
Overall, this seems to save 0.07166s, down to 1.6673s when generating all xlat headers sequentially. (before removing get_typedefs)
v3:
convert to python script instead of perl
- this should allow more developper do be able to review/edit it.
- it avoid adding a dependency on perl for the hypervisor build.
It can be twice as slow than the perl version, but overall, when doing a build with make, there isn't much difference between the perl and python script.
We might be able to speed the python script up by precompiling the many regex, but it's probably not worth it. (python already have a cache of compiled regex, but I think it's small, maybe 10 or so)
v2:
- Add .pl extension to the perl script
- remove "-w" from the shebang as it is duplicate of "use warning;"
- Add a note in the commit message that the "headers generated are identical".
xen/include/Makefile | 6 +- xen/tools/compat-xlat-header.py | 432 +++++++++++++++++++++++++++++ xen/tools/get-fields.sh | 473 -------------------------------- 3 files changed, 434 insertions(+), 477 deletions(-) create mode 100644 xen/tools/compat-xlat-header.py delete mode 100644 xen/tools/get-fields.sh
xen/include/Makefile +2 -4
@@ -60,9 +60,7 @@ cmd_compat_c = \ quiet_cmd_xlat_headers = GEN $@ cmd_xlat_headers = \ - while read what name; do \ - $(SHELL) $(srctree)/tools/get-fields.sh "$$what" compat_$$name $< || exit $$?; \ - done <$(patsubst $(obj)/compat/%,$(obj)/compat/.xlat/%,$(basename $<)).lst >[email protected]; \ + $(PYTHON) $(srctree)/tools/compat-xlat-header.py $< $(patsubst $(obj)/compat/%,$(obj)/compat/.xlat/%,$(basename $<)).lst > [email protected]; \ mv -f [email protected] $@ targets += $(headers-y)
@@ -80,7 +78,7 @@ $(obj)/compat/%.c: $(src)/public/%.h $(srcdir)/xlat.lst $(srctree)/tools/compat- $(call if_changed,compat_c) targets += $(patsubst compat/%, compat/.xlat/%, $(headers-y)) -$(obj)/compat/.xlat/%.h: $(obj)/compat/%.h $(obj)/compat/.xlat/%.lst $(srctree)/tools/get-fields.sh FORCE +$(obj)/compat/.xlat/%.h: $(obj)/compat/%.h $(obj)/compat/.xlat/%.lst $(srctree)/tools/compat-xlat-header.py FORCE $(call if_changed,xlat_headers) quiet_cmd_xlat_lst = GEN $@
xen/tools/compat-xlat-header.py new file +432 -0
432 lines not on this page · raw message
xen/tools/get-fields.sh deleted +0 -473
473 lines not on this page · raw message
Signature · 1 line
Anthony PERARD
Re: [XEN PATCH v4 2/3] build: replace get-fields.sh by a python script
Quoted 10 lines · Anthony PERARD
The
get-fields.shwhich generate all theinclude/compat/.xlat/*.hheaders is quite slow. It takes for example nearly 3 seconds to generateplatform.hon a recent machine, or 2.3 seconds formemory.h.Rewriting the mix of shell/sed/python into a single python script make the generation of those file a lot faster.
No functional change, the headers generated are identical.
Signed-off-by Anthony PERARD <[email protected]>
PATCH v4 3/3 build: compat-xlat-header.py: optimisation to search for just '{' instead of [{}]
In reply to Anthony PERARDfields and extrafields always all the parts of a sub-struct, so when there is '}', there is always a '{' before it. Also, both are lists.
--- xen/tools/compat-xlat-header.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
xen/tools/compat-xlat-header.py +2 -2
@@ -105,7 +105,7 @@ def handle_field(prefix, name, id, type, fields): else: k = id.replace('.', '_') print("%sXLAT_%s_HNDL_%s(_d_, _s_);" % (prefix, name, k), end='') - elif not re_brackets.search(' '.join(fields)): + elif not '{' in fields: tag = ' '.join(fields) tag = re.sub(r'\s*(struct|union)\s+(compat_)?(\w+)\s.*', '\\3', tag) print(" \\")
@@ -290,7 +290,7 @@ def build_body(name, tokens): print(" \\\n} while (0)") def check_field(kind, name, field, extrafields): - if not re_brackets.search(' '.join(extrafields)): + if not '{' in extrafields: print("; \\") if len(extrafields) != 0: for token in extrafields:
Signature · 1 line
Anthony PERARD
Re: [XEN PATCH v4 3/3] build: compat-xlat-header.py: optimisation to search for just '{' instead of [{}]
Quoted 23 lines · Anthony PERARD
fieldsandextrafieldsalways all the parts of a sub-struct, so when there is '}', there is always a '{' before it. Also, both are lists.Signed-off-by Anthony PERARD <[email protected]>--- xen/tools/compat-xlat-header.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)xen/tools/compat-xlat-header.py +1
-1@@ -105,7 +105,7 @@ def handle_field(prefix, name, id, type, fields): else: k = id.replace('.', '_') print("%sXLAT_%s_HNDL_%s(_d_, _s_);" % (prefix, name, k), end='') - elif not re_brackets.search(' '.join(fields)): + elif not '{' in fields: tag = ' '.join(fields) tag = re.sub(r'\s*(struct|union)\s+(compat_)?(\w+)\s.*', '\\3', tag) print(" \\")
xen/tools/compat-xlat-header.py def build_body(name, tokens): · line 296
Quoted 8 lines
xen/tools/compat-xlat-header.py +1
-1@@ -290,7 +290,7 @@ def build_body(name, tokens): print(" \\\n} while (0)") def check_field(kind, name, field, extrafields): - if not re_brackets.search(' '.join(extrafields)): + if not '{' in extrafields: print("; \\") if len(extrafields) != 0: for token in extrafields:
These are the only two users of re_brackets aren't they? In which case you should drop the re.compile() too.
~Andrew
Re: [XEN PATCH v4 3/3] build: compat-xlat-header.py: optimisation to search for just '{' instead of [{}]
Quoted 26 lines · Andrew Cooper
Quoted 25 lines · Anthony PERARD
fieldsandextrafieldsalways all the parts of a sub-struct, so when there is '}', there is always a '{' before it. Also, both are lists.Signed-off-by Anthony PERARD <[email protected]>--- xen/tools/compat-xlat-header.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)xen/tools/compat-xlat-header.py +1
-1@@ -105,7 +105,7 @@ def handle_field(prefix, name, id, type, fields): else: k = id.replace('.', '_') print("%sXLAT_%s_HNDL_%s(_d_, _s_);" % (prefix, name, k), end='') - elif not re_brackets.search(' '.join(fields)): + elif not '{' in fields: tag = ' '.join(fields) tag = re.sub(r'\s*(struct|union)\s+(compat_)?(\w+)\s.*', '\\3', tag) print(" \\")@@ -290,7 +290,7 @@ def build_body(name, tokens): print(" \\\n} while (0)")
Quoted 9 lines
xen/tools/compat-xlat-header.py def build_body(name, tokens): · line 296
Quoted 6 lines
xen/tools/compat-xlat-header.py +1
-1def check_field(kind, name, field, extrafields): - if not re_brackets.search(' '.join(extrafields)): + if not '{' in extrafields: print("; \\") if len(extrafields) != 0: for token in extrafields:These are the only two users of
re_bracketsaren't they? In which case you should drop there.compile()too.
Indeed, I miss that, we can drop re_brackets.
Cheers,
Signature · 1 line
Anthony PERARD
Re: [XEN PATCH v4 3/3] build: compat-xlat-header.py: optimisation to search for just '{' instead of [{}]
Quoted 35 lines · Anthony PERARD
Quoted 26 lines · Andrew Cooper
Quoted 25 lines · Anthony PERARD
fieldsandextrafieldsalways all the parts of a sub-struct, so when there is '}', there is always a '{' before it. Also, both are lists.Signed-off-by Anthony PERARD <[email protected]>--- xen/tools/compat-xlat-header.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)xen/tools/compat-xlat-header.py +1
-1@@ -105,7 +105,7 @@ def handle_field(prefix, name, id, type, fields): else: k = id.replace('.', '_') print("%sXLAT_%s_HNDL_%s(_d_, _s_);" % (prefix, name, k), end='') - elif not re_brackets.search(' '.join(fields)): + elif not '{' in fields: tag = ' '.join(fields) tag = re.sub(r'\s*(struct|union)\s+(compat_)?(\w+)\s.*', '\\3', tag) print(" \\")@@ -290,7 +290,7 @@ def build_body(name, tokens): print(" \\\n} while (0)")Quoted 8 lines
xen/tools/compat-xlat-header.py def build_body(name, tokens): · line 296
Quoted 6 lines
xen/tools/compat-xlat-header.py +1
-1def check_field(kind, name, field, extrafields): - if not re_brackets.search(' '.join(extrafields)): + if not '{' in extrafields: print("; \\") if len(extrafields) != 0: for token in extrafields:These are the only two users of
re_bracketsaren't they? In which case you should drop there.compile()too.
Quoted 1 line
Indeed, I miss that, we can drop
re_brackets.
I've folded this deletion and queued the series for when OSSTest gets unblocked.
~Andrew