Skip to contentSkip to conversation

xen: rework compat headers generation

PATCH In review Anthony PERARD 3 participants 9 messages Jan 19, 2023 – Jan 30, 2023

lore
Share & tools
lore
#0 lore raw

v4 cover xen: rework compat headers generation

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.build-system-xen-include-rework-v4

v4:
- new patch removing get_typedefs() from the existing script (and thus also removing it from the new script)
- Adding some optimisation, mainly pre-compile regex It's slightly faster.

v3:
- Rewrite script into python instead of perl.
(last patch of the series)

v2:
- new patch [1/4] to fix issue with command line that can be way too long
- other small changes, and reorder patches

Hi,

This patch series is about 2 improvement. First one is to use $(if_changed, ) in "include/Makefile" to make the generation of the compat headers less verbose and to have the command line part of the decision to rebuild the headers. Second one is to replace one slow script by a much faster one, and save time when generating the headers.

Thanks.

Anthony PERARD (3):
build: include/compat, remove typedefs handling
build: replace get-fields.sh by a python script
build: compat-xlat-header.py: optimisation to search for just '{'

    instead of [{}]
 xen/include/Makefile            |   6 +-
 xen/tools/compat-xlat-header.py | 432 ++++++++++++++++++++++++++
 xen/tools/get-fields.sh         | 528 --------------------------------
 3 files changed, 434 insertions(+), 532 deletions(-)
 create mode 100644 xen/tools/compat-xlat-header.py
 delete mode 100644 xen/tools/get-fields.sh
Signature · 1 line
Anthony PERARD
#1 lore raw

PATCH v4 1/3 build: include/compat, remove typedefs handling

Partial revert of c93bd0e6ea2a ("tmem: fix 32-on-64 support") Since c492e19fdd05 ("xen: remove tmem from hypervisor"), this code isn't used anymore.

---

Notes:
v4:
- new patch

 xen/tools/get-fields.sh | 57 +----------------------------------------
 1 file changed, 1 insertion(+), 56 deletions(-)
xen/tools/get-fields.sh +1 -56
index 002db2093f..ad4a7aacc6 100644
@@ -41,34 +41,6 @@ get_fields ()
 	done
 }
 
-get_typedefs ()
-{
-	local level=1 state=
-	for token in $1
-	do
-		case "$token" in
-		typedef)
-			test $level != 1 || state=1
-			;;
-		COMPAT_HANDLE\(*\))
-			test $level != 1 -o "$state" != 1 || state=2
-			;;
-		[\{\[])
-			level=$(expr $level + 1)
-			;;
-		[\}\]])
-			level=$(expr $level - 1)
-			;;
-		";")
-			test $level != 1 || state=
-			;;
-		[a-zA-Z_]*)
-			test $level != 1 -o "$state" != 2 || echo "$token"
-			;;
-		esac
-	done
-}
-
 build_enums ()
 {
 	local level=1 kind= fields= members= named= id= token
@@ -201,21 +173,7 @@ for line in sys.stdin.readlines():
 				fi
 				;;
 			[a-zA-Z]*)
-				if [ -z "$id" -a -z "$type" -a -z "$array_type" ]
-				then
-					for id in $typedefs
-					do
-						test $id != "$token" || type=$id
-					done
-					if [ -z "$type" ]
-					then
-						id=$token
-					else
-						id=
-					fi
-				else
-					id=$token
-				fi
+				id=$token
 				;;
 			[\,\;])
 				if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
@@ -330,18 +288,6 @@ build_body ()
 			if [ -n "$array" ]
 			then
 				array="$array $token"
-			elif [ -z "$id" -a -z "$type" -a -z "$array_type" ]
-			then
-				for id in $typedefs
-				do
-					test $id != "$token" || type=$id
-				done
-				if [ -z "$type" ]
-				then
-					id=$token
-				else
-					id=
-				fi
 			else
 				id=$token
 			fi
@@ -514,7 +460,6 @@ name=${2#compat_}
 name=${name#xen}
 case "$1" in
 "!")
-	typedefs="$(get_typedefs "$list")"
 	build_enums $name "$fields"
 	build_body $name "$fields"
 	;;
Signature · 1 line
Anthony PERARD
#3 lore raw

PATCH v4 2/3 build: replace get-fields.sh by a python script

In reply to Anthony PERARD

The get-fields.sh which generate all the include/compat/.xlat/*.h headers is quite slow. It takes for example nearly 3 seconds to generate platform.h on a recent machine, or 2.3 seconds for memory.h.

Rewriting the mix of shell/sed/python into a single python script make the generation of those file a lot faster.

No functional change, the headers generated are identical.

---

Notes:
To test the header generation, I've submit a branch to gitlab ci, where all the headers where generated, and for each one both the shell script and the python script where run and the result of both compared.

v4:
- Use "token in ()" instead of "token in []"
- Pre-compile regex
- removed get_typedefs() as it's been removed from get-fields.sh

Overall, this seems to save 0.07166s, down to 1.6673s when generating all xlat headers sequentially. (before removing get_typedefs)

v3:
convert to python script instead of perl
- this should allow more developper do be able to review/edit it.
- it avoid adding a dependency on perl for the hypervisor build.

It can be twice as slow than the perl version, but overall, when doing a build with make, there isn't much difference between the perl and python script.
We might be able to speed the python script up by precompiling the many regex, but it's probably not worth it. (python already have a cache of compiled regex, but I think it's small, maybe 10 or so)

v2:
- Add .pl extension to the perl script
- remove "-w" from the shebang as it is duplicate of "use warning;"
- Add a note in the commit message that the "headers generated are identical".

 xen/include/Makefile            |   6 +-
 xen/tools/compat-xlat-header.py | 432 +++++++++++++++++++++++++++++
 xen/tools/get-fields.sh         | 473 --------------------------------
 3 files changed, 434 insertions(+), 477 deletions(-)
 create mode 100644 xen/tools/compat-xlat-header.py
 delete mode 100644 xen/tools/get-fields.sh
xen/include/Makefile +2 -4
index 65be310eca..b950423efe 100644
@@ -60,9 +60,7 @@ cmd_compat_c = \
 
 quiet_cmd_xlat_headers = GEN     $@
 cmd_xlat_headers = \
-    while read what name; do \
-        $(SHELL) $(srctree)/tools/get-fields.sh "$$what" compat_$$name $< || exit $$?; \
-    done <$(patsubst $(obj)/compat/%,$(obj)/compat/.xlat/%,$(basename $<)).lst >[email protected]; \
+    $(PYTHON) $(srctree)/tools/compat-xlat-header.py $< $(patsubst $(obj)/compat/%,$(obj)/compat/.xlat/%,$(basename $<)).lst > [email protected]; \
     mv -f [email protected] $@
 
 targets += $(headers-y)
@@ -80,7 +78,7 @@ $(obj)/compat/%.c: $(src)/public/%.h $(srcdir)/xlat.lst $(srctree)/tools/compat-
 	$(call if_changed,compat_c)
 
 targets += $(patsubst compat/%, compat/.xlat/%, $(headers-y))
-$(obj)/compat/.xlat/%.h: $(obj)/compat/%.h $(obj)/compat/.xlat/%.lst $(srctree)/tools/get-fields.sh FORCE
+$(obj)/compat/.xlat/%.h: $(obj)/compat/%.h $(obj)/compat/.xlat/%.lst $(srctree)/tools/compat-xlat-header.py FORCE
 	$(call if_changed,xlat_headers)
 
 quiet_cmd_xlat_lst = GEN     $@
xen/tools/compat-xlat-header.py new file +432 -0

432 lines not on this page · raw message

xen/tools/get-fields.sh deleted +0 -473

473 lines not on this page · raw message

Signature · 1 line
Anthony PERARD
Acked-by on v4 2/3
#4 lore raw

Re: [XEN PATCH v4 2/3] build: replace get-fields.sh by a python script

Quoted 10 lines · Anthony PERARD

The get-fields.sh which generate all the include/compat/.xlat/*.h headers is quite slow. It takes for example nearly 3 seconds to generate platform.h on a recent machine, or 2.3 seconds for memory.h.

Rewriting the mix of shell/sed/python into a single python script make the generation of those file a lot faster.

No functional change, the headers generated are identical.

#5 lore raw

PATCH v4 3/3 build: compat-xlat-header.py: optimisation to search for just '{' instead of [{}]

In reply to Anthony PERARD

fields and extrafields always all the parts of a sub-struct, so when there is '}', there is always a '{' before it. Also, both are lists.

---
 xen/tools/compat-xlat-header.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/tools/compat-xlat-header.py +2 -2
index ae5c9f11c9..d0a864b68e 100644
@@ -105,7 +105,7 @@ def handle_field(prefix, name, id, type, fields):
         else:
             k = id.replace('.', '_')
             print("%sXLAT_%s_HNDL_%s(_d_, _s_);" % (prefix, name, k), end='')
-    elif not re_brackets.search(' '.join(fields)):
+    elif not '{' in fields:
         tag = ' '.join(fields)
         tag = re.sub(r'\s*(struct|union)\s+(compat_)?(\w+)\s.*', '\\3', tag)
         print(" \\")
@@ -290,7 +290,7 @@ def build_body(name, tokens):
     print(" \\\n} while (0)")
 
 def check_field(kind, name, field, extrafields):
-    if not re_brackets.search(' '.join(extrafields)):
+    if not '{' in extrafields:
         print("; \\")
         if len(extrafields) != 0:
             for token in extrafields:
Signature · 1 line
Anthony PERARD
#6 lore raw

Re: [XEN PATCH v4 3/3] build: compat-xlat-header.py: optimisation to search for just '{' instead of [{}]

Quoted 23 lines · Anthony PERARD

fields and extrafields always all the parts of a sub-struct, so when there is '}', there is always a '{' before it. Also, both are lists.

---
 xen/tools/compat-xlat-header.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/tools/compat-xlat-header.py +1 -1
index ae5c9f11c9..d0a864b68e 100644
@@ -105,7 +105,7 @@ def handle_field(prefix, name, id, type, fields):
         else:
             k = id.replace('.', '_')
             print("%sXLAT_%s_HNDL_%s(_d_, _s_);" % (prefix, name, k), end='')
-    elif not re_brackets.search(' '.join(fields)):
+    elif not '{' in fields:
         tag = ' '.join(fields)
         tag = re.sub(r'\s*(struct|union)\s+(compat_)?(\w+)\s.*', '\\3', tag)
         print(" \\")

xen/tools/compat-xlat-header.py def build_body(name, tokens): · line 296

Quoted 8 lines
xen/tools/compat-xlat-header.py +1 -1
@@ -290,7 +290,7 @@ def build_body(name, tokens):
     print(" \\\n} while (0)")
 
 def check_field(kind, name, field, extrafields):
-    if not re_brackets.search(' '.join(extrafields)):
+    if not '{' in extrafields:
         print("; \\")
         if len(extrafields) != 0:
             for token in extrafields:

These are the only two users of re_brackets aren't they?  In which case you should drop the re.compile() too.

~Andrew

#7 lore raw

Re: [XEN PATCH v4 3/3] build: compat-xlat-header.py: optimisation to search for just '{' instead of [{}]

Quoted 26 lines · Andrew Cooper
Quoted 25 lines · Anthony PERARD

fields and extrafields always all the parts of a sub-struct, so when there is '}', there is always a '{' before it. Also, both are lists.

---
 xen/tools/compat-xlat-header.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/tools/compat-xlat-header.py +1 -1
index ae5c9f11c9..d0a864b68e 100644
@@ -105,7 +105,7 @@ def handle_field(prefix, name, id, type, fields):
         else:
             k = id.replace('.', '_')
             print("%sXLAT_%s_HNDL_%s(_d_, _s_);" % (prefix, name, k), end='')
-    elif not re_brackets.search(' '.join(fields)):
+    elif not '{' in fields:
         tag = ' '.join(fields)
         tag = re.sub(r'\s*(struct|union)\s+(compat_)?(\w+)\s.*', '\\3', tag)
         print(" \\")
@@ -290,7 +290,7 @@ def build_body(name, tokens):
     print(" \\\n} while (0)")
 
Quoted 9 lines

xen/tools/compat-xlat-header.py def build_body(name, tokens): · line 296

Quoted 6 lines
xen/tools/compat-xlat-header.py +1 -1
 def check_field(kind, name, field, extrafields):
-    if not re_brackets.search(' '.join(extrafields)):
+    if not '{' in extrafields:
         print("; \\")
         if len(extrafields) != 0:
             for token in extrafields:

These are the only two users of re_brackets aren't they?  In which case you should drop the re.compile() too.

Indeed, I miss that, we can drop re_brackets.

Cheers,

Signature · 1 line
Anthony PERARD
#8 lore raw

Re: [XEN PATCH v4 3/3] build: compat-xlat-header.py: optimisation to search for just '{' instead of [{}]

Quoted 35 lines · Anthony PERARD
Quoted 26 lines · Andrew Cooper
Quoted 25 lines · Anthony PERARD

fields and extrafields always all the parts of a sub-struct, so when there is '}', there is always a '{' before it. Also, both are lists.

---
 xen/tools/compat-xlat-header.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/tools/compat-xlat-header.py +1 -1
index ae5c9f11c9..d0a864b68e 100644
@@ -105,7 +105,7 @@ def handle_field(prefix, name, id, type, fields):
         else:
             k = id.replace('.', '_')
             print("%sXLAT_%s_HNDL_%s(_d_, _s_);" % (prefix, name, k), end='')
-    elif not re_brackets.search(' '.join(fields)):
+    elif not '{' in fields:
         tag = ' '.join(fields)
         tag = re.sub(r'\s*(struct|union)\s+(compat_)?(\w+)\s.*', '\\3', tag)
         print(" \\")
@@ -290,7 +290,7 @@ def build_body(name, tokens):
     print(" \\\n} while (0)")
 
Quoted 8 lines

xen/tools/compat-xlat-header.py def build_body(name, tokens): · line 296

Quoted 6 lines
xen/tools/compat-xlat-header.py +1 -1
 def check_field(kind, name, field, extrafields):
-    if not re_brackets.search(' '.join(extrafields)):
+    if not '{' in extrafields:
         print("; \\")
         if len(extrafields) != 0:
             for token in extrafields:

These are the only two users of re_brackets aren't they?  In which case you should drop the re.compile() too.

Quoted 1 line

Indeed, I miss that, we can drop re_brackets.

I've folded this deletion and queued the series for when OSSTest gets unblocked.

~Andrew