Skip to contentSkip to conversation

xen: rework compat headers generation

PATCH Superseded Anthony PERARD 3 participants 16 messages Jun 14, 2022 – Aug 3, 2022

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover xen: rework compat headers generation

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.build-system-xen-include-rework-v2

v2:
- new patch [1/4] to fix issue with command line that can be way too long
- other small changes, and reorder patches

Hi,

This patch series is about 2 improvement. First one is to use $(if_changed, ) in "include/Makefile" to make the generation of the compat headers less verbose and to have the command line part of the decision to rebuild the headers. Second one is to replace one slow script by a much faster one, and save time when generating the headers.

There's some number here:
https://lore.kernel.org/xen-devel/Yp3%2F%2Fc%[email protected]/

On my machine when doing a full build [of the hypervisor], with ccache enabled, it saves about 1.17 seconds (out of ~17s), and without ccache, it saves about 2.0 seconds (out of ~37s).

Thanks.

Anthony PERARD (4):
build,include: rework shell script for headers++.chk
build: remove auto.conf prerequisite from compat/xlat.h target
build: set PERL
build: replace get-fields.sh by a perl script

 xen/Makefile                    |   1 +
 xen/include/Makefile            |  25 +-
 README                          |   1 +
 xen/tools/compat-xlat-header.pl | 539 ++++++++++++++++++++++++++++++++
 xen/tools/get-fields.sh         | 528 -------------------------------
 5 files changed, 557 insertions(+), 537 deletions(-)
 create mode 100755 xen/tools/compat-xlat-header.pl
 delete mode 100644 xen/tools/get-fields.sh
Signature · 1 line
Anthony PERARD
#1 lore raw

PATCH v2 1/4 build,include: rework shell script for headers++.chk

The command line generated for headers++.chk by make is quite long, and in some environment it is too long. This issue have been seen in Yocto build environment.

Error messages:
make[9]: execvp: /bin/sh: Argument list too long make[9]: *** [include/Makefile:181: include/headers++.chk] Error 127

Rework so that we do the foreach loop in shell rather that make to reduce the command line size by a lot. We also need a way to get headers prerequisite for some public headers so we use a switch "case" in shell to be able to do some simple pattern matching. Variables alone in POSIX shell don't allow to work with associative array or variables with "/".

Fixes 28e13c7f43 ("build: xen/include: use if_changed")

---

Notes:
v2:
- fix typo in commit message
- fix out-of-tree build

v1:
- was sent as a reply to v1 of the series

 xen/include/Makefile | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
xen/include/Makefile +13 -4
index 6d9bcc19b0..49c75a78f9 100644
@@ -158,13 +158,22 @@ define cmd_headerscxx_chk
 	    touch [email protected];                                                     \
 	    exit 0;                                                           \
 	fi;                                                                   \
-	$(foreach i, $(filter %.h,$^),                                        \
-	    echo "#include "\"$(i)\"                                          \
+	get_prereq() {                                                        \
+	    case $$1 in                                                       \
+	    $(foreach i, $(filter %.h,$^),                                    \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+		$(i)$(close)                                                  \
+		echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+			-include c$(j))";;))                                  \
+	    esac;                                                             \
+	};                                                                    \
+	for i in $(filter %.h,$^); do                                         \
+	    echo "#include "\"$$i\"                                           \
 	    | $(CXX) -x c++ -std=gnu++98 -Wall -Werror -D__XEN_TOOLS__        \
 	      -include stdint.h -include $(srcdir)/public/xen.h               \
-	      $(foreach j, $($(patsubst $(srctree)/%,%,$i)-prereq), -include c$(j)) \
+	      `get_prereq $$i`                                                \
 	      -S -o /dev/null -                                               \
-	    || exit $$?; echo $(i) >> [email protected];) \
+	    || exit $$?; echo $$i >> [email protected]; done;                            \
 	mv [email protected] $@
 endef
 
Signature · 1 line
Anthony PERARD
#2 lore raw

Re: [XEN PATCH v2 1/4] build,include: rework shell script for headers++.chk

Quoted 40 lines · Anthony PERARD

The command line generated for headers++.chk by make is quite long, and in some environment it is too long. This issue have been seen in Yocto build environment.

Error messages:
make[9]: execvp: /bin/sh: Argument list too long make[9]: *** [include/Makefile:181: include/headers++.chk] Error 127

Rework so that we do the foreach loop in shell rather that make to reduce the command line size by a lot. We also need a way to get headers prerequisite for some public headers so we use a switch "case" in shell to be able to do some simple pattern matching. Variables alone in POSIX shell don't allow to work with associative array or variables with "/".

Fixes 28e13c7f43 ("build: xen/include: use if_changed")

---

Notes:
v2:
- fix typo in commit message
- fix out-of-tree build

v1:
- was sent as a reply to v1 of the series

 xen/include/Makefile | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
xen/include/Makefile +0 -0
index 6d9bcc19b0..49c75a78f9 100644
@@ -158,13 +158,22 @@ define cmd_headerscxx_chk
 	    touch [email protected];                                                     \
 	    exit 0;                                                           \
 	fi;                                                                   \

xen/include/Makefile define cmd_headerscxx_chk · line 168

Quoted 10 lines
xen/include/Makefile +8 -2
-	$(foreach i, $(filter %.h,$^),                                        \
-	    echo "#include "\"$(i)\"                                          \
+	get_prereq() {                                                        \
+	    case $$1 in                                                       \
+	    $(foreach i, $(filter %.h,$^),                                    \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+		$(i)$(close)                                                  \
+		echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+			-include c$(j))";;))                                  \
+	    esac;                                                             \

If I'm reading this right (indentation looks to be a little misleading and hence one needs to count parentheses) the case statement could (in theory) remain without any "body". As per the command language spec I'm looking at this (if it works in the first place) is an extension, and formally there's always at least one label required. Since we aim to be portable in such regards, I'd like to request that there be a final otherwise empty *) line.

xen/include/Makefile define cmd_headerscxx_chk · line 174

Quoted 7 lines
xen/include/Makefile +4 -1
+	};                                                                    \
+	for i in $(filter %.h,$^); do                                         \
+	    echo "#include "\"$$i\"                                           \
 	    | $(CXX) -x c++ -std=gnu++98 -Wall -Werror -D__XEN_TOOLS__        \
 	      -include stdint.h -include $(srcdir)/public/xen.h               \
-	      $(foreach j, $($(patsubst $(srctree)/%,%,$i)-prereq), -include c$(j)) \
+	      `get_prereq $$i`                                                \

While I know we use back-ticked quoting elsewhere, I'd generally recommend to use $() for readability. But maybe others view this exactly the other way around ...

And a question without good context to put at: Isn't headers99.chk in similar need of converting? It looks only slightly less involved than the C++ one.

Jan

Quoted 6 lines
xen/include/Makefile +1 -1
 	      -S -o /dev/null -                                               \
-	    || exit $$?; echo $(i) >> [email protected];) \
+	    || exit $$?; echo $$i >> [email protected]; done;                            \
 	mv [email protected] $@
 endef
 
#3 lore raw

Re: [XEN PATCH v2 1/4] build,include: rework shell script for headers++.chk

Quoted 20 lines · Jan Beulich
Quoted 8 lines · Anthony PERARD
xen/include/Makefile +0 -0
index 6d9bcc19b0..49c75a78f9 100644
@@ -158,13 +158,22 @@ define cmd_headerscxx_chk
 	    touch [email protected];                                                     \
 	    exit 0;                                                           \
 	fi;                                                                   \

xen/include/Makefile define cmd_headerscxx_chk · line 168

Quoted 10 lines
xen/include/Makefile +8 -2
-	$(foreach i, $(filter %.h,$^),                                        \
-	    echo "#include "\"$(i)\"                                          \
+	get_prereq() {                                                        \
+	    case $$1 in                                                       \
+	    $(foreach i, $(filter %.h,$^),                                    \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+		$(i)$(close)                                                  \
+		echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+			-include c$(j))";;))                                  \
+	    esac;                                                             \
Quoted 7 lines

If I'm reading this right (indentation looks to be a little misleading and hence one needs to count parentheses) the case statement could (in theory) remain without any "body". As per the command language spec I'm looking at this (if it works in the first place) is an extension, and formally there's always at least one label required. Since we aim to be portable in such regards, I'd like to request that there be a final otherwise empty *) line.

When looking at the shell grammar at [1], an empty body seems to be allowed. But I can add "*)" at the end for peace of mind.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_10_02

As for misleading indentation, I've got my $EDITOR to show me matching parentheses, so I don't need to count them. But if I rewrite that function as following, would it be easier to follow?

+	get_prereq() {                                                        \
+	case $$1 in                                                           \
+	$(foreach i, $(filter %.h,$^),                                        \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+		$(i)$(close)                                                  \
+		echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+			-include c$(j))";;))                                  \
+	*) ;;                                                                 \
+	esac;                                                                 \
+	};                                                                    \
Quoted 11 lines

xen/include/Makefile define cmd_headerscxx_chk · line 174

Quoted 7 lines
xen/include/Makefile +4 -1
+	};                                                                    \
+	for i in $(filter %.h,$^); do                                         \
+	    echo "#include "\"$$i\"                                           \
 	    | $(CXX) -x c++ -std=gnu++98 -Wall -Werror -D__XEN_TOOLS__        \
 	      -include stdint.h -include $(srcdir)/public/xen.h               \
-	      $(foreach j, $($(patsubst $(srctree)/%,%,$i)-prereq), -include c$(j)) \
+	      `get_prereq $$i`                                                \

While I know we use back-ticked quoting elsewhere, I'd generally recommend to use $() for readability. But maybe others view this exactly the other way around ...

Well, in Makefile it's `` vs $$(). At least, we don't have to write $$$(open)$(close) here :-).

I guess $$(get_prereq $$i) isn't too bad here, I'll replace the backquote.

Quoted 3 lines

And a question without good context to put at: Isn't headers99.chk in similar need of converting? It looks only slightly less involved than the C++ one.

It doesn't really need converting at the moment, because there's only two headers to check, so the resulting command line is small. But converting it at the same time is probably a good idea to avoid having two different implementations of the header check.

Thanks,

Signature · 1 line
Anthony PERARD
#4 lore raw

Re: [XEN PATCH v2 1/4] build,include: rework shell script for headers++.chk

Quoted 29 lines · Anthony PERARD
Quoted 20 lines · Jan Beulich
Quoted 8 lines · Anthony PERARD
xen/include/Makefile +0 -0
index 6d9bcc19b0..49c75a78f9 100644
@@ -158,13 +158,22 @@ define cmd_headerscxx_chk
 	    touch [email protected];                                                     \
 	    exit 0;                                                           \
 	fi;                                                                   \

xen/include/Makefile define cmd_headerscxx_chk · line 168

Quoted 10 lines
xen/include/Makefile +8 -2
-	$(foreach i, $(filter %.h,$^),                                        \
-	    echo "#include "\"$(i)\"                                          \
+	get_prereq() {                                                        \
+	    case $$1 in                                                       \
+	    $(foreach i, $(filter %.h,$^),                                    \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+		$(i)$(close)                                                  \
+		echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+			-include c$(j))";;))                                  \
+	    esac;                                                             \
Quoted 7 lines

If I'm reading this right (indentation looks to be a little misleading and hence one needs to count parentheses) the case statement could (in theory) remain without any "body". As per the command language spec I'm looking at this (if it works in the first place) is an extension, and formally there's always at least one label required. Since we aim to be portable in such regards, I'd like to request that there be a final otherwise empty *) line.

Quoted 4 lines

When looking at the shell grammar at [1], an empty body seems to be allowed. But I can add "*)" at the end for peace of mind.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_10_02

Hmm, indeed. As opposed to

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04

Quoted 4 lines

As for misleading indentation, I've got my $EDITOR to show me matching parentheses, so I don't need to count them. But if I rewrite that function as following, would it be easier to follow?

xen/include/Makefile define cmd_headerscxx_chk · line 169

Quoted 10 lines
xen/include/Makefile +10 -0
+	get_prereq() {                                                        \
+	case $$1 in                                                           \
+	$(foreach i, $(filter %.h,$^),                                        \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+		$(i)$(close)                                                  \
+		echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+			-include c$(j))";;))                                  \
+	*) ;;                                                                 \
+	esac;                                                                 \
+	};                                                                    \

Hmm, not really, no (and it may be more obvious looking at the reply context above): My primary concern is the use of hard tabs beyond the leading one (which is uniform across all lines and hence doesn't alter apparent alignment even with the + or > prefixed).

Jan

#5 lore raw

PATCH v2 1/4 build,include: rework shell script for headers++.chk

In reply to Anthony PERARD

The command line generated for headers++.chk by make is quite long, and in some environment it is too long. This issue have been seen in Yocto build environment.

Error messages:
make[9]: execvp: /bin/sh: Argument list too long make[9]: *** [include/Makefile:181: include/headers++.chk] Error 127

Rework so that we do the foreach loop in shell rather that make to reduce the command line size by a lot. We also need a way to get headers prerequisite for some public headers so we use a switch "case" in shell to be able to do some simple pattern matching. Variables alone in POSIX shell don't allow to work with associative array or variables with "/".

Also rework headers99.chk as it has a similar implementation, even if with only two headers to check the command line isn't too long at the moment.

Fixes 28e13c7f43 ("build: xen/include: use if_changed")

---

Notes:
v3:
- add one more pattern to avoid a possible empty body for "case"
- use $() instead of `` to execute get_prereq()
- also convert headers99_chk
- convert some 'tab' to 'space', have only 1 tab at start of line

v2:
- fix typo in commit message
- fix out-of-tree build

v1:
- was sent as a reply to v1 of the series

 xen/include/Makefile | 37 +++++++++++++++++++++++++++++--------
 1 file changed, 29 insertions(+), 8 deletions(-)
xen/include/Makefile +29 -8
index 617599df7e..510f65c92a 100644
@@ -141,13 +141,24 @@ cmd_header_chk = \
 quiet_cmd_headers99_chk = CHK     $@
 define cmd_headers99_chk
 	rm -f [email protected]; \
-	$(foreach i, $(filter %.h,$^),                                        \
-	    echo "#include "\"$(i)\"                                          \
+	get_prereq() {                                                        \
+	    case $$1 in                                                       \
+	    $(foreach i, $(filter %.h,$^),                                    \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+	        $(i)$(close)                                                  \
+	        echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+	                -include $(j).h)";;))                                 \
+	    *) ;;                                                             \
+	    esac;                                                             \
+	};                                                                    \
+	for i in $(filter %.h,$^); do                                         \
+	    echo "#include "\"$$i\"                                           \
 	    | $(CC) -x c -std=c99 -Wall -Werror                               \
 	      -include stdint.h                                               \
-	      $(foreach j, $($(patsubst $(srctree)/%,%,$i)-prereq), -include $(j).h) \
+	      $$(get_prereq $$i)                                              \
 	      -S -o /dev/null -                                               \
-	    || exit $$?; echo $(i) >> [email protected];) \
+	    || exit $$?; echo $$i >> [email protected];                                  \
+	done;                                                                 \
 	mv [email protected] $@
 endef
 
@@ -158,13 +169,23 @@ define cmd_headerscxx_chk
 	    touch [email protected];                                                     \
 	    exit 0;                                                           \
 	fi;                                                                   \
-	$(foreach i, $(filter %.h,$^),                                        \
-	    echo "#include "\"$(i)\"                                          \
+	get_prereq() {                                                        \
+	    case $$1 in                                                       \
+	    $(foreach i, $(filter %.h,$^),                                    \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+	        $(i)$(close)                                                  \
+	        echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+	                -include c$(j))";;))                                  \
+	    *) ;;                                                             \
+	    esac;                                                             \
+	};                                                                    \
+	for i in $(filter %.h,$^); do                                         \
+	    echo "#include "\"$$i\"                                           \
 	    | $(CXX) -x c++ -std=gnu++98 -Wall -Werror -D__XEN_TOOLS__        \
 	      -include stdint.h -include $(srcdir)/public/xen.h               \
-	      $(foreach j, $($(patsubst $(srctree)/%,%,$i)-prereq), -include c$(j)) \
+	      $$(get_prereq $$i)                                              \
 	      -S -o /dev/null -                                               \
-	    || exit $$?; echo $(i) >> [email protected];) \
+	    || exit $$?; echo $$i >> [email protected]; done;                            \
 	mv [email protected] $@
 endef
 
Signature · 1 line
Anthony PERARD
Reviewed-by Tested-by on v2 1/4
#6 lore raw

Re: [XEN PATCH v2.1 1/4] build,include: rework shell script for headers++.chk

Hi Anthony,

Quoted 15 lines · Anthony PERARD

The command line generated for headers++.chk by make is quite long, and in some environment it is too long. This issue have been seen in Yocto build environment.

Error messages:
make[9]: execvp: /bin/sh: Argument list too long make[9]: *** [include/Makefile:181: include/headers++.chk] Error 127

Rework so that we do the foreach loop in shell rather that make to reduce the command line size by a lot. We also need a way to get headers prerequisite for some public headers so we use a switch "case" in shell to be able to do some simple pattern matching. Variables alone in POSIX shell don't allow to work with associative array or variables with "/".

Quoted 8 lines

Also rework headers99.chk as it has a similar implementation, even if with only two headers to check the command line isn't too long at the moment.

Fixes 28e13c7f43 ("build: xen/include: use if_changed")

Cheers,
Michal

Acked-by
#7 lore raw

Re: [XEN PATCH v2.1 1/4] build,include: rework shell script for headers++.chk

Quoted 22 lines · Michal Orzel

Hi Anthony,

Quoted 19 lines · Anthony PERARD

The command line generated for headers++.chk by make is quite long, and in some environment it is too long. This issue have been seen in Yocto build environment.

Error messages:
make[9]: execvp: /bin/sh: Argument list too long make[9]: *** [include/Makefile:181: include/headers++.chk] Error 127

Rework so that we do the foreach loop in shell rather that make to reduce the command line size by a lot. We also need a way to get headers prerequisite for some public headers so we use a switch "case" in shell to be able to do some simple pattern matching. Variables alone in POSIX shell don't allow to work with associative array or variables with "/".

Also rework headers99.chk as it has a similar implementation, even if with only two headers to check the command line isn't too long at the moment.

Quoted 6 lines
Quoted 4 lines
Fixes 28e13c7f43 ("build: xen/include: use if_changed")
Quoted 4 lines

Cheers,
Michal

#8 lore raw

Re: [XEN PATCH v2.1 1/4] build,include: rework shell script for headers++.chk

In reply to Anthony PERARD
Quoted 15 lines · Anthony PERARD

The command line generated for headers++.chk by make is quite long, and in some environment it is too long. This issue have been seen in Yocto build environment.

Error messages:
make[9]: execvp: /bin/sh: Argument list too long make[9]: *** [include/Makefile:181: include/headers++.chk] Error 127

Rework so that we do the foreach loop in shell rather that make to reduce the command line size by a lot. We also need a way to get headers prerequisite for some public headers so we use a switch "case" in shell to be able to do some simple pattern matching. Variables alone in POSIX shell don't allow to work with associative array or variables with "/".

Quoted 8 lines

Also rework headers99.chk as it has a similar implementation, even if with only two headers to check the command line isn't too long at the moment.

Fixes 28e13c7f43 ("build: xen/include: use if_changed")

I have committed this, but strictly speaking imo the R-b should have been dropped because ...

Quoted 8 lines

---

Notes:
v3:
- add one more pattern to avoid a possible empty body for "case"
- use $() instead of `` to execute get_prereq()
- also convert headers99_chk
- convert some 'tab' to 'space', have only 1 tab at start of line

... at least the added headers99_chk conversion was not a purely mechanical change.

Jan

Quoted 71 lines

v2:
- fix typo in commit message
- fix out-of-tree build

v1:
- was sent as a reply to v1 of the series

 xen/include/Makefile | 37 +++++++++++++++++++++++++++++--------
 1 file changed, 29 insertions(+), 8 deletions(-)
xen/include/Makefile +29 -8
index 617599df7e..510f65c92a 100644
@@ -141,13 +141,24 @@ cmd_header_chk = \
 quiet_cmd_headers99_chk = CHK     $@
 define cmd_headers99_chk
 	rm -f [email protected]; \
-	$(foreach i, $(filter %.h,$^),                                        \
-	    echo "#include "\"$(i)\"                                          \
+	get_prereq() {                                                        \
+	    case $$1 in                                                       \
+	    $(foreach i, $(filter %.h,$^),                                    \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+	        $(i)$(close)                                                  \
+	        echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+	                -include $(j).h)";;))                                 \
+	    *) ;;                                                             \
+	    esac;                                                             \
+	};                                                                    \
+	for i in $(filter %.h,$^); do                                         \
+	    echo "#include "\"$$i\"                                           \
 	    | $(CC) -x c -std=c99 -Wall -Werror                               \
 	      -include stdint.h                                               \
-	      $(foreach j, $($(patsubst $(srctree)/%,%,$i)-prereq), -include $(j).h) \
+	      $$(get_prereq $$i)                                              \
 	      -S -o /dev/null -                                               \
-	    || exit $$?; echo $(i) >> [email protected];) \
+	    || exit $$?; echo $$i >> [email protected];                                  \
+	done;                                                                 \
 	mv [email protected] $@
 endef
 
@@ -158,13 +169,23 @@ define cmd_headerscxx_chk
 	    touch [email protected];                                                     \
 	    exit 0;                                                           \
 	fi;                                                                   \
-	$(foreach i, $(filter %.h,$^),                                        \
-	    echo "#include "\"$(i)\"                                          \
+	get_prereq() {                                                        \
+	    case $$1 in                                                       \
+	    $(foreach i, $(filter %.h,$^),                                    \
+	    $(if $($(patsubst $(srctree)/%,%,$(i))-prereq),                   \
+	        $(i)$(close)                                                  \
+	        echo "$(foreach j, $($(patsubst $(srctree)/%,%,$(i))-prereq), \
+	                -include c$(j))";;))                                  \
+	    *) ;;                                                             \
+	    esac;                                                             \
+	};                                                                    \
+	for i in $(filter %.h,$^); do                                         \
+	    echo "#include "\"$$i\"                                           \
 	    | $(CXX) -x c++ -std=gnu++98 -Wall -Werror -D__XEN_TOOLS__        \
 	      -include stdint.h -include $(srcdir)/public/xen.h               \
-	      $(foreach j, $($(patsubst $(srctree)/%,%,$i)-prereq), -include c$(j)) \
+	      $$(get_prereq $$i)                                              \
 	      -S -o /dev/null -                                               \
-	    || exit $$?; echo $(i) >> [email protected];) \
+	    || exit $$?; echo $$i >> [email protected]; done;                            \
 	mv [email protected] $@
 endef
 
#9 lore raw

PATCH v2 2/4 build: remove auto.conf prerequisite from compat/xlat.h target

In reply to Anthony PERARD

Now that the command line generating "xlat.h" is check on rebuild, the header will be regenerated whenever the list of xlat headers changes due to change in ".config". We don't need to force a regeneration for every changes in ".config".

---
 xen/include/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/include/Makefile +1 -1
index 49c75a78f9..0d3e3d66e0 100644
@@ -101,7 +101,7 @@ cmd_xlat_h = \
 	cat $(filter %.h,$^) >[email protected]; \
 	mv -f [email protected] $@
 
-$(obj)/compat/xlat.h: $(addprefix $(obj)/compat/.xlat/,$(xlat-y)) $(obj)/config/auto.conf FORCE
+$(obj)/compat/xlat.h: $(addprefix $(obj)/compat/.xlat/,$(xlat-y)) FORCE
 	$(call if_changed,xlat_h)
 
 ifeq ($(XEN_TARGET_ARCH),$(XEN_COMPILE_ARCH))
Signature · 1 line
Anthony PERARD
#10 lore raw

PATCH v2 3/4 build: set PERL

In reply to Anthony PERARD

We are going to use it in a moment.

Also update README about Perl requirement.

---

Notes:
v2:
- update ./README

 xen/Makefile | 1 +
 README       | 1 +
 2 files changed, 2 insertions(+)
xen/Makefile +1 -0
index 82f5310b12..a6650a2acc 100644
@@ -22,6 +22,7 @@ PYTHON_INTERPRETER	:= $(word 1,$(shell which python3 python python2 2>/dev/null)
 export PYTHON		?= $(PYTHON_INTERPRETER)
 
 export CHECKPOLICY	?= checkpolicy
+export PERL		?= perl
 
 $(if $(filter __%, $(MAKECMDGOALS)), \
     $(error targets prefixed with '__' are only for internal use))
README +1 -0
index 5e55047ffd..c1c18de7e0 100644
@@ -64,6 +64,7 @@ provided by your OS distributor:
     * iproute package (/sbin/ip)
     * GNU bison and GNU flex
     * ACPI ASL compiler (iasl)
+    * Perl
 
 In addition to the above there are a number of optional build
 prerequisites. Omitting these will cause the related features to be
Signature · 1 line
Anthony PERARD
Acked-by on v2 3/4
#11 lore raw

Re: [XEN PATCH v2 3/4] build: set PERL

xen/Makefile PYTHON_INTERPRETER := $(word 1,$(shell which python3 python python2 2>/dev/null) · line 28

Quoted 10 lines · Anthony PERARD
xen/Makefile +1 -0
@@ -22,6 +22,7 @@ PYTHON_INTERPRETER	:= $(word 1,$(shell which python3 python python2 2>/dev/null)
 export PYTHON		?= $(PYTHON_INTERPRETER)
 
 export CHECKPOLICY	?= checkpolicy
+export PERL		?= perl
 
 $(if $(filter __%, $(MAKECMDGOALS)), \
     $(error targets prefixed with '__' are only for internal use))

Considering my patch yesterday that moved the exporting of CC etc, I wonder if - at the very least for consistency - this and the neighbouring pre-existing exports shouldn't all be moved below the inclusion of ./Config.mk as well. After all ./config might override any of those. (Yes, the ones here don't prevent such overriding, but only as long as there aren't any further make quirks.)

Since this may want doing in a separate patch, this one is

Jan

#12 lore raw

Re: [XEN PATCH v2 3/4] build: set PERL

Quoted 9 lines · Jan Beulich
Quoted 8 lines · Anthony PERARD
xen/Makefile +1 -0
@@ -22,6 +22,7 @@ PYTHON_INTERPRETER	:= $(word 1,$(shell which python3 python python2 2>/dev/null)
 export PYTHON		?= $(PYTHON_INTERPRETER)
 
 export CHECKPOLICY	?= checkpolicy
+export PERL		?= perl
 
Quoted 9 lines

xen/Makefile PYTHON_INTERPRETER := $(word 1,$(shell which python3 python python2 2>/dev/null) · line 28

Quoted 2 lines
xen/Makefile +0 -0
 $(if $(filter __%, $(MAKECMDGOALS)), \
     $(error targets prefixed with '__' are only for internal use))

Considering my patch yesterday that moved the exporting of CC etc, I wonder if - at the very least for consistency - this and the neighbouring pre-existing exports shouldn't all be moved below the inclusion of ./Config.mk as well. After all ./config might override any of those. (Yes, the ones here don't prevent such overriding, but only as long as there aren't any further make quirks.)

Sounds good, we can move CHECKPOLICY and PERL, but there's a problem with PYTHON. Config.mk define a different value for PYTHON, so the one in xen/Makefile needs to be set before including ./Config.mk.

Thanks,

Signature · 1 line
Anthony PERARD
#13 lore raw

PATCH v2 4/4 build: replace get-fields.sh by a perl script

In reply to Anthony PERARD

The get-fields.sh which generate all the include/compat/.xlat/*.h headers is quite slow. It takes for example nearly 3 seconds to generate platform.h on a recent machine, or 2.3 seconds for memory.h.

Since it's only text processing, rewriting the mix of shell/sed/python into a single perl script make the generation of those file a lot faster.

I tried to keep a similar look for the code, to keep the code similar between the shell and perl, and to ease review. So some code in perl might look weird or could be written better.

No functional change, the headers generated are identical.

---

Notes:
v2:
- Add .pl extension to the perl script
- remove "-w" from the shebang as it is duplicate of "use warning;"
- Add a note in the commit message that the "headers generated are identical".

 xen/include/Makefile            |   6 +-
 xen/tools/compat-xlat-header.pl | 539 ++++++++++++++++++++++++++++++++
 xen/tools/get-fields.sh         | 528 -------------------------------
 3 files changed, 541 insertions(+), 532 deletions(-)
 create mode 100755 xen/tools/compat-xlat-header.pl
 delete mode 100644 xen/tools/get-fields.sh
xen/include/Makefile +2 -4
index 0d3e3d66e0..31b0be14bc 100644
@@ -60,9 +60,7 @@ cmd_compat_c = \
 
 quiet_cmd_xlat_headers = GEN     $@
 cmd_xlat_headers = \
-    while read what name; do \
-        $(SHELL) $(srctree)/tools/get-fields.sh "$$what" compat_$$name $< || exit $$?; \
-    done <$(patsubst $(obj)/compat/%,$(obj)/compat/.xlat/%,$(basename $<)).lst >[email protected]; \
+    $(PERL) $(srctree)/tools/compat-xlat-header.pl $< $(patsubst $(obj)/compat/%,$(obj)/compat/.xlat/%,$(basename $<)).lst > [email protected]; \
     mv -f [email protected] $@
 
 targets += $(headers-y)
@@ -80,7 +78,7 @@ $(obj)/compat/%.c: $(src)/public/%.h $(srcdir)/xlat.lst $(srctree)/tools/compat-
 	$(call if_changed,compat_c)
 
 targets += $(patsubst compat/%, compat/.xlat/%, $(headers-y))
-$(obj)/compat/.xlat/%.h: $(obj)/compat/%.h $(obj)/compat/.xlat/%.lst $(srctree)/tools/get-fields.sh FORCE
+$(obj)/compat/.xlat/%.h: $(obj)/compat/%.h $(obj)/compat/.xlat/%.lst $(srctree)/tools/compat-xlat-header.pl FORCE
 	$(call if_changed,xlat_headers)
 
 quiet_cmd_xlat_lst = GEN     $@
xen/tools/compat-xlat-header.pl new file +539 -0

539 lines not on this page · raw message

xen/tools/get-fields.sh deleted +0 -528

528 lines not on this page · raw message

Signature · 1 line
Anthony PERARD
#14 lore raw

Re: [XEN PATCH v2 4/4] build: replace get-fields.sh by a perl script

Quoted 8 lines · Anthony PERARD

The get-fields.sh which generate all the include/compat/.xlat/*.h headers is quite slow. It takes for example nearly 3 seconds to generate platform.h on a recent machine, or 2.3 seconds for memory.h.

Since it's only text processing, rewriting the mix of shell/sed/python into a single perl script make the generation of those file a lot faster.

Quoted 7 lines

I tried to keep a similar look for the code, to keep the code similar between the shell and perl, and to ease review. So some code in perl might look weird or could be written better.

No functional change, the headers generated are identical.

Hi,

This patch have been on the mailing list for a little while now, could it be acked then committed along with the previous one now?

Thanks,

Signature · 1 line
Anthony PERARD
#15 lore raw

Re: [XEN PATCH v2 4/4] build: replace get-fields.sh by a perl script

Quoted 13 lines · Anthony PERARD
Quoted 12 lines · Anthony PERARD

The get-fields.sh which generate all the include/compat/.xlat/*.h headers is quite slow. It takes for example nearly 3 seconds to generate platform.h on a recent machine, or 2.3 seconds for memory.h.

Since it's only text processing, rewriting the mix of shell/sed/python into a single perl script make the generation of those file a lot faster.

I tried to keep a similar look for the code, to keep the code similar between the shell and perl, and to ease review. So some code in perl might look weird or could be written better.

Quoted 6 lines
Quoted 3 lines

No functional change, the headers generated are identical.

This patch have been on the mailing list for a little while now, could it be acked then committed along with the previous one now?

Well, I would certainly be happy to ack it, but only on the basis of a suitable R-b by somebody else. My Perl is close to non-existing, so I can't sensibly review this myself.

Jan