Speeding up the Makefile parsing

Until now, our UPPERCASE function was implemented purely in make for performance reasons, but our LOWERCASE function was implemented by calling "tr", which was reasonable due to the fact that LOWERCASE was rarely used.

An upcoming change is going to make LOWERCASE used quite a bit more, so we turn it into a pure make function.

Moreover, we want this LOWERCASE function to turn a "_" into a "-" and not a ".", so we slightly adjust the existing FROM and TO lists to make this possible. This doesn't change the behavior of the UPPERCASE macro because both "-" and "." are converted into "_" by this function.

Signed-off-by Thomas Petazzoni <[email protected]>

---
 package/pkg-utils.mk | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

package/pkg-utils.mk +1 -4

index 91a1981..8c3e534 100644
@@ -13,7 +13,7 @@
 # as this macro is used a lot it matters
 # This works by creating translation character pairs (E.G. a:A b:B)
 # and then looping though all of them running $(subst from,to,text)
-[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z . -
+[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z - .
 [TO]   := A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ _
 
 UPPERCASE = $(strip $(eval __tmp := $1) \

@@ -23,13 +23,12 @@ UPPERCASE = $(strip $(eval __tmp := $1) \
 	$(__tmp)))) \
 	$(__tmp))
 
-# LOWERCASE macro -- transforms its arguments to lowercase
-# The above non-tr implementation is not needed, because LOWERCASE is not
-# called very often

package/pkg-utils.mk +0 -0

 #
 # Manipulation of .config files based on the Kconfig
 

This shows a few examples of what is needed to get the host package dependencies correct.

Signed-off-by Thomas Petazzoni <[email protected]>

---
 package/expat/Config.in        | 2 ++
 package/expat/Config.in.host   | 4 ++++
 package/pkgconf/Config.in      | 2 ++
 package/pkgconf/Config.in.host | 2 ++
 package/python/Config.in       | 3 +++
 package/python/Config.in.host  | 3 +++
 6 files changed, 16 insertions(+)
 create mode 100644 package/expat/Config.in.host
 create mode 100644 package/pkgconf/Config.in.host
 create mode 100644 package/python/Config.in.host

package/expat/Config.in +0 -0

index 758fb7d..5b4bd39 100644

package/expat/Config.in +1 -0

+
 config BR2_PACKAGE_EXPAT
 	bool "expat"
 	help

package/expat/Config.in.host new file +4 -0

new file mode 100644
index 0000000..9d0b0af
@@ -0,0 +1,4 @@
+config BR2_PACKAGE_HOST_EXPAT
+	bool
+	select BR2_PACKAGE_HOST_PKGCONF
+

package/pkgconf/Config.in +2 -0

index f95847f..56c1c48 100644
@@ -1,3 +1,5 @@
+source "package/pkgconf/Config.in.host"
+
 config BR2_PACKAGE_PKGCONF
 	bool "pkgconf"
 	help

package/pkgconf/Config.in.host new file +2 -0

new file mode 100644
index 0000000..e324360
@@ -0,0 +1,2 @@
+config BR2_PACKAGE_HOST_PKGCONF
+	bool

package/python/Config.in +3 -0

index a0c78c6..253f36b 100644
@@ -1,9 +1,12 @@
+source "package/python/Config.in.host"
+
 config BR2_PACKAGE_PYTHON
 	bool "python"
 	depends on BR2_USE_WCHAR
 	# uses fork()
 	depends on BR2_USE_MMU
 	select BR2_PACKAGE_LIBFFI
+	select BR2_PACKAGE_HOST_PYTHON
 	help
 	  The python language interpreter.
 

package/python/Config.in.host new file +3 -0

new file mode 100644
index 0000000..2eca521
@@ -0,0 +1,3 @@
+config BR2_PACKAGE_HOST_PYTHON
+	bool
+	select BR2_PACKAGE_HOST_EXPAT

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Hello,
The current strategy of Buildroot with regard to .mk files is to always include all of them. However, with an ever-increasing number of packages, the number of .mk files to parse is growing significantly, and this makes the execution of any trivial make target take at least 12 seconds on a relatively fast SSD equipped laptop. For example, a simple thing like "make help" or "make external-deps" takes 12 seconds solely because of the parsing time of more than a thousand of package makefiles.

The proposed RFC solution is to look at the BR2_PACKAGE_<foo> variables, and from them derive the name of the package .mk files to be included. It is a simple change in the main Makefile, but it has two major consequences that we need to discuss on whether they are acceptable or not:

* We need to have BR2_PACKAGE_HOST_<foo> options for *all* host packages. Of course, the vast majority of them can be blind options, but it means that we have to ensure all the host packages are properly selected at the Config.in level. That's a fairly significant change.

* Since the logic assume that BR2_PACKAGE_FOO is always package/foo/foo.mk, it means we have to 1/ guarantee that the package option name matches the package file name, and 2/ guarantee that we don't use any sub-directories to store packages.

Quoted 11 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Hello,
The current strategy of Buildroot with regard to .mk files is to always include all of them. However, with an ever-increasing number of packages, the number of .mk files to parse is growing significantly, and this makes the execution of any trivial make target take at least 12 seconds on a relatively fast SSD equipped laptop. For example, a simple thing like "make help" or "make external-deps" takes 12 seconds solely because of the parsing time of more than a thousand of package makefiles.

.. and how fast is it after this series?

Quoted 11 lines

The proposed RFC solution is to look at the BR2_PACKAGE_<foo> variables, and from them derive the name of the package .mk files to be included. It is a simple change in the main Makefile, but it has two major consequences that we need to discuss on whether they are acceptable or not:

* We need to have BR2_PACKAGE_HOST_<foo> options for *all* host packages. Of course, the vast majority of them can be blind options, but it means that we have to ensure all the host packages are properly selected at the Config.in level. That's a fairly significant change.

I don't think this is too bad. Maybe we could look into generating the _DEPENDENCIES line from depends/select statements in the Config.in automatically, then we only need to explicitly handle the optional dependencies in the .mk file (or decide to also handle them in Kconfig if possible).

Quoted 4 lines

* Since the logic assume that BR2_PACKAGE_FOO is always package/foo/foo.mk, it means we have to 1/ guarantee that the package option name matches the package file name, and 2/ guarantee that we don't use any sub-directories to store packages.

The 2nd part is more serious as we use subdirs today to group related packages and have supported them "forever". We also used to tell people (before BR2_EXTERNAL) to put their own packages under package/<company>.

I guess playing tricks with $(wildcard) to find the .mk file is going to add quite some performance overhead.

Dear Peter Korsgaard,

Quoted 12 lines · Peter Korsgaard

Quoted 11 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Hello,
The current strategy of Buildroot with regard to .mk files is to always include all of them. However, with an ever-increasing number of packages, the number of .mk files to parse is growing significantly, and this makes the execution of any trivial make target take at least 12 seconds on a relatively fast SSD equipped laptop. For example, a simple thing like "make help" or "make external-deps" takes 12 seconds solely because of the parsing time of more than a thousand of package makefiles.

Quoted 1 line

.. and how fast is it after this series?

Before the series:

$ time make help
[...]
real	0m15.760s
user	0m14.818s
sys	0m0.969s

After the series:

$ time make help
[...]
real	0m1.209s
user	0m0.807s
sys	0m0.416s

Quoted 12 lines

Quoted 11 lines

The proposed RFC solution is to look at the BR2_PACKAGE_<foo> variables, and from them derive the name of the package .mk files to be included. It is a simple change in the main Makefile, but it has two major consequences that we need to discuss on whether they are acceptable or not:

* We need to have BR2_PACKAGE_HOST_<foo> options for *all* host packages. Of course, the vast majority of them can be blind options, but it means that we have to ensure all the host packages are properly selected at the Config.in level. That's a fairly significant change.

Quoted 5 lines

I don't think this is too bad. Maybe we could look into generating the _DEPENDENCIES line from depends/select statements in the Config.in automatically, then we only need to explicitly handle the optional dependencies in the .mk file (or decide to also handle them in Kconfig if possible).

Why not, but doing this would require a pre-processing step to process the Config.in file and generates the makefile dependencies from it. Also, having the mandatory dependencies handled automatically but not the optional dependencies seems a bit weird, no?

That being said, doing for host packages what we do today for target packages doesn't that horrible, especially since we don't have all the crazy dependencies on toolchain options that are complicated to handle for target packages.

Quoted 10 lines

Quoted 5 lines

* Since the logic assume that BR2_PACKAGE_FOO is always package/foo/foo.mk, it means we have to 1/ guarantee that the package option name matches the package file name, and 2/ guarantee that we don't use any sub-directories to store packages.

The 2nd part is more serious as we use subdirs today to group related packages and have supported them "forever". We also used to tell people (before BR2_EXTERNAL) to put their own packages under package/<company>.

Quoted 2 lines

I guess playing tricks with $(wildcard) to find the .mk file is going to add quite some performance overhead.

Yes. This part is more problematic, and I don't have a good solution for it.

Thomas