Skip to contentSkip to conversation

package/linux: fix LINUX_PKGDIR with br2-external linux-extensions

PATCH New Yann E. MORIN 3 participants 4 messages Jan 24, 2020 – Mar 7, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/linux: fix LINUX_PKGDIR with br2-external linux-extensions

When 'make' includes a new Makefile, it appends its path to the MAKEFILE_LIST variable. From that variable, we construct a few set of derivative variables:

    pkgdir = $(dir $(lastword $(MAKEFILE_LIST)))
    pkgname = $(lastword $(subst /, ,$(pkgdir)))

Essentially, pkgdir is the full directory where the package is located (either relative to Buildroot's top directory for in-tree packages, or absolute for packages in br2-external trees), while pkgname is the last component of that directory.

pkgdir is in turn used to seed FOO_PKGDIR.

This all happens when we eventually call the package-generic infra, later down in the file.

When they are parsed, the Makefiles for each linux-extensions are appended to MAKEFILE_LIST, after the linux.mk one. But since they are located in the same directory as the main linux.mk, the last component of MaKEFILE_LIST, which is no longer the main linux.mk, will still yield the correct values for the linux package.

This is a tough assumption we made there and then.

When we added the support for br2-external linux extensions, we where very cautious to explicitly scan them from a directory named 'linux', so that this would yield the correct package name.

And that worked well so far, until someone needed to build an older kernel, for which our conditional patch is needed, and which just failed:

/bin/bash: [...]/buildroot-external-linux-test/linux//0001-timeconst.pl-Eliminate-Perl-warning.patch.conditional: No such file or directory

When we scan linux extensions from a br2-external tree, the last component of MAKEFILE_LIST is no longer in the same directory as the main linux.mk, and thus the assumption above falls to pieces...

Again, when we added support for linux extensions from br2-external, although we cared about the package name (pkgname), we completely missed out on the package directory, and the LINUX_PKGDIR variable.

We do not have a very clean way out of this mess, but we have a nice dirty trick scan the linux extensions from a br2-external tree before we scan the in-tree ones. That way, the last component of MAKEFILE_LIST is back to one that is in the same directory as the main linux.mk, and we're back on tracks.

This is still very fragile, though, but short of a complete overhaul on how packages are parsed and evaluated, this is the best we can come in short order.

Cc Heiko Thiery <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 linux/linux.mk | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)
linux/linux.mk +15 -10
index 6e9d11dd38..6df62df41b 100644
@@ -527,18 +527,23 @@ endef
 #
 # Note: our package infrastructure uses the full-path of the last-scanned
 # Makefile to determine what package we're currently defining, using the
-# last directory component in the path. As such, including other Makefiles,
-# like below, before we call one of the *-package macros usually doesn't
-# work.
-# However, since the files we include here are in the same directory as
-# the current Makefile, we are OK. But this is a hard requirement: files
-# included here *must* either be in this same directory OR within a
-# another directory with the name "linux" (in the BR2_EXTERNAL case).
-include $(sort $(wildcard linux/linux-ext-*.mk))
-
-# Import linux-kernel-extensions from br2-externals
+# last directory component in the path. Additionally, the full path of
+# the package directory is also stored in _PKGDIR (e.g. to find patches)
+#
+# As such, including other Makefiles, like below, before we call one of
+# the *-package macros usually doesn't work.
+#
+# However, by including the in-tree extensions after the ones from the
+# br2-external trees, we're back to the situation where the last Makefile
+# scanned *is* in included from the correct directory.
+#
+# NOTE: this is very fragile, and extra care must be taken to ensure that
+# we always end up with an in-tree included file. That's mostly OK, because
+# we do have in-tree linux-extensions.
+#
 include $(sort $(wildcard $(foreach ext,$(BR2_EXTERNAL_DIRS), \
 	$(ext)/linux/linux-ext-*.mk)))
+include $(sort $(wildcard linux/linux-ext-*.mk))
 
 LINUX_PATCH_DEPENDENCIES += $(foreach ext,$(LINUX_EXTENSIONS),\
 	$(if $(BR2_LINUX_KERNEL_EXT_$(call UPPERCASE,$(ext))),$(ext)))
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 package/linux: fix LINUX_PKGDIR with br2-external linux-extensions

Hi,

Am Fr., 24. Jan. 2020 um 22:39 Uhr schrieb Yann E. MORIN <[email protected]>:

Quoted 49 lines

When 'make' includes a new Makefile, it appends its path to the MAKEFILE_LIST variable. From that variable, we construct a few set of derivative variables:

    pkgdir = $(dir $(lastword $(MAKEFILE_LIST)))
    pkgname = $(lastword $(subst /, ,$(pkgdir)))

Essentially, pkgdir is the full directory where the package is located (either relative to Buildroot's top directory for in-tree packages, or absolute for packages in br2-external trees), while pkgname is the last component of that directory.

pkgdir is in turn used to seed FOO_PKGDIR.

This all happens when we eventually call the package-generic infra, later down in the file.

When they are parsed, the Makefiles for each linux-extensions are appended to MAKEFILE_LIST, after the linux.mk one. But since they are located in the same directory as the main linux.mk, the last component of MaKEFILE_LIST, which is no longer the main linux.mk, will still yield the correct values for the linux package.

This is a tough assumption we made there and then.

When we added the support for br2-external linux extensions, we where very cautious to explicitly scan them from a directory named 'linux', so that this would yield the correct package name.

And that worked well so far, until someone needed to build an older kernel, for which our conditional patch is needed, and which just failed:

/bin/bash: [...]/buildroot-external-linux-test/linux//0001-timeconst.pl-Eliminate-Perl-warning.patch.conditional: No such file or directory

When we scan linux extensions from a br2-external tree, the last component of MAKEFILE_LIST is no longer in the same directory as the main linux.mk, and thus the assumption above falls to pieces...

Again, when we added support for linux extensions from br2-external, although we cared about the package name (pkgname), we completely missed out on the package directory, and the LINUX_PKGDIR variable.

We do not have a very clean way out of this mess, but we have a nice dirty trick scan the linux extensions from a br2-external tree before we scan the in-tree ones. That way, the last component of MAKEFILE_LIST is back to one that is in the same directory as the main linux.mk, and we're back on tracks.

Quoted 4 lines

This is still very fragile, though, but short of a complete overhaul on how packages are parsed and evaluated, this is the best we can come in short order.

Thank you for the very detailed explanation.

Quoted 5 lines
Cc Heiko Thiery <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Quoted 45 lines
---
 linux/linux.mk | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)
linux/linux.mk +15 -11
index 6e9d11dd38..6df62df41b 100644
@@ -527,18 +527,23 @@ endef
 #
 # Note: our package infrastructure uses the full-path of the last-scanned
 # Makefile to determine what package we're currently defining, using the
-# last directory component in the path. As such, including other Makefiles,
-# like below, before we call one of the *-package macros usually doesn't
-# work.
-# However, since the files we include here are in the same directory as
-# the current Makefile, we are OK. But this is a hard requirement: files
-# included here *must* either be in this same directory OR within a
-# another directory with the name "linux" (in the BR2_EXTERNAL case).
-include $(sort $(wildcard linux/linux-ext-*.mk))
-
-# Import linux-kernel-extensions from br2-externals
+# last directory component in the path. Additionally, the full path of
+# the package directory is also stored in _PKGDIR (e.g. to find patches)
+#
+# As such, including other Makefiles, like below, before we call one of
+# the *-package macros usually doesn't work.
+#
+# However, by including the in-tree extensions after the ones from the
+# br2-external trees, we're back to the situation where the last Makefile
+# scanned *is* in included from the correct directory.
+#
+# NOTE: this is very fragile, and extra care must be taken to ensure that
+# we always end up with an in-tree included file. That's mostly OK, because
+# we do have in-tree linux-extensions.
+#
 include $(sort $(wildcard $(foreach ext,$(BR2_EXTERNAL_DIRS), \
        $(ext)/linux/linux-ext-*.mk)))
+include $(sort $(wildcard linux/linux-ext-*.mk))
 
 LINUX_PATCH_DEPENDENCIES += $(foreach ext,$(LINUX_EXTENSIONS),\
        $(if $(BR2_LINUX_KERNEL_EXT_$(call UPPERCASE,$(ext))),$(ext)))
--

2.20.1

#2 lore raw

PATCH 1/1 package/linux: fix LINUX_PKGDIR with br2-external linux-extensions

In reply to Yann E. MORIN
Quoted 54 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

When 'make' includes a new Makefile, it appends its path to the MAKEFILE_LIST variable. From that variable, we construct a few set of derivative variables:

    pkgdir = $(dir $(lastword $(MAKEFILE_LIST)))
    pkgname = $(lastword $(subst /, ,$(pkgdir)))

Essentially, pkgdir is the full directory where the package is located (either relative to Buildroot's top directory for in-tree packages, or absolute for packages in br2-external trees), while pkgname is the last component of that directory.

pkgdir is in turn used to seed FOO_PKGDIR.

This all happens when we eventually call the package-generic infra, later down in the file.

When they are parsed, the Makefiles for each linux-extensions are appended to MAKEFILE_LIST, after the linux.mk one. But since they are located in the same directory as the main linux.mk, the last component of MaKEFILE_LIST, which is no longer the main linux.mk, will still yield the correct values for the linux package.

This is a tough assumption we made there and then.

When we added the support for br2-external linux extensions, we where very cautious to explicitly scan them from a directory named 'linux', so that this would yield the correct package name.

And that worked well so far, until someone needed to build an older kernel, for which our conditional patch is needed, and which just failed:

/bin/bash: [...]/buildroot-external-linux-test/linux//0001-timeconst.pl-Eliminate-Perl-warning.patch.conditional: No such file or directory

When we scan linux extensions from a br2-external tree, the last component of MAKEFILE_LIST is no longer in the same directory as the main linux.mk, and thus the assumption above falls to pieces...

Again, when we added support for linux extensions from br2-external, although we cared about the package name (pkgname), we completely missed out on the package directory, and the LINUX_PKGDIR variable.

We do not have a very clean way out of this mess, but we have a nice dirty trick scan the linux extensions from a br2-external tree before we scan the in-tree ones. That way, the last component of MAKEFILE_LIST is back to one that is in the same directory as the main linux.mk, and we're back on tracks.

This is still very fragile, though, but short of a complete overhaul on how packages are parsed and evaluated, this is the best we can come in short order.

Quoted 5 lines
Cc Heiko Thiery <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>

Quite a mess, but a very nice commit message.

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#3 lore raw

PATCH 1/1 package/linux: fix LINUX_PKGDIR with br2-external linux-extensions

In reply to Yann E. MORIN
Quoted 54 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

When 'make' includes a new Makefile, it appends its path to the MAKEFILE_LIST variable. From that variable, we construct a few set of derivative variables:

    pkgdir = $(dir $(lastword $(MAKEFILE_LIST)))
    pkgname = $(lastword $(subst /, ,$(pkgdir)))

Essentially, pkgdir is the full directory where the package is located (either relative to Buildroot's top directory for in-tree packages, or absolute for packages in br2-external trees), while pkgname is the last component of that directory.

pkgdir is in turn used to seed FOO_PKGDIR.

This all happens when we eventually call the package-generic infra, later down in the file.

When they are parsed, the Makefiles for each linux-extensions are appended to MAKEFILE_LIST, after the linux.mk one. But since they are located in the same directory as the main linux.mk, the last component of MaKEFILE_LIST, which is no longer the main linux.mk, will still yield the correct values for the linux package.

This is a tough assumption we made there and then.

When we added the support for br2-external linux extensions, we where very cautious to explicitly scan them from a directory named 'linux', so that this would yield the correct package name.

And that worked well so far, until someone needed to build an older kernel, for which our conditional patch is needed, and which just failed:

/bin/bash: [...]/buildroot-external-linux-test/linux//0001-timeconst.pl-Eliminate-Perl-warning.patch.conditional: No such file or directory

When we scan linux extensions from a br2-external tree, the last component of MAKEFILE_LIST is no longer in the same directory as the main linux.mk, and thus the assumption above falls to pieces...

Again, when we added support for linux extensions from br2-external, although we cared about the package name (pkgname), we completely missed out on the package directory, and the LINUX_PKGDIR variable.

We do not have a very clean way out of this mess, but we have a nice dirty trick scan the linux extensions from a br2-external tree before we scan the in-tree ones. That way, the last component of MAKEFILE_LIST is back to one that is in the same directory as the main linux.mk, and we're back on tracks.

This is still very fragile, though, but short of a complete overhaul on how packages are parsed and evaluated, this is the best we can come in short order.

Quoted 5 lines
Cc Heiko Thiery <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>

Committed to 2019.11.x (2019.02.x does not have support for external linux extensions), thanks.

Signature · 1 line
Bye, Peter Korsgaard