linux: fix using extensions (branch yem/kernel-ext)
cover linux: fix using extensions (branch yem/kernel-ext)
Hello All!
This small series is an attempt at fixing the use of extensions, which somehow broke unnoticeably since we switched to using the kconfig-package infrastructure for the linux package.
Regards,
Yann E. MORIN.
The following changes since commit 6407a0070ff6ba6626bbbf2e0271c38c88c52a83:
busybox: enable arp applet (2015-03-12 23:52:12 +0100)
are available in the git repository at:
git://git.busybox.net/~ymorin/git/buildroot yem/kernel-ext
for you to fetch changes up to 7a6b64928c074474e7577f762550fe60916ed052:
linux: fix use of extensions (2015-03-13 19:55:21 +0100)
----------------------------------------------------------------
Yann E. MORIN (2):
linux: add note about why it's safe to include other .mk files
linux: fix use of extensions
linux/linux-ext-fbtft.mk | 6 ++---- linux/linux-ext-rtai.mk | 6 ++---- linux/linux-ext-xenomai.mk | 6 ++---- linux/linux.mk | 15 +++++++++++++++ 4 files changed, 21 insertions(+), 12 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/2 linux: add note about why it's safe to include other .mk files
--- linux/linux.mk | 8 ++++++++ 1 file changed, 8 insertions(+)
linux/linux.mk +8 -0
@@ -323,6 +323,14 @@ define LINUX_INSTALL_TARGET_CMDS $(LINUX_INSTALL_HOST_TOOLS) endef +# Note: our package infrastructure uses the full-path of the last-scanned +# Makefile to determine what package we're currently defining, using the +# last directory component in the path. As such, including other Makefile, +# like below, before we call one of the *-package macro is usally not +# working. +# However, since the files we include here are in the same directory as +# the current Makefile, we are OK. But this is a hard requirement: files +# included here *must* be in the same directory! include $(sort $(wildcard linux/linux-ext-*.mk)) $(eval $(kconfig-package))
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH 1/2 linux: add note about why it's safe to include other .mk files
Dear Yann E. MORIN,
Quoted 3 lines · Yann E. MORIN
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Thomas De Schampheleire <[email protected]>
Applied, thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH 2/2 linux: fix use of extensions
In reply to Yann E. MORINCurrently, using externsaions is flawed and does not work in a very reproducible way, starting from a _clean_ tree:
make menuconfig
-> enable a kernel and any externsion (fbtft is nice, since
it works for recent kernels)
make linux-menuconfig
Observe how it is not patching the kernel tree with the extensions prior to running the linux configuratin UI?
This is because dependencies are only acted on at configure time, which is a step further after the kconfig stage. This probably was not an issue before we switched to the kconfig infra for the kernel, but that use-case was completely missed at the time (blame me!).
Fix that:
- first, the dependency on extensions is moved to a dependency of the
patch step;
- then, to avoid circular dependencies (e.g. linux->rtai->linux), do not add extensions to LINUX_DEPENDENCIES, instead, add them to a special variable, from which we derive both the list of dependencies and the list of post-patch hooks.
This makes it slightly easier to write linux extensions: no need to delve in the .stamp_patched internals for each extension, just a function to (conditionally) define and a variable to assign.
(Note: this should go into a section of the manual...)
--- linux/linux-ext-fbtft.mk | 6 ++---- linux/linux-ext-rtai.mk | 6 ++---- linux/linux-ext-xenomai.mk | 6 ++---- linux/linux.mk | 7 +++++++ 4 files changed, 13 insertions(+), 12 deletions(-)
linux/linux-ext-fbtft.mk +2 -4
@@ -5,8 +5,8 @@ ################################################################################ ifeq ($(BR2_LINUX_KERNEL_EXT_FBTFT),y) -# Add dependency to fbtft package (download helper for the fbtft source) -LINUX_DEPENDENCIES += fbtft + +LINUX_EXTENSIONS += fbtft # for linux >= 3.15 install to drivers/video/fbdev/fbtft # for linux < 3.15 install to drivers/video/fbtft
@@ -23,6 +23,4 @@ define FBTFT_PREPARE_KERNEL echo 'obj-y += fbtft/' >> $${dest}/Makefile endef -LINUX_PRE_PATCH_HOOKS += FBTFT_PREPARE_KERNEL - endif #BR2_LINUX_KERNEL_EXT_FBTFT
linux/linux-ext-rtai.mk +2 -4
@@ -5,8 +5,8 @@ ################################################################################ ifeq ($(BR2_LINUX_KERNEL_EXT_RTAI),y) -# Add dependency to RTAI (user-space) which provide kernel patches -LINUX_DEPENDENCIES += rtai-patch + +LINUX_EXTENSIONS += rtai RTAI_PATCH = $(call qstrip,$(BR2_LINUX_KERNEL_EXT_RTAI_PATCH))
@@ -42,6 +42,4 @@ define RTAI_PREPARE_KERNEL endef endif -LINUX_PRE_PATCH_HOOKS += RTAI_PREPARE_KERNEL - endif #BR2_LINUX_EXT_RTAI
linux/linux-ext-xenomai.mk +2 -4
@@ -5,8 +5,8 @@ ################################################################################ ifeq ($(BR2_LINUX_KERNEL_EXT_XENOMAI),y) -# Add dependency to xenomai (user-space) which provide ksrc part -LINUX_DEPENDENCIES += xenomai + +LINUX_EXTENSIONS += xenomai # Adeos patch version XENOMAI_ADEOS_PATCH = $(call qstrip,$(BR2_LINUX_KERNEL_EXT_XENOMAI_ADEOS_PATCH))
@@ -25,6 +25,4 @@ define XENOMAI_PREPARE_KERNEL --verbose endef -LINUX_PRE_PATCH_HOOKS += XENOMAI_PREPARE_KERNEL - endif #BR2_LINUX_EXT_XENOMAI
linux/linux.mk +7 -0
@@ -333,8 +333,15 @@ endef # included here *must* be in the same directory! include $(sort $(wildcard linux/linux-ext-*.mk)) +# Include extension-provided post-patch hooks +LINUX_PRE_PATCH_HOOKS = $(call UPPERCASE,$(patsubst %,%_PREPARE_KERNEL,$(LINUX_EXTENSIONS))) + $(eval $(kconfig-package)) +# We need to extract and patch all extensions before we can patch the # kernel. +# Needs to be _after_ we call kconfig-package, because we need LINUX_DIR +$(LINUX_DIR)/.stamp_patched: | $(patsubst %,%-patch,$(LINUX_EXTENSIONS)) + # Support for rebuilding the kernel after the cpio archive has # been generated in $(BINARIES_DIR)/rootfs.cpio. $(LINUX_DIR)/.stamp_initramfs_rebuilt: $(LINUX_DIR)/.stamp_target_installed $(LINUX_DIR)/.stamp_images_installed $(BINARIES_DIR)/rootfs.cpio
Signature · 1 line
1.9.1
PATCH 2/2 linux: fix use of extensions
Dear Yann E. MORIN,
Quoted 1 line · Yann E. MORIN
Currently, using externsaions is flawed and does not work in a very
extensions
Quoted 10 lines
reproducible way, starting from a _clean_ tree:
make menuconfig
-> enable a kernel and any externsion (fbtft is nice, since
it works for recent kernels)make linux-menuconfig
Observe how it is not patching the kernel tree with the extensions prior to running the linux configuratin UI?
configuration
Quoted 4 lines
This is because dependencies are only acted on at configure time, which is a step further after the kconfig stage. This probably was not an issue before we switched to the kconfig infra for the kernel, but that use-case was completely missed at the time (blame me!).
Hum, I am not sure to see why the switch to the kconfig-package infrastructure would have modified this behavior. So I'd like to understand how it used to work, if it ever worked (but I believe it did, no?).
Quoted 4 lines
Fix that:
- first, the dependency on extensions is moved to a dependency of the
patch step;
This is a bit problematic because then the dependency is unknown to the package infrastructure. Which means that things like 'make graph-depends' will no longer display this dependency.
To solve this, we would have to add a <pkg>_PATCH_DEPENDENCIES variable in pkg-generic, or something like that.
But again, I'd like to understand why we didn't need that until now.
Quoted 9 lines
- then, to avoid circular dependencies (e.g.
linux->rtai->linux), do not add extensions toLINUX_DEPENDENCIES, instead, add them to a special variable, from which we derive both the list of dependencies and the list of post-patch hooks.This makes it slightly easier to write linux extensions: no need to delve in the .stamp_patched internals for each extension, just a function to (conditionally) define and a variable to assign.
I'm not sure to understand how the linux extensions had to delve into the .stamp_patched internals. They were just registering a POST_PATCH hook, no?
Quoted 1 line
(Note: this should go into a section of the manual...)
Yes, indeed, once we settle on a solution :)
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH 2/2 linux: fix use of extensions
Thomas, All,
On 2015-03-13 21:47 +0100, Thomas Petazzoni spake thusly:
Quoted 3 lines · Yann E. MORIN
Quoted 1 line
Currently, using externsaions is flawed and does not work in a very
extensions
[--SNIP--]
Quoted 2 lines
Quoted 1 line
to running the linux configuratin UI?
configuration
Damn, I even re-read myself before sending. Seems it was not enough... :-/
Quoted 9 lines
Quoted 4 lines
This is because dependencies are only acted on at configure time, which is a step further after the kconfig stage. This probably was not an issue before we switched to the kconfig infra for the kernel, but that use-case was completely missed at the time (blame me!).
Hum, I am not sure to see why the switch to the kconfig-package infrastructure would have modified this behavior. So I'd like to understand how it used to work, if it ever worked (but I believe it did, no?).
Fair enough, I'll double-check that it did / did not work back then.
Quoted 8 lines
Quoted 4 lines
Fix that:
- first, the dependency on extensions is moved to a dependency of the
patch step;This is a bit problematic because then the dependency is unknown to the package infrastructure. Which means that things like 'make graph-depends' will no longer display this dependency.
Right. Note however that this was already the case for the RTAI externsion, because it was declaring:
LINUX_DEPENDENCIES += rtai-patch
so that was already missed (or at least mis-interpreted) by graph-depends anyway.
Quoted 4 lines
To solve this, we would have to add a <pkg>_PATCH_DEPENDENCIES variable in pkg-generic, or something like that.
But again, I'd like to understand why we didn't need that until now.
As said above, I'll double-check on that.
Quoted 5 lines
Quoted 5 lines
- then, to avoid circular dependencies (e.g.
linux->rtai->linux), do not add extensions toLINUX_DEPENDENCIES, instead, add them to a special variable, from which we derive both the list of dependencies and the list of post-patch hooks.
Quoted 7 lines
Quoted 3 lines
This makes it slightly easier to write linux extensions: no need to delve in the .stamp_patched internals for each extension, just a function to (conditionally) define and a variable to assign.
I'm not sure to understand how the linux extensions had to delve into the .stamp_patched internals. They were just registering a
POST_PATCHhook, no?
No, they _did not_ have to so far.
What I meant is that the switch to a dependency of the patch step required that they would now all have had to write something like:
$(LINUX_DIR)/.stamp_patched: | EXT-patch
(where EXT is the name of the extension.)
To avoid such an arcane code that would have to be replicated (and potentially tracked down in case we change something in dependency handling), I found it would be better to have it all handled in a single location.
Anyway, I'll look at your suggestion of introducing FOO_PATCH_DEPENDENCIES. However, linux is the sole package that requires such handling, and I wonder if it is worth introducing for just a single package (note that I rehash your own argument, hehe! ;-) )
Quoted 3 lines
Quoted 1 line
(Note: this should go into a section of the manual...)
Yes, indeed, once we settle on a solution :)
Eh, yes! :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/2 linux: fix use of extensions
Dear Yann E. MORIN,
Quoted 6 lines · Yann E. MORIN
Quoted 4 lines
Hum, I am not sure to see why the switch to the kconfig-package infrastructure would have modified this behavior. So I'd like to understand how it used to work, if it ever worked (but I believe it did, no?).
Fair enough, I'll double-check that it did / did not work back then.
Great.
Quoted 11 lines
Quoted 3 lines
This is a bit problematic because then the dependency is unknown to the package infrastructure. Which means that things like 'make graph-depends' will no longer display this dependency.
Right. Note however that this was already the case for the RTAI externsion, because it was declaring:
LINUX_DEPENDENCIES += rtai-patchso that was already missed (or at least mis-interpreted) by graph-depends anyway.
Indeed, correct. And that's not nice :/
Quoted 9 lines
Quoted 3 lines
I'm not sure to understand how the linux extensions had to delve into the .stamp_patched internals. They were just registering a
POST_PATCHhook, no?No, they
_didnot_have to so far.What I meant is that the switch to a dependency of the patch step required that they would now all have had to write something like:
Quoted 8 lines
$(LINUX_DIR)/.stamp_patched: | EXT-patch(where EXT is the name of the extension.)
To avoid such an arcane code that would have to be replicated (and potentially tracked down in case we change something in dependency handling), I found it would be better to have it all handled in a single location.
Ok, understood.
Quoted 5 lines
Anyway, I'll look at your suggestion of introducing
FOO_PATCH_DEPENDENCIES. However, linux is the sole package that requires such handling, and I wonder if it is worth introducing for just a single package (note that I rehash your own argument, hehe! ;-) )
Point taken :-)
But my argument about putting things in the infra only if at least a significant number of packages need it is only valid if there's another way of doing it that works. For example, if you have three packages that do --disable-foobar, then it's not a strong argument to put it in the infra because it can perfectly be done in a per-package fashion.
However, things such making sure that the infra is aware of weird dependencies is not something you can fix at the per-package level, you need some support from the infrastructure.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH 2/2 linux: fix use of extensions
In reply to Thomas PetazzoniThomas, All,
On 2015-03-13 21:47 +0100, Thomas Petazzoni spake thusly:
Quoted 1 line · Yann E. MORIN
[--SNIP--]
Quoted 9 lines
Quoted 4 lines
This is because dependencies are only acted on at configure time, which is a step further after the kconfig stage. This probably was not an issue before we switched to the kconfig infra for the kernel, but that use-case was completely missed at the time (blame me!).
Hum, I am not sure to see why the switch to the kconfig-package infrastructure would have modified this behavior. So I'd like to understand how it used to work, if it ever worked (but I believe it did, no?).
OK, so I think I know what happened when we migrated linux over to the kconfig-package infra, and what we loose then.
Back before we used the kconfig-package infra, linux-menuconfig (as well as the other configurators) was written as custom rules, that were basically something like:
linux-menuconfig: linux-configure
$(MAKE) -C $(LINUX_DIR) menuconfigAnd there we had our dependency on a previously configured linux tree, and *that* was kicking-in our dependency on extensions.
Fast forward post 2015.02, and we now have linux use the kconfig-package infra, which completely decorrelates the kconfig-part of the configuration, from the actual package-part of the configuration, as it introduces an intermediate 'kconfig-fixup' step:
linux-configure -------> kconfig-fixup --> .config --> $(LINUX_CONFIG_FILE)
/
linux-menuconfig --'So, linux-menuconfig now no longer depends on the linux tree to be configured. Hence the extensions breaking...
And this, I hope, concludes the nalaysis and is enough to convince you this is a new problem, caused by the kconfig-package infra. Hehe! :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/2 linux: fix use of extensions
In reply to Thomas PetazzoniThomas, All,
On 2015-03-13 21:47 +0100, Thomas Petazzoni spake thusly:
Quoted 1 line · Yann E. MORIN
[--SNIP--]
Quoted 11 lines
Quoted 4 lines
Fix that:
- first, the dependency on extensions is moved to a dependency of the
patch step;This is a bit problematic because then the dependency is unknown to the package infrastructure. Which means that things like 'make graph-depends' will no longer display this dependency.
To solve this, we would have to add a <pkg>_PATCH_DEPENDENCIES variable in pkg-generic, or something like that.
OK, so I tried to implement your proposal, but I stumbled against various issues:
1- what should be done with a patch-dependency: must it be handled like
a normal dependency, i.e. be fully built and installed? Or should it just be extracted and patched itself?
We're trying to introduce this new type of dependency to solve the linux extension case. So, we do not really require the extension to be built in the first place; we only need their files so we can vampirise them, so simply extracting and patching is all we currently need. The RTAI extension so far goes as far as explicitly adding a dependency on "rtai-patch" instead of just "rtai", as rtai depends on linux. So, to use this new feature for linux extensions, we really want *not* to build the extension.
However, introducing this new feature in the pkg-generic infra means it is available to other packages as well, and we can not predict what packages could use it for. Would there be packages that want/need a built patch-dependency?
There's a kind of discrepancy in there, as normal dependencies are dependencies to be built, while patch-dependencies are dependencies that only need to be, well, patched... (I find it odd.)
2- extensions are really not supported for _OVERRIDE_SRCDIR, since they
are handled as pre-patch hooks, and _OVERRIDE_SRCDIR does not do patches.
Is that something we want to live by, or do we want to support extensions even in the _OVERRIDE_SRCDIR?
Note that this is not a new issue: it has always been the case so far, even before the migration to kconfig-package.
[Though this be madness, yet there is method in't.]
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'