Add support for specifying a custom kernel directory

Add the ability to specify a path to a custom directory where kernel sources may be contained. This is useful when doing kernel development in an existing git tree.

Signed-off-by Florian Fainelli <[email protected]>

---
 linux/Config.in                        | 10 ++++++++++
 linux/linux.mk                         |  3 +++
 package/linux-headers/linux-headers.mk |  5 +++++
 3 files changed, 18 insertions(+)

linux/Config.in +10 -0

index 33ae944557b6..7d4dcddc7ea7 100644
@@ -87,6 +87,12 @@ config BR2_LINUX_KERNEL_CUSTOM_SVN
 	  This option allows Buildroot to get the Linux kernel source
 	  code from a Subversion repository.
 
+config BR2_LINUX_KERNEL_CUSTOM_DIRECTORY
+	bool "Custom directory"
+	help
+	  This option allows Buildroot to use the Linux kernel source
+	  code from an existing directory.
+
 endchoice
 
 config BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE

@@ -97,6 +103,10 @@ config BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION
 	string "URL of custom kernel tarball"
 	depends on BR2_LINUX_KERNEL_CUSTOM_TARBALL
 
+config BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION
+	string "Path of the custom kernel source"
+	depends on BR2_LINUX_KERNEL_CUSTOM_DIRECTORY
+
 if BR2_LINUX_KERNEL_CUSTOM_GIT || BR2_LINUX_KERNEL_CUSTOM_HG || BR2_LINUX_KERNEL_CUSTOM_SVN
 
 config BR2_LINUX_KERNEL_CUSTOM_REPO_URL

linux/linux.mk +3 -0

index bd5589bae02d..192fd0d3ffb3 100644
@@ -29,6 +29,9 @@ LINUX_SITE_METHOD = hg
 else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_SVN),y)
 LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_SITE_METHOD = svn
+else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
+LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION))
+LINUX_SITE_METHOD = local
 else ifeq ($(BR2_LINUX_KERNEL_LATEST_CIP_VERSION),y)
 LINUX_SITE = git://git.kernel.org/pub/scm/linux/kernel/git/bwh/linux-cip.git
 else ifneq ($(findstring -rc,$(LINUX_VERSION)),)

package/linux-headers/linux-headers.mk +5 -0

index f1e379060834..e42af8a4d672 100644
@@ -26,6 +26,9 @@ LINUX_HEADERS_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_HEADERS_SITE_METHOD = hg
 # use same hg tarball as linux kernel
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.gz
+else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
+LINUX_HEADERS_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION))
+LINUX_HEADERS_SITE_METHOD = local
 else
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
 # In X.Y.Z, get X and Y. We replace dots and dashes by spaces in order

@@ -122,6 +125,7 @@ define LINUX_HEADERS_INSTALL_STAGING_CMDS
 			headers_install)
 endef
 
+ifneq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
 ifeq ($(BR2_KERNEL_HEADERS_VERSION)$(BR2_KERNEL_HEADERS_AS_KERNEL),y)
 define LINUX_HEADERS_CHECK_VERSION
 	$(call check_kernel_headers_version,\

@@ -130,5 +134,6 @@ define LINUX_HEADERS_CHECK_VERSION
 endef
 LINUX_HEADERS_POST_INSTALL_STAGING_HOOKS += LINUX_HEADERS_CHECK_VERSION
 endif
+endif
 
 $(eval $(generic-package))

Signature · 6 lines

2.7.4

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:

Quoted 3 lines

Add the ability to specify a path to a custom directory where kernel sources may be contained. This is useful when doing kernel development in an existing git tree.

This case is covered by the override-sourcedir mechanism.

Create a file (by default, local.mk) in your config directory (the one with the Buildroot .config file), and edit this file with:

LINUX_OVERRIDE_SRCDIR = /path/to/your/linux

and Buildroot will use that as a rsync source, instead of downloading the kernel sources.

See also e782cd5b1bc (Revert "Added local directory as source of kernel code") for more in-depth explanations. ;-)

Hello,

Quoted 15 lines · Yann E. MORIN

On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:

Quoted 3 lines

Add the ability to specify a path to a custom directory where kernel sources may be contained. This is useful when doing kernel development in an existing git tree.

This case is covered by the override-sourcedir mechanism.

Create a file (by default, local.mk) in your config directory (the one with the Buildroot .config file), and edit this file with:

LINUX_OVERRIDE_SRCDIR = /path/to/your/linux

and Buildroot will use that as a rsync source, instead of downloading the kernel sources.

Quoted 2 lines

See also e782cd5b1bc (Revert "Added local directory as source of kernel code") for more in-depth explanations. ;-)

Fully agreed with Yann here: there is no point in adding a Linux-specific solution for this use case, as we already have a much more general solution that works for all packages, including the 'linux' package.

Quoted 19 lines

Hello,

Quoted 15 lines · Yann E. MORIN

On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:

Quoted 3 lines

Add the ability to specify a path to a custom directory where kernel sources may be contained. This is useful when doing kernel development in an existing git tree.

This case is covered by the override-sourcedir mechanism.

Create a file (by default, local.mk) in your config directory (the one with the Buildroot .config file), and edit this file with:

LINUX_OVERRIDE_SRCDIR = /path/to/your/linux

and Buildroot will use that as a rsync source, instead of downloading the kernel sources.

Quoted 7 lines

Quoted 2 lines

See also e782cd5b1bc (Revert "Added local directory as source of kernel code") for more in-depth explanations. ;-)

Fully agreed with Yann here: there is no point in adding a Linux-specific solution for this use case, as we already have a much more general solution that works for all packages, including the 'linux' package.

That seems fine in premise, but there are still a few angles that just don't feel right or I am most certainly just misusing this:

- we still need to specify a "custom" kernel version, and apparently it cannot be empty and just mean "does not matter", that's extremely impractical, if the workflow involves a bisection over several kernel versions, there is no way I am specifying the kernel version every time I run git bisect

- how about kernel-headers? My local.mk now has this:

LINUX_OVERRIDE_SRCDIR = /home/florian/dev/linux LINUX_HEADERS_OVERRIDE_SRCDIR = /home/florian/dev/linux

but it still won't let me build with "Same as kernel being built", see below [1]

- if someone is seeing any issues with Buildroot using this (or anything for that matter), it would a lot easier for you to just tell them to send their .config file, now you also need to ask them whether they are using local.mk, and send it too, that assumes people read "how to submit bugs" documentation anyway

[1]:
package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE cannot be empty when LINUX_HEADERS_SOURCE is not. Stop. Makefile:79: recipe for target '_all' failed
make: *** [_all] Error 2
zsh: exit 2 make

Florian, All,

On 2017-10-29 10:26 -0700, Florian Fainelli spake thusly:

Quoted 28 lines · Thomas Petazzoni

Quoted 19 lines

Hello,

Quoted 15 lines · Yann E. MORIN

On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:

Quoted 3 lines

Add the ability to specify a path to a custom directory where kernel sources may be contained. This is useful when doing kernel development in an existing git tree.

This case is covered by the override-sourcedir mechanism.

Create a file (by default, local.mk) in your config directory (the one with the Buildroot .config file), and edit this file with:

LINUX_OVERRIDE_SRCDIR = /path/to/your/linux

and Buildroot will use that as a rsync source, instead of downloading the kernel sources.

Quoted 7 lines

Quoted 2 lines

See also e782cd5b1bc (Revert "Added local directory as source of kernel code") for more in-depth explanations. ;-)

Fully agreed with Yann here: there is no point in adding a Linux-specific solution for this use case, as we already have a much more general solution that works for all packages, including the 'linux' package.

Quoted 8 lines

That seems fine in premise, but there are still a few angles that just don't feel right or I am most certainly just misusing this:

- we still need to specify a "custom" kernel version, and apparently it cannot be empty and just mean "does not matter", that's extremely impractical, if the workflow involves a bisection over several kernel versions, there is no way I am specifying the kernel version every time I run git bisect

Just leave trhe version at its default value; it is only ever used to know what to download in the first place, and since you use an override, it will download nothing.

Quoted 7 lines

- how about kernel-headers? My local.mk now has this:

LINUX_OVERRIDE_SRCDIR = /home/florian/dev/linux LINUX_HEADERS_OVERRIDE_SRCDIR = /home/florian/dev/linux

but it still won't let me build with "Same as kernel being built", see below [1]

In that case, you have a bigger problem: you will have to rebuild the toolchain every time, as the headers are changing. And thus, you will have to rebuild everything else (all packages).

And if/when your bisect crosses kernel version boundaries, you'll have to adjust the "Custom kernel headers series" as well too.

So I'd say that you would probably be more impacted by the build time first, no?

So, for bisecting a kernel, I'd try to not have to rebuild the toolchain every tine, so stick with a static headers version.

Unless you are explicitly trying to bisect a headers problem...

Quoted 6 lines

- if someone is seeing any issues with Buildroot using this (or anything for that matter), it would a lot easier for you to just tell them to send their .config file, now you also need to ask them whether they are using local.mk, and send it too, that assumes people read "how to submit bugs" documentation anyway

Meh, you know what we say about documentation? No one reads it! ;-)

But yeah, we've had to deal with that a few times, and it soon becomes apparent that the user is using override-srcdir, because the paths then read something like 'linux-custom', which is easy to spot.

Quoted 6 lines

[1]:
package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE cannot be empty when LINUX_HEADERS_SOURCE is not. Stop. Makefile:79: recipe for target '_all' failed
make: *** [_all] Error 2
zsh: exit 2 make

Hmm... Even when you keep the kernel version to its default value?

Le 10/29/17 ? 11:12, Yann E. MORIN a ?crit?:

Quoted 8 lines

On 2017-10-29 10:26 -0700, Florian Fainelli spake thusly:

Quoted 5 lines

package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE cannot be empty when LINUX_HEADERS_SOURCE is not. Stop. Makefile:79: recipe for target '_all' failed
make: *** [_all] Error 2
zsh: exit 2 make

Hmm... Even when you keep the kernel version to its default value?

That was also because I had set the "custom kernel version" to an empty string, which could have indicated "don't care" but was still needed. This is also probably fine to set to anything once you set your kernel headers to a particular version.

It seems like we should not prompted any kernel version in case since we use a local kernel directory, or, the version should be derived directly from the kernel directory so it does not have to be manually specified at all but that's a tad more complicated.

Thanks for your help, it's been pretty interesting to use another build system thus far :)

Florian, All,

On 2017-10-29 20:27 -0700, Florian Fainelli spake thusly:

Quoted 15 lines

Le 10/29/17 ? 11:12, Yann E. MORIN a ?crit?:

Quoted 8 lines

On 2017-10-29 10:26 -0700, Florian Fainelli spake thusly:

Quoted 5 lines

package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE cannot be empty when LINUX_HEADERS_SOURCE is not. Stop. Makefile:79: recipe for target '_all' failed
make: *** [_all] Error 2
zsh: exit 2 make

Hmm... Even when you keep the kernel version to its default value?

That was also because I had set the "custom kernel version" to an empty string, which could have indicated "don't care" but was still needed. This is also probably fine to set to anything once you set your kernel headers to a particular version.

Quoted 4 lines

It seems like we should not prompted any kernel version in case since we use a local kernel directory, or, the version should be derived directly from the kernel directory so it does not have to be manually specified at all but that's a tad more complicated.

But in truth, the override-srcdir mechanism is just that: an _override_ of the configured values, to be used only for development.

So, the premise is that you start off from a "known" configuration, and override just that one or two packages you are working on.

Quoted 2 lines

Thanks for your help, it's been pretty interesting to use another build system thus far :)

I was rather surprised (in a good sense!) to see you here, yes. Trying the Dark Side of the Force, are we? ;-]

OK. So by the same token, I should not expect that rsync operation syncing my development tree to output/build/linux-custom to work past the initial build/configuration? Right now it does not, I can make as many changes as I want in the directory reference from local.mk and output/build/linux-custom remains the same as it was the first time it got rsync'd, is that really expected?

I really wonder how you guys are doing serious kernel development with buildroot?

OK. So by the same token, I should not expect that rsync operation syncing my development tree to output/build/linux-custom to work past the initial build/configuration? Right now it does not, I can make as many changes as I want in the directory reference from local.mk and output/build/linux-custom remains the same as it was the first time it got rsync'd, is that really expected?

I really wonder how you guys are doing serious kernel development with buildroot?

This series allows specifying a custom kernel directory which could be an existing git tree. This is particularly useful when doing kernel development and having an existing git checkout where development is happening.

Ideally, I would like to avoid doing the rsync operation involved with the "local" method and instead just have a symbolic link from /path/to/kernel -> output/build/linux.

Comments welcome!

Florian Fainelli (2):
pkg-generic: Don't check for trailing slashes for local method
linux: Add support for specifying a custom directory

 linux/Config.in                        | 10 ++++++++++
 linux/linux.mk                         |  3 +++
 package/linux-headers/linux-headers.mk |  5 +++++
 package/pkg-generic.mk                 |  2 ++
 4 files changed, 20 insertions(+)

Signature · 6 lines

2.7.4

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Florian, All,

On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:

Quoted 3 lines

This series allows specifying a custom kernel directory which could be an existing git tree. This is particularly useful when doing kernel development and having an existing git checkout where development is happening.

As I replied to the patch itself: this is already covered by the opverride-srcdir mechanism, so I've marked this series as rejected in our patchwork.

Quoted 2 lines

Ideally, I would like to avoid doing the rsync operation involved with the "local" method and instead just have a symbolic link from /path/to/kernel -> output/build/linux.

The first rsync may take a few seconds, true, but subs=equent ones will be much faster if all you do is change a few files, especially on a hot directory, so there is not much gain to have with a local symlink.

OTOH, I am currently working on out-of-tree per-pacakge build, which will (eventually) allow us to make use of the O= mechanism to build the kernel out-of-tree, and get rid of the rsync altogether. Don't hold your breath, though. ;-)