make: add modules_update target
PATCH 1/1 make: add modules_update target
Here is a a patch that adds a kernel build target called "modules_update".
The existing "modules_install" target blows away the entire /lib/modules/uname -r/kernel directory and copies out every single module when called at the top level.
This new "modules_update" target only copies out modules that have changed, using "cp -u". This less zealous method is a more efficient approach to module installation for kernel developers working on single, or small numbers of modules.
Where "make modules_install" was taking ~ 3 minutes, "make modules_update" is taking < 1 minute, as a rough benchmark.
--- Makefile | 34 ++++++++++++++++++++++++++++++++++ scripts/Makefile.modupdt | 29 +++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+)
Makefile +34 -0
@@ -922,6 +919,40 @@ modules: $(vmlinux-dirs) $(if $(KBUILD_B .PHONY: modules_prepare modules_prepare: prepare scripts +# Target to update modules +.PHONY: modules_update +modules_update: _modupdt_ _modupdt_post + +.PHONY: _modupdt_ +_modupdt_: + @if [ -z "`$(DEPMOD) -V 2>/dev/null | grep module-init-tools`" ]; then \ + echo "Warning: you may need to install module-init-tools"; \ + echo "See http://www.codemonkey.org.uk/docs/post-halloween-2.6.txt";\ + sleep 1; \ + fi + @rm -f $(MODLIB)/source + @mkdir -p $(MODLIB)/kernel + @ln -s $(srctree) $(MODLIB)/source + @if [ ! $(objtree) -ef $(MODLIB)/build ]; then \ + rm -f $(MODLIB)/build ; \ + ln -s $(objtree) $(MODLIB)/build ; \ + fi + $(Q)$(MAKE) -rR -f $(srctree)/scripts/Makefile.modupdt + +# If System.map exists, run depmod. This deliberately does not have a +# dependency on System.map since that would run the dependency tree on +# vmlinux. This depmod is only for convenience to give the initial +# boot a modules.dep even before / is mounted read-write. However the +# boot script depmod is the master version. +ifeq "$(strip $(INSTALL_MOD_PATH))" "" +depmod_opts := +else +depmod_opts := -b $(INSTALL_MOD_PATH) -r +endif +.PHONY: _modupdt_post +_modupdt_post: _modupdt_ + if [ -r System.map -a -x $(DEPMOD) ]; then $(DEPMOD) -ae -F System.map $(depmod_opts) $(KERNELRELEASE); fi + # Target to install modules .PHONY: modules_install modules_install: _modinst_ _modinst_post
scripts/Makefile.modupdt +29 -0
@@ -0,0 +1,29 @@ +# ========================================================================== +# Installing modules +# ========================================================================== + +.PHONY: __modupdt +__modupdt: + +include scripts/Kbuild.include + +# + +__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod))) +modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o))) + +.PHONY: $(modules) +__modupdt: $(modules) + @: + + cmd_modules_update = mkdir -p $(2); cp -vu $@ $(2) | $(AWK) -F"'" '{print $$1}' | sed "s/\`/ UPDATE /" + +# Modules built outside the kernel source tree go into extra by default +INSTALL_MOD_DIR ?= extra +ext-mod-dir = $(INSTALL_MOD_DIR)$(subst $(KBUILD_EXTMOD),,$(@D)) + +modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D)) + +$(modules): + $(call cmd,modules_update,$(MODLIB)/$(modinst_dir)) +
Quoted 11 lines · Kylene Jo Hall
Here is a a patch that adds a kernel build target called "
modules_update".The existing "
modules_install" target blows away the entire/lib/modules/uname -r/kernel directory and copies out every single module when called at the top level.This new "
modules_update" target only copies out modules that have changed, using "cp -u". This less zealous method is a more efficient approach to module installation for kernel developers working on single, or small numbers of modules.
Hi Kylene,
This works as long as the .config hasn't been changed so that some configuration options haven't been changed so that a driver which had been previously built as a module is now built into the kernel. In that case, you really want to make sure the no-longer applicable .ko file has been removed from the system. If the developer knows that to be true, they can use your proposed modules_update without any problems.
As a suggestion, something that might be worth trying would be to change to modules_install so that it uses cp -u, but also so that it tries to delete all files that could have previously installed as modules (by using the obj-y list). This should hopefully speed up modules_install, and make it do the right thing all the time.
Regards,
- Ted
Quoted 17 lines · Theodore Ts'o
Quoted 5 lines · Kylene Jo Hall
This new "
modules_update" target only copies out modules that have changed, using "cp -u". This less zealous method is a more efficient approach to module installation for kernel developers working on single, or small numbers of modules.Hi Kylene,
This works as long as the .config hasn't been changed so that some configuration options haven't been changed so that a driver which had been previously built as a module is now built into the kernel. In that case, you really want to make sure the no-longer applicable .ko file has been removed from the system. If the developer knows that to be true, they can use your proposed
modules_updatewithout any problems.
Quoted 7 lines
As a suggestion, something that might be worth trying would be to change to
modules_installso that it uses cp -u, but also so that it tries to delete all files that could have previously installed as modules (by using the obj-y list). This should hopefully speed upmodules_install, and make it do the right thing all the time.
How about using rsync with --delete as a substitute for cp (if rsync is available)?
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 2 lines · Avi Kivity
How about using rsync with --delete as a substitute for cp (if rsync is available)?
I thought about this, but a "grep -r rsync" didn't turn up any previous hits in the kernel build process. I didn't want to introduce this as a new dependency for kernel building, if it's possible to avoid...
:-Dustin
Quoted 9 lines · Dustin Kirkland
Quoted 2 lines · Avi Kivity
How about using rsync with --delete as a substitute for cp (if rsync is available)?
I thought about this, but a "grep -r rsync" didn't turn up any previous hits in the kernel build process. I didn't want to introduce this as a new dependency for kernel building, if it's possible to avoid...
Use rsync only if it is available:
rsync-available := $(shell rsync --version > /dev/null 2>&1 && echo y)
copy := $(if $(rsync-available), rsync --delete, cp)
modules_install:
[...]
$(copy) source targetSignature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 8 lines · Avi Kivity
Use rsync only if it is available:
rsync-available := $(shell rsync --version > /dev/null 2>&1 && echo y) copy := $(if $(rsync-available), rsync --delete, cp) modules_install: [...] $(copy) source target
Actually, rsync --delete is not a viable option either. If you first build a kernel with a particular item built as a module, and then afterward rebuild with the same item as built-in (or not at all), the .ko file remains in your kernel build tree (ie, it won't be deleted on the source such that the --delete would have your desired effect).
Furthermore, rsync's performance is considerably worse in my testing than "cp -u", almost back to the original performance of the "rm -rf, cp all" of the original modules_install. (Note that I tested, rsync's default, checksum, mod-times, and size-only.)
:-Dustin
Quoted 7 lines · Theodore Ts'o
This works as long as the .config hasn't been changed so that some configuration options haven't been changed so that a driver which had been previously built as a module is now built into the kernel. In that case, you really want to make sure the no-longer applicable .ko file has been removed from the system. If the developer knows that to be true, they can use your proposed
modules_updatewithout any problems.
Quoted 5 lines
As a suggestion, something that might be worth trying would be to change to
modules_installso that it uses cp -u, but also so that it tries to delete all files that could have previously installed as modules (by using the obj-y list). This should hopefully speed upmodules_install, and make it do the right thing all the time.
Ted-
So I found the obj-y list a little tough to work with, as the objects listed there do not contain fully qualified paths describing their locations.
Here's another approach...
What you would think of some logic such that
if .config file has changed since last modules_install
call normal brute force modules_install target
else
call this new, more efficient modules_update target
I was looking through the top level Makefile for a flag or some such that would determine if a .config file has changed since last build, to no avail.
It would be pretty easy to maintain an md5sum/sha1sum signature of the .config used to perform a modules install/update (.config.md5), and compare the signature of the current .conf with the previous to determine change.
It looks like it may not be easy to drop in modules_update as a more efficient alternative to modules_install, but note that is not the patch that Kylie submitted...
I suggest that it might be sufficient to document modules_update as a handy, more efficient development build target when someone is working on one particular module-built aspect of the kernel. And clearly note that to perform a full, clean /lib/modules/uname -r/kernel synchronization with a given build tree, the more thorough modules_install target is preferred.
Thoughts?
:-Dustin
Quoted 3 lines · Dustin Kirkland
It looks like it may not be easy to drop in
modules_updateas a more efficient alternative tomodules_install, but note that is not the patch that Kylie submitted...
The problem to be solved is the long time it takes to do "make modules_install" when working on a single module. Instead of bringing in more or less complex solutions what about extending "make dir/module.ko" to include the installation of the module.
Something like:
"make MI=1 dir/module.ko"
where MI=1 tells us to install the said module.
I'm not particular found of the syntax - anyone with a better proposal?
Untested sample patch below.
Sam
Makefile +5 -0
@@ -1312,6 +1312,11 @@ # Modules $(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1) \ $(build)=$(build-dir) $(@:.ko=.o) $(Q)$(MAKE) -rR -f $(srctree)/scripts/Makefile.modpost +ifneq ($(MI),) + cp $@ $(MODLIB)/kernel/$(dir $@) + if [ -r System.map -a -x $(DEPMOD) ]; then \ + $(DEPMOD) -ae -F System.map $(depmod_opts) $(KERNELRELEASE); fi +endif # FIXME Should go into a make.lib or something # ===========================================================================
Quoted 9 lines · Sam Ravnborg
The problem to be solved is the long time it takes to do "make
modules_install" when working on a single module. Instead of bringing in more or less complex solutions what about extending "makedir/module.ko" to include the installation of the module.Something like:
"make MI=1dir/module.ko"
where MI=1 tells us to install the said module.
Um, wouldn't that imply that either (a) the compile is being done as root, or (b) the /lib/modules/* is writeable by a non-root userid? I suppose the install command could be prefixed by sudo, but that seems awkward (and not everyone uses sudo).
- Ted
Quoted 7 lines · Theodore Ts'o
Quoted 6 lines · Sam Ravnborg
The problem to be solved is the long time it takes to do "make
modules_install" when working on a single module. Instead of bringing in more or less complex solutions what about extending "makedir/module.ko" to include the installation of the module.
Quoted 8 lines
Quoted 3 lines
Something like:
"make MI=1dir/module.ko"
where MI=1 tells us to install the said module.Um, wouldn't that imply that either (a) the compile is being done as root, or (b) the
/lib/modules/*is writeable by a non-root userid? I suppose the install command could be prefixed by sudo, but that seems awkward (and not everyone uses sudo).
kbuild has support for the above scenario already - I just forgot. Say you are hacking ext3.
Do a successfull make and install all modules.
Manually remove the ext3 module from /lib/modules/... And use the external module support in kbuild like this:
# Got to relevant directory
$> cd fs/ext3
# To build the module:
$> make -C ../.. M=pwd
# To install the module:
$> make -C ../.. M=pwd modules_install
Sam