Running make install over sshfs is painful now
Dmitry Torokhov
Acked-by David RientjesHi,
With the following commit running "make modules_install install" over sshfs (and I suppose nfs) is extremely painful:
commit fb994ecc2b1c214951366c2ba5d8b121f0010d1f
Author: Greg Thelen <[email protected]> Date: Wed May 5 10:41:44 2010 -0700
kbuild: Fix checking of scm-identifier variable
I'm looking Makefile in the -mm branch (dated 2010-04-28-16-53) and seeing what looks like a bug in the checking of scm-identifier. The "ifneq ($scm-identifier)" seems to always execute "ifeq ($(LOCALVERSION,)) ...". This patch fixes the checking of scm-identifier.
I compile my kernel on my workstation and then mount working tree over sshfs and install on my laptop over wireless. It worked well for the last few years but the latest changeset slows this process down to a crawl - traversing git repository over the network is not the fastest process out there. Any cfhance we could generate the tag at compile time as opposed to install time?
Thanks.
Signature · 1 line
Dmitry
Quoted 10 lines · Dmitry Torokhov
Hi,
With the following commit running "make
modules_installinstall" over sshfs (and I suppose nfs) is extremely painful:commit
fb994ecc2b1c214951366c2ba5d8b121f0010d1fAuthor: Greg Thelen <[email protected]> Date: Wed May 5 10:41:44 2010 -0700kbuild: Fix checking of scm-identifier variable
...
Commit 85a256d8 and this one just exposed a problem we had before with CONFIG_LOCALVERSION_AUTO, namely that the Makefile computes the release string on _every_ invocation:
$ strace -feexecve make help 2>&1 | grep -v ' -1 ' | grep git
[pid 13788] execve("/usr/bin/git", ["git", "rev-parse", "--verify",
"--short", "HEAD"], [/* 88 vars */]) = 0
[pid 13790] execve("/usr/bin/git", ["git", "describe", "--exact-match"],
[/* 88 vars */]) = 0
[pid 13792] execve("/usr/bin/git", ["git", "describe"], [/* 88 vars */]) = 0
[pid 13795] execve("/usr/bin/git", ["git", "config", "--get",
"svn-remote.svn.url"], [/* 88 vars */]) = 0
[pid 13796] execve("/usr/bin/git", ["git", "update-index", "--refresh",
"--unmerged"], [/* 88 vars */]) = 0
[pid 13797] execve("/usr/bin/git", ["git", "diff-index", "--name-only",
"HEAD"], [/* 88 vars */]Process 13786 suspendedQuoted 2 lines
Any cfhance we could generate the tag at compile time as opposed to install time?
The Makefile already stores the string in include/config/kernel.release during 'make prepare', but it is nevertheless recomputed every time. Need to clean this up.
Michal
PATCH 1/1 kbuild: Clean up and speed up the localversion logic
Now that we run scripts/setlocalversion during every build, it makes sense to move all the localversion logic there. This cleans up the toplevel Makefile and also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.
---
I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.
--- Makefile | 74 +------------------ scripts/package/Makefile | 2 +- scripts/setlocalversion | 179 ++++++++++++++++++++++++++++++++++------------ 3 files changed, 136 insertions(+), 119 deletions(-)
Makefile +3 -71
@@ -883,80 +883,12 @@ PHONY += $(vmlinux-dirs) $(vmlinux-dirs): prepare scripts $(Q)$(MAKE) $(build)=$@ -# Build the kernel release string -# -# The KERNELRELEASE value built here is stored in the file -# include/config/kernel.release, and is used when executing several -# make targets, such as "make install" or "make modules_install." -# -# The eventual kernel release string consists of the following fields, -# shown in a hierarchical format to show how smaller parts are concatenated -# to form the larger and final value, with values coming from places like -# the Makefile, kernel config options, make command line options and/or -# SCM tag information. -# -# $(KERNELVERSION) -# $(VERSION) eg, 2 -# $(PATCHLEVEL) eg, 6 -# $(SUBLEVEL) eg, 18 -# $(EXTRAVERSION) eg, -rc6 -# $(localver-full) -# $(localver) -# localversion* (files without backups, containing '~') -# $(CONFIG_LOCALVERSION) (from kernel config setting) -# $(LOCALVERSION) (from make command line, if provided) -# $(localver-extra) -# $(scm-identifier) (unique SCM tag, if one exists) -# ./scripts/setlocalversion (only with CONFIG_LOCALVERSION_AUTO) -# .scmversion (only with CONFIG_LOCALVERSION_AUTO) -# + (only without CONFIG_LOCALVERSION_AUTO -# and without LOCALVERSION= and -# repository is at non-tagged commit) -# -# For kernels without CONFIG_LOCALVERSION_AUTO compiled from an SCM that has -# been revised beyond a tagged commit, `+' is appended to the version string -# when not overridden by using "make LOCALVERSION=". This indicates that the -# kernel is not a vanilla release version and has been modified. - -pattern = ".*/localversion[^~]*" -string = $(shell cat /dev/null \ - `find $(objtree) $(srctree) -maxdepth 1 -regex $(pattern) | sort -u`) - -localver = $(subst $(space),, $(string) \ - $(patsubst "%",%,$(CONFIG_LOCALVERSION))) - -# scripts/setlocalversion is called to create a unique identifier if the source -# is managed by a known SCM and the repository has been revised since the last -# tagged (release) commit. The format of the identifier is determined by the -# SCM's implementation. -# -# .scmversion is used when generating rpm packages so we do not loose -# the version information from the SCM when we do the build of the kernel -# from the copied source -ifeq ($(wildcard .scmversion),) - scm-identifier = $(shell $(CONFIG_SHELL) \ - $(srctree)/scripts/setlocalversion $(srctree)) -else - scm-identifier = $(shell cat .scmversion 2> /dev/null) -endif - -ifdef CONFIG_LOCALVERSION_AUTO - localver-extra = $(scm-identifier) -else - ifneq ($(scm-identifier),) - ifeq ($(LOCALVERSION),) - localver-extra = + - endif - endif -endif - -localver-full = $(localver)$(LOCALVERSION)$(localver-extra) - # Store (new) KERNELRELASE string in include/config/kernel.release -kernelrelease = $(KERNELVERSION)$(localver-full) +localversion = $(shell $(CONFIG_SHELL) \ + $(srctree)/scripts/setlocalversion $(srctree)) include/config/kernel.release: include/config/auto.conf FORCE $(Q)rm -f $@ - $(Q)echo $(kernelrelease) > $@ + $(Q)echo $(KERNELVERSION)$(localversion) > $@ # Things we need to do before we recursively start building the kernel
scripts/package/Makefile +1 -1
@@ -44,7 +44,7 @@ rpm-pkg rpm: $(objtree)/kernel.spec FORCE fi $(MAKE) clean $(PREV) ln -sf $(srctree) $(KERNELPATH) - $(CONFIG_SHELL) $(srctree)/scripts/setlocalversion > $(objtree)/.scmversion + $(CONFIG_SHELL) $(srctree)/scripts/setlocalversion --scm-only > $(objtree)/.scmversion $(PREV) tar -cz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/. $(PREV) rm $(KERNELPATH) rm -f $(objtree)/.scmversion
scripts/setlocalversion +132 -47
205 lines not on this page · raw message
Signature · 1 line
1.7.1
Re: [PATCH] kbuild: Clean up and speed up the localversion logic
Quoted 12 lines · Michal Marek
Now that we run
scripts/setlocalversionduring every build, it makes sense to move all the localversion logic there. This cleans up the toplevelMakefileand also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.Reported-by Dmitry Torokhov <[email protected]>Cc David Rientjes <[email protected]>Cc Greg Thelen <[email protected]>Cc Nico Schottelius <[email protected]>Signed-off-by Michal Marek <[email protected]>
Quoted 5 lines
---
I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.
I agree it would be better to move this to scripts/setlocalversion, thanks for looking into it. I'll put this into my build cycles and see if it causes any issues in my workflow that is known to be good with 2.6.35-rc3. Since this is 2.6.36 material, it seems like we have some time.
Re: [PATCH] kbuild: Clean up and speed up the localversion logic
Quoted 16 lines · David Rientjes
Quoted 14 lines · Michal Marek
Now that we run
scripts/setlocalversionduring every build, it makes sense to move all the localversion logic there. This cleans up the toplevelMakefileand also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.Reported-by Dmitry Torokhov <[email protected]>Cc David Rientjes <[email protected]>Cc Greg Thelen <[email protected]>Cc Nico Schottelius <[email protected]>Signed-off-by Michal Marek <[email protected]>---
Quoted 8 lines
Quoted 3 lines
I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.
I agree it would be better to move this to
scripts/setlocalversion, thanks for looking into it. I'll put this into my build cycles and see if it causes any issues in my workflow that is known to be good with 2.6.35-rc3. Since this is 2.6.36 material, it seems like we have some time.
No, I am sorry, it is not .36 material as the building on remote box was messed up in .35 cycle.
Signature · 1 line
Dmitry
Re: [PATCH] kbuild: Clean up and speed up the localversion logic
In reply to Michal MarekQuoted 12 lines · Michal Marek
Now that we run
scripts/setlocalversionduring every build, it makes sense to move all the localversion logic there. This cleans up the toplevelMakefileand also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.Reported-by Dmitry Torokhov <[email protected]>Cc David Rientjes <[email protected]>Cc Greg Thelen <[email protected]>Cc Nico Schottelius <[email protected]>Signed-off-by Michal Marek <[email protected]>
Quoted 5 lines
---
I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.
Seems to be working for me, I do not need to go and fetch coffee while installig newly compiled kernel anymore.
Thanks for your help Michal.
Signature · 1 line
Dmitry
Re: [PATCH] kbuild: Clean up and speed up the localversion logic
Quoted 15 lines · Dmitry Torokhov
Quoted 14 lines · Michal Marek
Now that we run
scripts/setlocalversionduring every build, it makes sense to move all the localversion logic there. This cleans up the toplevelMakefileand also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.Reported-by Dmitry Torokhov <[email protected]>Cc David Rientjes <[email protected]>Cc Greg Thelen <[email protected]>Cc Nico Schottelius <[email protected]>Signed-off-by Michal Marek <[email protected]>---
Quoted 7 lines
Quoted 3 lines
I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.
Seems to be working for me, I do not need to go and fetch coffee while installig newly compiled kernel anymore.
Michal,
Any chance this could be merged in 2.6.35? Without the patch installing over the network is not really an option anymore.
Thanks.
Signature · 1 line
Dmitry
Re: [PATCH] kbuild: Clean up and speed up the localversion logic
Quoted 20 lines · Dmitry Torokhov
Quoted 19 lines · Dmitry Torokhov
Quoted 17 lines · Michal Marek
Now that we run
scripts/setlocalversionduring every build, it makes sense to move all the localversion logic there. This cleans up the toplevelMakefileand also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.Reported-by Dmitry Torokhov <[email protected]>Cc David Rientjes <[email protected]>Cc Greg Thelen <[email protected]>Cc Nico Schottelius <[email protected]>Signed-off-by Michal Marek <[email protected]>---
I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.
Quoted 8 lines
Quoted 3 lines
Seems to be working for me, I do not need to go and fetch coffee while installig newly compiled kernel anymore.
Michal,
Any chance this could be merged in 2.6.35? Without the patch installing over the network is not really an option anymore.
I now found that my patch breaks 'make LOCALVERSION=... '. The following patch should fix it.
Michal
commit 0a564b2645c8766a669c55bde1f1ef5b0518caec
Author: Michal Marek <[email protected]> Date: Wed Jun 30 16:41:23 2010 +0200
kbuild: Propagate LOCALVERSION= down to scripts/setlocalversion
Variables given on the make commandline are not exported to $(shell ...) commands, so run the setlocalversion script in the make rule directly.
Makefile +1 -3
@@ -884,11 +884,9 @@ $(vmlinux-dirs): prepare scripts $(Q)$(MAKE) $(build)=$@ # Store (new) KERNELRELASE string in include/config/kernel.release -localversion = $(shell $(CONFIG_SHELL) \ - $(srctree)/scripts/setlocalversion $(srctree)) include/config/kernel.release: include/config/auto.conf FORCE $(Q)rm -f $@ - $(Q)echo $(KERNELVERSION)$(localversion) > $@ + $(Q)echo "$(KERNELVERSION)$$($(CONFIG_SHELL) scripts/setlocalversion $(srctree))" > $@ # Things we need to do before we recursively start building the kernel
Re: [PATCH] kbuild: Clean up and speed up the localversion logic
In reply to Michal MarekMichal Marek [Thu, Jun 17, 2010 at 03:40:38PM +0200]:
Quoted 6 lines
Now that we run
scripts/setlocalversionduring every build, it makes sense to move all the localversion logic there. This cleans up the toplevelMakefileand also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.
Sounds good in general.
scripts/setlocalversion line 15
Quoted 10 lines
scripts/setlocalversion +1
-1@@ -10,73 +10,158 @@ # usage() { - echo "Usage: $0 [srctree]" >&2 + echo "Usage: $0 [--scm-only] [srctree]" >&2 exit 1 }
If more stuff like this is coming, we should probably rewrite option handling. No need to change right now, but to keep in mind.
scripts/setlocalversion line 24
Quoted 9 lines
scripts/setlocalversion +8
-1-cd "${1:-.}" || usage +scm_only=false +srctree=. +if test "$1" = "--scm-only"; then + scm_only=true + shift +fi +if test $# -gt 0; then + srctree=$1
You probably want
srctree="$1"
to catch spaces and co. in the path.
Otherwise I just had a quick view over it, but seems to look good.
Nico
Signature · 4 lines
New PGP key: 7ED9 F7D3 6B10 81D7 0EC5 5C09 D7DC C8E4 3187 7DF0 Please resign, if you signed 9885188C or 8D0E27A4. Currently moving *.schottelius.org to http://www.nico.schottelius.org/ ...
Re: [PATCH] kbuild: Clean up and speed up the localversion logic
Quoted 13 lines · Nico Schottelius
Michal Marek [Thu, Jun 17, 2010 at 03:40:38PM +0200]:
scripts/setlocalversion line 24
Quoted 9 lines
scripts/setlocalversion +8
-1-cd "${1:-.}" || usage +scm_only=false +srctree=. +if test "$1" = "--scm-only"; then + scm_only=true + shift +fi +if test $# -gt 0; then + srctree=$1You probably want
Quoted 3 lines
srctree="$1"
to catch spaces and co. in the path.
That's the same, the value in variable assignment is not subject to word splitting.
Quoted 1 line
Otherwise I just had a quick view over it, but seems to look good.
Thanks!
/me is now wondering if it's ok to push this for 2.6.35. At least, there were no bugreports from linux-next users so far.
Michal
Re: [PATCH] kbuild: Clean up and speed up the localversion logic
Michal Marek [Tue, Jun 22, 2010 at 01:13:55PM +0200]:
Quoted 8 lines · Nico Schottelius
Quoted 4 lines
Michal Marek [Thu, Jun 17, 2010 at 03:40:38PM +0200]:
srctree="$1"to catch spaces and co. in the path.
That's the same, the value in variable assignment is not subject to word splitting.
Oh, indeed, you're right! Learned something!
Quoted 6 lines
Quoted 1 line
Otherwise I just had a quick view over it, but seems to look good.
Thanks!
/me is now wondering if it's ok to push this for 2.6.35. At least, there were no bugreports from linux-next users so far.
Well, sometime you'll have to push it and I'm not sure how many "no bugreports" you'll want to wait for :-)
Cheers,
Nico
Signature · 4 lines
New PGP key: 7ED9 F7D3 6B10 81D7 0EC5 5C09 D7DC C8E4 3187 7DF0 Please resign, if you signed 9885188C or 8D0E27A4. Currently moving *.schottelius.org to http://www.nico.schottelius.org/ ...