Skip to contentSkip to conversation

Running make install over sshfs is painful now

PATCH In review Dmitry Torokhov 4 participants 11 messages Jun 7, 2010 – Jul 2, 2010 Kbuild

lore
Share & tools
lore

Dmitry Torokhov

author Acked-by David Rientjes
#0 lore raw

Hi,

With the following commit running "make modules_install install" over sshfs (and I suppose nfs) is extremely painful:

commit fb994ecc2b1c214951366c2ba5d8b121f0010d1f

Author: Greg Thelen <[email protected]>
Date:   Wed May 5 10:41:44 2010 -0700

kbuild: Fix checking of scm-identifier variable

I'm looking Makefile in the -mm branch (dated 2010-04-28-16-53) and seeing what looks like a bug in the checking of scm-identifier. The "ifneq ($scm-identifier)" seems to always execute "ifeq ($(LOCALVERSION,)) ...". This patch fixes the checking of scm-identifier.

I compile my kernel on my workstation and then mount working tree over sshfs and install on my laptop over wireless. It worked well for the last few years but the latest changeset slows this process down to a crawl - traversing git repository over the network is not the fastest process out there. Any cfhance we could generate the tag at compile time as opposed to install time?

Thanks.

Signature · 1 line
Dmitry
#1 lore raw
Quoted 10 lines · Dmitry Torokhov

Hi,

With the following commit running "make modules_install install" over sshfs (and I suppose nfs) is extremely painful:

commit fb994ecc2b1c214951366c2ba5d8b121f0010d1f

Author: Greg Thelen <[email protected]>
Date:   Wed May 5 10:41:44 2010 -0700

kbuild: Fix checking of scm-identifier variable

...

Commit 85a256d8 and this one just exposed a problem we had before with CONFIG_LOCALVERSION_AUTO, namely that the Makefile computes the release string on _every_ invocation:

$ strace -feexecve make help 2>&1 | grep -v ' -1 ' | grep git
[pid 13788] execve("/usr/bin/git", ["git", "rev-parse", "--verify",
"--short", "HEAD"], [/* 88 vars */]) = 0
[pid 13790] execve("/usr/bin/git", ["git", "describe", "--exact-match"],
[/* 88 vars */]) = 0
[pid 13792] execve("/usr/bin/git", ["git", "describe"], [/* 88 vars */]) = 0
[pid 13795] execve("/usr/bin/git", ["git", "config", "--get",
"svn-remote.svn.url"], [/* 88 vars */]) = 0
[pid 13796] execve("/usr/bin/git", ["git", "update-index", "--refresh",
"--unmerged"], [/* 88 vars */]) = 0
[pid 13797] execve("/usr/bin/git", ["git", "diff-index", "--name-only",
"HEAD"], [/* 88 vars */]Process 13786 suspended
Quoted 2 lines

Any cfhance we could generate the tag at compile time as opposed to install time?

The Makefile already stores the string in include/config/kernel.release during 'make prepare', but it is nevertheless recomputed every time. Need to clean this up.

Michal

#2 lore raw

PATCH 1/1 kbuild: Clean up and speed up the localversion logic

Now that we run scripts/setlocalversion during every build, it makes sense to move all the localversion logic there. This cleans up the toplevel Makefile and also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.

Cc David Rientjes <[email protected]>
Cc Greg Thelen <[email protected]>
Cc Nico Schottelius <[email protected]>

---

I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.

---
 Makefile                 |   74 +------------------
 scripts/package/Makefile |    2 +-
 scripts/setlocalversion  |  179 ++++++++++++++++++++++++++++++++++------------
 3 files changed, 136 insertions(+), 119 deletions(-)
Makefile +3 -71
index d49d96c..f6d2cb2 100644
@@ -883,80 +883,12 @@ PHONY += $(vmlinux-dirs)
 $(vmlinux-dirs): prepare scripts
 	$(Q)$(MAKE) $(build)=$@
 
-# Build the kernel release string
-#
-# The KERNELRELEASE value built here is stored in the file
-# include/config/kernel.release, and is used when executing several
-# make targets, such as "make install" or "make modules_install."
-#
-# The eventual kernel release string consists of the following fields,
-# shown in a hierarchical format to show how smaller parts are concatenated
-# to form the larger and final value, with values coming from places like
-# the Makefile, kernel config options, make command line options and/or
-# SCM tag information.
-#
-#	$(KERNELVERSION)
-#	  $(VERSION)			eg, 2
-#	  $(PATCHLEVEL)			eg, 6
-#	  $(SUBLEVEL)			eg, 18
-#	  $(EXTRAVERSION)		eg, -rc6
-#	$(localver-full)
-#	  $(localver)
-#	    localversion*		(files without backups, containing '~')
-#	    $(CONFIG_LOCALVERSION)	(from kernel config setting)
-#	  $(LOCALVERSION)		(from make command line, if provided)
-#	  $(localver-extra)
-#	    $(scm-identifier)		(unique SCM tag, if one exists)
-#	      ./scripts/setlocalversion	(only with CONFIG_LOCALVERSION_AUTO)
-#	      .scmversion		(only with CONFIG_LOCALVERSION_AUTO)
-#	    +				(only without CONFIG_LOCALVERSION_AUTO
-#					 and without LOCALVERSION= and
-#					 repository is at non-tagged commit)
-#
-# For kernels without CONFIG_LOCALVERSION_AUTO compiled from an SCM that has
-# been revised beyond a tagged commit, `+' is appended to the version string
-# when not overridden by using "make LOCALVERSION=".  This indicates that the
-# kernel is not a vanilla release version and has been modified.
-
-pattern = ".*/localversion[^~]*"
-string  = $(shell cat /dev/null \
-	   `find $(objtree) $(srctree) -maxdepth 1 -regex $(pattern) | sort -u`)
-
-localver = $(subst $(space),, $(string) \
-			      $(patsubst "%",%,$(CONFIG_LOCALVERSION)))
-
-# scripts/setlocalversion is called to create a unique identifier if the source
-# is managed by a known SCM and the repository has been revised since the last
-# tagged (release) commit.  The format of the identifier is determined by the
-# SCM's implementation.
-#
-# .scmversion is used when generating rpm packages so we do not loose
-# the version information from the SCM when we do the build of the kernel
-# from the copied source
-ifeq ($(wildcard .scmversion),)
-        scm-identifier = $(shell $(CONFIG_SHELL) \
-                         $(srctree)/scripts/setlocalversion $(srctree))
-else
-        scm-identifier = $(shell cat .scmversion 2> /dev/null)
-endif
-
-ifdef CONFIG_LOCALVERSION_AUTO
-	localver-extra = $(scm-identifier)
-else
-	ifneq ($(scm-identifier),)
-		ifeq ($(LOCALVERSION),)
-			localver-extra = +
-		endif
-	endif
-endif
-
-localver-full = $(localver)$(LOCALVERSION)$(localver-extra)
-
 # Store (new) KERNELRELASE string in include/config/kernel.release
-kernelrelease = $(KERNELVERSION)$(localver-full)
+localversion = $(shell $(CONFIG_SHELL) \
+	       $(srctree)/scripts/setlocalversion $(srctree))
 include/config/kernel.release: include/config/auto.conf FORCE
 	$(Q)rm -f $@
-	$(Q)echo $(kernelrelease) > $@
+	$(Q)echo $(KERNELVERSION)$(localversion) > $@
 
 
 # Things we need to do before we recursively start building the kernel
scripts/package/Makefile +1 -1
index 62fcc3a..3a681ef 100644
@@ -44,7 +44,7 @@ rpm-pkg rpm: $(objtree)/kernel.spec FORCE
 	fi
 	$(MAKE) clean
 	$(PREV) ln -sf $(srctree) $(KERNELPATH)
-	$(CONFIG_SHELL) $(srctree)/scripts/setlocalversion > $(objtree)/.scmversion
+	$(CONFIG_SHELL) $(srctree)/scripts/setlocalversion --scm-only > $(objtree)/.scmversion
 	$(PREV) tar -cz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
 	$(PREV) rm $(KERNELPATH)
 	rm -f $(objtree)/.scmversion
scripts/setlocalversion +132 -47

205 lines not on this page · raw message

Signature · 1 line
1.7.1
#3 lore raw

Re: [PATCH] kbuild: Clean up and speed up the localversion logic

Quoted 12 lines · Michal Marek

Now that we run scripts/setlocalversion during every build, it makes sense to move all the localversion logic there. This cleans up the toplevel Makefile and also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.

Cc David Rientjes <[email protected]>
Cc Greg Thelen <[email protected]>
Cc Nico Schottelius <[email protected]>
Quoted 5 lines

---

I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.

I agree it would be better to move this to scripts/setlocalversion, thanks for looking into it. I'll put this into my build cycles and see if it causes any issues in my workflow that is known to be good with 2.6.35-rc3. Since this is 2.6.36 material, it seems like we have some time.

#4 lore raw

Re: [PATCH] kbuild: Clean up and speed up the localversion logic

Quoted 16 lines · David Rientjes
Quoted 14 lines · Michal Marek

Now that we run scripts/setlocalversion during every build, it makes sense to move all the localversion logic there. This cleans up the toplevel Makefile and also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.

Cc David Rientjes <[email protected]>
Cc Greg Thelen <[email protected]>
Cc Nico Schottelius <[email protected]>

---

Quoted 8 lines
Quoted 3 lines

I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.

I agree it would be better to move this to scripts/setlocalversion, thanks for looking into it. I'll put this into my build cycles and see if it causes any issues in my workflow that is known to be good with 2.6.35-rc3. Since this is 2.6.36 material, it seems like we have some time.

No, I am sorry, it is not .36 material as the building on remote box was messed up in .35 cycle.

Signature · 1 line
Dmitry
#5 lore raw

Re: [PATCH] kbuild: Clean up and speed up the localversion logic

In reply to Michal Marek
Quoted 12 lines · Michal Marek

Now that we run scripts/setlocalversion during every build, it makes sense to move all the localversion logic there. This cleans up the toplevel Makefile and also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.

Cc David Rientjes <[email protected]>
Cc Greg Thelen <[email protected]>
Cc Nico Schottelius <[email protected]>
Quoted 5 lines

---

I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.

Seems to be working for me, I do not need to go and fetch coffee while installig newly compiled kernel anymore.

Thanks for your help Michal.

Signature · 1 line
Dmitry
#6 lore raw

Re: [PATCH] kbuild: Clean up and speed up the localversion logic

Quoted 15 lines · Dmitry Torokhov
Quoted 14 lines · Michal Marek

Now that we run scripts/setlocalversion during every build, it makes sense to move all the localversion logic there. This cleans up the toplevel Makefile and also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.

Cc David Rientjes <[email protected]>
Cc Greg Thelen <[email protected]>
Cc Nico Schottelius <[email protected]>

---

Quoted 7 lines
Quoted 3 lines

I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.

Seems to be working for me, I do not need to go and fetch coffee while installig newly compiled kernel anymore.

Michal,

Any chance this could be merged in 2.6.35? Without the patch installing over the network is not really an option anymore.

Thanks.

Signature · 1 line
Dmitry
#7 lore raw

Re: [PATCH] kbuild: Clean up and speed up the localversion logic

Quoted 20 lines · Dmitry Torokhov
Quoted 19 lines · Dmitry Torokhov
Quoted 17 lines · Michal Marek

Now that we run scripts/setlocalversion during every build, it makes sense to move all the localversion logic there. This cleans up the toplevel Makefile and also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.

Cc David Rientjes <[email protected]>
Cc Greg Thelen <[email protected]>
Cc Nico Schottelius <[email protected]>

---

I tried to test this in various scenarios, but if anyone of you could give it a try, that would be great. The patch is against 2.6.35-rc3.

Quoted 8 lines
Quoted 3 lines

Seems to be working for me, I do not need to go and fetch coffee while installig newly compiled kernel anymore.

Michal,

Any chance this could be merged in 2.6.35? Without the patch installing over the network is not really an option anymore.

I now found that my patch breaks 'make LOCALVERSION=... '. The following patch should fix it.

Michal

commit 0a564b2645c8766a669c55bde1f1ef5b0518caec

Author: Michal Marek <[email protected]>
Date:   Wed Jun 30 16:41:23 2010 +0200

kbuild: Propagate LOCALVERSION= down to scripts/setlocalversion

Variables given on the make commandline are not exported to $(shell ...) commands, so run the setlocalversion script in the make rule directly.

Makefile +1 -3
index a86ac8c..12ab175 100644
@@ -884,11 +884,9 @@ $(vmlinux-dirs): prepare scripts
 	$(Q)$(MAKE) $(build)=$@
 
 # Store (new) KERNELRELASE string in include/config/kernel.release
-localversion = $(shell $(CONFIG_SHELL) \
-	       $(srctree)/scripts/setlocalversion $(srctree))
 include/config/kernel.release: include/config/auto.conf FORCE
 	$(Q)rm -f $@
-	$(Q)echo $(KERNELVERSION)$(localversion) > $@
+	$(Q)echo "$(KERNELVERSION)$$($(CONFIG_SHELL) scripts/setlocalversion $(srctree))" > $@
 
 
 # Things we need to do before we recursively start building the kernel
#8 lore raw

Re: [PATCH] kbuild: Clean up and speed up the localversion logic

In reply to Michal Marek

Michal Marek [Thu, Jun 17, 2010 at 03:40:38PM +0200]:

Quoted 6 lines

Now that we run scripts/setlocalversion during every build, it makes sense to move all the localversion logic there. This cleans up the toplevel Makefile and also makes sure that the script is called only once in 'make prepare' (previously, it would be called every time due to a variable expansion in an ifneq statement). No user-visible change is intended, unless one runs the setlocalversion script directly.

Sounds good in general.

scripts/setlocalversion line 15

Quoted 10 lines
scripts/setlocalversion +1 -1
@@ -10,73 +10,158 @@
 #
 
 usage() {
-	echo "Usage: $0 [srctree]" >&2
+	echo "Usage: $0 [--scm-only] [srctree]" >&2
 	exit 1
 }

If more stuff like this is coming, we should probably rewrite option handling. No need to change right now, but to keep in mind.

scripts/setlocalversion line 24

Quoted 9 lines
scripts/setlocalversion +8 -1
-cd "${1:-.}" || usage
+scm_only=false
+srctree=.
+if test "$1" = "--scm-only"; then
+	scm_only=true
+	shift
+fi
+if test $# -gt 0; then
+	srctree=$1

You probably want

srctree="$1"

to catch spaces and co. in the path.

Otherwise I just had a quick view over it, but seems to look good.

Nico

Signature · 4 lines
New PGP key: 7ED9 F7D3 6B10 81D7 0EC5  5C09 D7DC C8E4 3187 7DF0
Please resign, if you signed 9885188C or 8D0E27A4.

Currently moving *.schottelius.org to http://www.nico.schottelius.org/ ...
#9 lore raw

Re: [PATCH] kbuild: Clean up and speed up the localversion logic

Quoted 13 lines · Nico Schottelius

Michal Marek [Thu, Jun 17, 2010 at 03:40:38PM +0200]:

scripts/setlocalversion line 24

Quoted 9 lines
scripts/setlocalversion +8 -1
-cd "${1:-.}" || usage
+scm_only=false
+srctree=.
+if test "$1" = "--scm-only"; then
+	scm_only=true
+	shift
+fi
+if test $# -gt 0; then
+	srctree=$1

You probably want

Quoted 3 lines

srctree="$1"

to catch spaces and co. in the path.

That's the same, the value in variable assignment is not subject to word splitting.

Quoted 1 line

Otherwise I just had a quick view over it, but seems to look good.

Thanks!

/me is now wondering if it's ok to push this for 2.6.35. At least, there were no bugreports from linux-next users so far.

Michal

#10 lore raw

Re: [PATCH] kbuild: Clean up and speed up the localversion logic

Michal Marek [Tue, Jun 22, 2010 at 01:13:55PM +0200]:

Quoted 8 lines · Nico Schottelius
Quoted 4 lines

Michal Marek [Thu, Jun 17, 2010 at 03:40:38PM +0200]:
srctree="$1"

to catch spaces and co. in the path.

That's the same, the value in variable assignment is not subject to word splitting.

Oh, indeed, you're right! Learned something!

Quoted 6 lines
Quoted 1 line

Otherwise I just had a quick view over it, but seems to look good.

Thanks!

/me is now wondering if it's ok to push this for 2.6.35. At least, there were no bugreports from linux-next users so far.

Well, sometime you'll have to push it and I'm not sure how many "no bugreports" you'll want to wait for :-)

Cheers,

Nico

Signature · 4 lines
New PGP key: 7ED9 F7D3 6B10 81D7 0EC5  5C09 D7DC C8E4 3187 7DF0
Please resign, if you signed 9885188C or 8D0E27A4.

Currently moving *.schottelius.org to http://www.nico.schottelius.org/ ...