kbuild changes for v3.11-rc1

Hi Linus,

please pull the kbuild bits for v3.11-rc1:

- fix for make headers_install argv explosion with too long path
- scripts/setlocalversion does not call git update-index needlessly
- fix for the src.rpm produced by make rpm-pkg. The new make image_name can be useful also for other packaging tools.
- scripts/mod/devicetable-offsets.o is not rebuilt during each make run
- make modules_install dependency fix
- scripts/sortextable portability fix
- fix for kbuild to generate the output directory for all object files in subdirs.
- a couple of minor fixes

Michal

The following changes since commit f722406faae2d073cc1d01063d1123c35425939e:

Linux 3.10-rc1 (2013-05-11 17:14:08 -0700)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild.git kbuild

for you to fetch changes up to 4d47dde47f7dd95042fa56283d948f50dd4b509c:

kbuild: create directory for dir/file.o (2013-07-03 22:47:08 +0200)

----------------------------------------------------------------
Christian Kujau (1):
scripts/setlocalversion on write-protected source tree

Daniel Tang (1):
Fix a build warning in scripts/mod/file2alias.c

Geert Uytterhoeven (1):
scripts/mod: Spelling s/DEVICEVTABLE/DEVICETABLE/

James Hogan (1):
Makefile.lib: align DTB quiet_cmd

Jan Beulich (1):
improve modalias building

Mike Marciniszyn (1):
kbuild: fix error when building from src rpm

Nicolas Dichtel (1):
kbuild: fix make headers_install when path is too long

Robert Richter (1):
Makefile: Fix install error with make -j option

Yaakov Selkowitz (1):
tools/include: use stdint types for user-space byteshift headers

张忠山 (1):
kbuild: create directory for dir/file.o

 Makefile                           |  8 ++++++--
 scripts/Makefile.headersinst       | 20 ++++++++++++++------
 scripts/Makefile.lib               |  4 ++--
 scripts/headers_install.sh         |  7 +++++--
 scripts/mod/Makefile               | 15 +++++----------
 scripts/mod/file2alias.c           | 10 ++++++----
 scripts/package/mkspec             |  1 +
 scripts/setlocalversion            |  3 ---
 tools/include/tools/be_byteshift.h | 34 +++++++++++++++++-----------------
 tools/include/tools/le_byteshift.h | 34 +++++++++++++++++-----------------
 10 files changed, 73 insertions(+), 63 deletions(-)

THIS IS SOME HORRIBLY BROKEN CRAP.

"make install" and "make modules_install" ABSOLUTELY MUST NOT MODIFY THE SOURCE TREE.

Dammit, this has happened before, and it was broken then, and it is broken now.

If they do, they are *F*CKING*BROKEN*. They are really really badly broken, since we do *not* want root to write to the source tree. You should build the tree as a normal user, and install as root, and dammit, if there are any root-owned files in the source tree after that "make install", then the build system is broken.

You need to start being more careful. And I would seriously suggest you start doing some explicit testing for this. You can do things like "find . -user root", and if that shows a single file in the kernel tree after a "make [modules_]install", then there's a problem.

Commit d2aae8477cd00325bb7c7c7e95be488088900c48 is broken. It causes root to re-write "include/config/kernel.release".

There is no excuse for this. That commit is shit. There's no way in hell that "make modules_install" should ever rebuild anything, so adding that kind of dependency is fundamentally wrong and broken.

And that totally crap commit is even marked for stable.

I hate hate hate when this kind of crap happens. In this case I noticed it because the git commit abbreviation rules are different for root and for a normal user on my machine, and so running that version-generation script as root actually GIVES THE WRONG ANSWER - it gives a different version than the one the kernel was actually built with.

So no. We do *not* start adding random dependencies to the install targets. Because they damn well should not be building anything.

Linus

It's now dropped from my stable to-apply queue.

greg k-h

Good idea. I added this after the *install calls in my test script.

Yeah. It also reveals another bug that we rewrite the kernel.release file each time. This patch should fix it, but please do not apply it yet:

From f522eef24216f229c82b6bbb55f326814db2bac0 Mon Sep 17 00:00:00 2001
From: Michal Marek <[email protected]>
Date: Thu, 11 Jul 2013 15:34:51 +0200
Subject: [PATCH] kbuild: Do not overwrite include/config/kernel.release
needlessly

Use filechk to detect if the content changed or not.

index 4e3575c..3aa232f 100644
@@ -794,10 +794,13 @@ PHONY += $(vmlinux-dirs)
 $(vmlinux-dirs): prepare scripts
 	$(Q)$(MAKE) $(build)=$@
 
+define filechk_kernel.release
+	echo "$(KERNELVERSION)$$($(CONFIG_SHELL) $(srctree)/scripts/setlocalversion $(srctree))"
+endef
+
 # Store (new) KERNELRELEASE string in include/config/kernel.release
 include/config/kernel.release: include/config/auto.conf FORCE
-	$(Q)rm -f $@
-	$(Q)echo "$(KERNELVERSION)$$($(CONFIG_SHELL) $(srctree)/scripts/setlocalversion $(srctree))" > $@
+	$(call filechk,kernel.release)
 
 
 # Things we need to do before we recursively start building the kernel

We also run dozens of gcc checks during every make invocation, including make install or make help, so there is some room for improvement. But at least, these do not write to the source or object tree.

Michal

So I think not rewriting the file is a good thing, but in this case it would actually have hidden the real bug.

At least for most people. For me, it would have rewritten the file anyway, because as a normal user I have

core.abbrev=12

in my ~/.gitconfig, but when running it as root, it will take the abbrev setting from the git defaults (which I think is 7). So the content actually *changes* if root generates the version as opposed to my normal user account.

Of course, that's arguably something we could fix in our localversion script (by just forcing a particular abbreviation length), but I actually like how a plain "git describe" (which will use the user-specified SHA1 abbreviations) matches "uname -r" (which will obviously take the version from the kernel version file at build-time). So I'm not convinced that it is wrong for us to just use the format that the user specified in their gitconfig, even if it does result in "odd" situations like this where "make kernelrelease" will then potentially give different output for different users.

It would be good to try to minimize these kinds of things, for the simple reason that I'm not at all sure that we necessarily get all the temporary file security issues right.

Running as little as possible as root is good practice for a very real
reason: there can be very subtle security issues. Let's not worry about it *too* much, but keeping it in mind is good.

At least for the gcc checking scripts I looked at, we seem to be doing things safely (ie using pipes instead of temp-files in /tmp).

Linus

The odd thing I have in a specific configuration is that depmod is missing the kernelrelease when I only build install rules, though I had built everything before and kernelrelease should be there:

 $ make <makeflags>
 $ make <makeflags> modules_install zinstall
 ...
   DEPMOD  
 Usage: .../scripts/depmod.sh /sbin/depmod <kernelrelease>

Note that makeflags include the -j option, INSTALL_MOD_PATH and INSTALL_PATH variables set, so no root perms required in this case.

I agree, the fix to add a dependency to the kernelrelease file is wrong here since it rewrites the object tree, which should be readonly for install rules.

I could reproduce the error above in my setup and will look at it again, hopefully fixing it more carefully.

Appologize the trouble this made.

Thanks,

-Robert

(trimmed cc list)

I narrowed this down. The problem is that zinstall on ARCH=arm has a dependency to vmlinux which does a prepare/prepare3 and finally does a forced rebuild of kernel.release even if it exists already.

Rebuilding it removes kernel.release first and then recreates it. This might race with another parallel make job running depmod.

So on arm and maybe other archs we need the same as for x86:

1648e4f8 x86, kbuild: make "make install" not depend on vmlinux

The patch below fixes this for arm. It is build-tested in my environment, but please test it in others too if possible.

Thanks,

-Robert

From: Robert Richter <[email protected]>
Date: Wed, 17 Jul 2013 16:17:20 +0200
Subject: [PATCH] arm, kbuild: make "make install" not depend on vmlinux

Install targets (install, zinstall, uinstall) on arm have a dependency to vmlinux. This may cause parts of the kernel to be rebuilt during installation. We must avoid this since this may run as root. Install targets "ABSOLUTELY MUST NOT MODIFY THE SOURCE TREE." as Linus emphasized this in:

http://lkml.org/lkml/2013/7/10/600

So on arm and maybe other archs we need the same as for x86:

1648e4f8 x86, kbuild: make "make install" not depend on vmlinux

This patch fixes this for arm. Dependencies are removed and instead a check to install.sh is added for the files that are needed.

This issue was uncovered by this build error where the -j option is used in conjunction with install targets:

 $ make <makeflags>
 $ make <makeflags> zinstall
 ...
   DEPMOD
 Usage: .../scripts/depmod.sh /sbin/depmod <kernelrelease>

(INSTALL_MOD_PATH and INSTALL_PATH variables set, so no root perms required in this case.)

The problem is that zinstall on arm due to its dependency to vmlinux does a prepare/prepare3 and finally does a forced rewrite of kernel.release even if it exists already.

Rebuilding kernel.release removes it first and then recreates it. This might race with another parallel make job running depmod.

So this patch should fix this one too.

Also quoting $(KERNELRELEASE) arg for install.sh as this messes argument order in case it is empty (which is the case if the kernel was not built yet).

---
 arch/arm/Makefile        |    9 +++++++--
 arch/arm/boot/Makefile   |   16 ++++++++--------
 arch/arm/boot/install.sh |   14 ++++++++++++++
 3 files changed, 29 insertions(+), 10 deletions(-)

index c0ac0f5..fe63986 100644
@@ -295,10 +295,15 @@ archprepare:
 # Convert bzImage to zImage
 bzImage: zImage
 
-zImage Image xipImage bootpImage uImage: vmlinux
+BOOT_TARGETS	= zImage Image xipImage bootpImage uImage
+INSTALL_TARGETS	= zinstall uinstall install
+
+PHONY += bzImage $(BOOT_TARGETS) $(INSTALL_TARGETS)
+
+$(BOOT_TARGETS): vmlinux
 	$(Q)$(MAKE) $(build)=$(boot) MACHINE=$(MACHINE) $(boot)/$@
 
-zinstall uinstall install: vmlinux
+$(INSTALL_TARGETS):
 	$(Q)$(MAKE) $(build)=$(boot) MACHINE=$(MACHINE) $@
 
 %.dtb: | scripts

index 84aa2ca..ec2f806 100644
@@ -95,24 +95,24 @@ initrd:
 	@test "$(INITRD)" != "" || \
 	(echo You must specify INITRD; exit -1)
 
-install: $(obj)/Image
-	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
+install:
+	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh "$(KERNELRELEASE)" \
 	$(obj)/Image System.map "$(INSTALL_PATH)"
 
-zinstall: $(obj)/zImage
-	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
+zinstall:
+	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh "$(KERNELRELEASE)" \
 	$(obj)/zImage System.map "$(INSTALL_PATH)"
 
-uinstall: $(obj)/uImage
-	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
+uinstall:
+	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh "$(KERNELRELEASE)" \
 	$(obj)/uImage System.map "$(INSTALL_PATH)"
 
 zi:
-	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
+	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh "$(KERNELRELEASE)" \
 	$(obj)/zImage System.map "$(INSTALL_PATH)"
 
 i:
-	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
+	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh "$(KERNELRELEASE)" \
 	$(obj)/Image System.map "$(INSTALL_PATH)"
 
 subdir-	    := bootp compressed dts

index 06ea7d4..2a45092 100644
@@ -20,6 +20,20 @@
 #   $4 - default install path (blank if root directory)
 #
 
+verify () {
+	if [ ! -f "$1" ]; then
+		echo ""                                                   1>&2
+		echo " *** Missing file: $1"                              1>&2
+		echo ' *** You need to run "make" before "make install".' 1>&2
+		echo ""                                                   1>&2
+		exit 1
+	fi
+}
+
+# Make sure the files actually exist
+verify "$2"
+verify "$3"
+
 # User may have a custom install script
 if [ -x ~/bin/${INSTALLKERNEL} ]; then exec ~/bin/${INSTALLKERNEL} "$@"; fi
 if [ -x /sbin/${INSTALLKERNEL} ]; then exec /sbin/${INSTALLKERNEL} "$@"; fi

1.7.7.6

Ack. This looks like the right thing to do.

That said, I think for clarity you might perhaps want to make the unrelated change of adding double quotes around $(KERNELRELEASE) a separate patch. Not a big deal, but it took me a moment to say "why did he do that" (in fact, I'm not sure KERNEL_RELEASE can actually validly have spaces in it, but quoting it is certainly not wrong either - but it might be unnecessary.

Linus

Thanks for looking at this.

If kernel.release doen't exist KERNEL_RELEASE is empty, thus all args for install.sh shift one arg left. Putting it in quotes avoids this. Noticed this since the first verified file in install.sh ($2) should be the image file but was already System.map ($3).

Will splitt the patches and resend.

-Robert

You know what? I read the patch, but I didn't read your changelog well enough. You already explained that in the commit log, and I just missed it like some blind monkey. My bad.

No need to split, I was wrong, it wasn't some unrelated cleanup, and you were right the first time.

So Michal (or ARM people - whoever wants to take the patch), just take my ack. No objections.

Linus

I can add it to the kbuild tree if needed. Otherwise you can add

Michal

This didn't make it upstream yet, can somebody at it to a tree?

Thanks,

-Robert

Robert, All,

On 2013-09-30 10:49 +0200, Robert Richter spake thusly:

Since it's been acked-by Linus, I'll queue it in my tree, for Michal to pull from. Expect a pull-request soon.

Regards,
Yann E. MORIN.

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Sorry for chiming in that late, but I didn't think of this when reading the original submission.

Just doing "make oldconfig; make install" used to work.
Removing the dependency of "make vmlinux" on vmlinux breaks this, doesn't it?

I had the habit of doing the above many years ago, when I was mostly doing native builds, and before I had my own custom linux-install-kernel script that e.g.
knows how to copy kernels and modules around to NFS servers.

Not that I'm strongly attached to it, but there may be other users...

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

On ARM and maybe other architectures.

Yes.

We can't eat the cake and have it :). What can be done is to make arch/arm/boot/install.sh print a friendlier error message, like the x86 version does:

if [ ! -f "$1" ]; then
echo "" 1>&2
echo " *** Missing file: $1" 1>&2
echo ' *** You need to run "make" before "make install".' 1>&2
echo "" 1>&2
exit 1
fi

Michal

Until 2009 it also worked on x86 ;-)

Sure.

Definitely!

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds