Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed
PATCH 1/1 Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed
If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.
Avoid touching config_data.gz by using filechk to compare the existing config_data.gz and update only if it changed.
The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.
For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.
--- kernel/Makefile | 2 +- scripts/Makefile.lib | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
kernel/Makefile +1 -1
@@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz targets += config_data.gz $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE - $(call if_changed,gzip) + $(call filechk,gzip) $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
scripts/Makefile.lib +2 -0
@@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@ quiet_cmd_gzip = GZIP $@ cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@ +filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 + # DTC # --------------------------------------------------------------------------- DTC ?= $(objtree)/scripts/dtc/dtc
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 31 lines · Elliot Berman
If you update the timestamp of
KCONFIG_CONFIGwithout actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.Avoid touching config_data.gz by using filechk to compare the existing config_data.gz and update only if it changed.
The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.
For a simple example on my x86 machine, I modified x86 default defconfig to set
CONFIG_IKCONFIG=yand run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.Signed-off-by Elliot Berman <[email protected]>--- kernel/Makefile | 2 +- scripts/Makefile.lib | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)kernel/Makefile +0
-0
kernel/Makefile $(obj)/configs.o: $(obj)/config_data.gz · line 143
Quoted 5 lines
kernel/Makefile +1
-1@@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz targets += config_data.gz $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE - $(call if_changed,gzip) + $(call filechk,gzip)
I do not think this is the right approach
because gzip is executed every time, even
if the time stamp is not changed.
Quoted 20 lines
kernel/Makefile +0
-0$(obj)/kheaders.o: $(obj)/kheaders_data.tar.xzscripts/Makefile.lib +2
-1@@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@ quiet_cmd_gzip = GZIP $@ cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@ +filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 + # DTC # --------------------------------------------------------------------------- DTC ?= $(objtree)/scripts/dtc/dtc --The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 33 lines · Masahiro Yamada
Quoted 32 lines · Elliot Berman
If you update the timestamp of
KCONFIG_CONFIGwithout actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.Avoid touching config_data.gz by using filechk to compare the existing config_data.gz and update only if it changed.
The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.
For a simple example on my x86 machine, I modified x86 default defconfig to set
CONFIG_IKCONFIG=yand run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.Signed-off-by Elliot Berman <[email protected]>--- kernel/Makefile | 2 +- scripts/Makefile.lib | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)kernel/Makefile +0
-0@@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz
Quoted 10 lines
kernel/Makefile $(obj)/configs.o: $(obj)/config_data.gz · line 143
Quoted 4 lines
kernel/Makefile +1
-1targets += config_data.gz $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE - $(call if_changed,gzip) + $(call filechk,gzip)I do not think this is the right approach
because gzip is executed every time, even
if the time stamp is not changed.
Since .config is relatively small, gzip was quickly producing the same binary in multiple runs on my host. I thought about following gen_ikheaders.sh approach of comparing the md5sum, but I felt it was more complex than using filechk and re-compressing every time. I'll send a v2 tomorrow which follows gen_ikheaders.sh approach.
Quoted 21 lines
Quoted 21 lines
kernel/Makefile +0
-0$(obj)/kheaders.o: $(obj)/kheaders_data.tar.xzscripts/Makefile.lib +2
-1@@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@ quiet_cmd_gzip = GZIP $@ cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@ +filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 + # DTC # --------------------------------------------------------------------------- DTC ?= $(objtree)/scripts/dtc/dtc --The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
PATCH v2 1/1 Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed
In reply to Elliot BermanIf you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.
Avoid touching config_data.gz by using a script to compare the existing config_data.gz, KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script follows gen_kheaders.sh approach for determing in update is needed. The script intentionally avoids re-compressing KCONFIG_CONFIG.
The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.
For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.
Changes in v2:
- Use md5 checksum to compare .config instead of gzip'ing again
--- kernel/.gitignore | 1 + kernel/Makefile | 4 +++- kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) create mode 100755 kernel/gen_config_data.sh
kernel/.gitignore +1 -0
@@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +config_data.gz.md5 kheaders.md5 timeconst.h hz.bc
kernel/Makefile +3 -1
@@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o $(obj)/configs.o: $(obj)/config_data.gz targets += config_data.gz +quiet_cmd_genicfg = CHK $(obj)/config_data.gz + cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $< $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE - $(call if_changed,gzip) + $(call cmd,genicfg) $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
kernel/gen_config_data.sh new file +31 -0
@@ -0,0 +1,31 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 + +# This script generates a compressed version of .config, if its checksum has changed +set -e + +this_file="$(readlink -f "$0")" +outfile=$1 +infile=$2 + +config_md5="$(md5sum $infile | cut -d ' ' -f1)" +# Any changes to this script will also cause a rebuild of config_data. +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)" +if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi + +if [ -f $outfile.md5 ] && + [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] && + [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] && + [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then + exit +fi + +if [ "${quiet}" != "silent_" ]; then + echo " GEN $outfile" +fi + +${KGZIP} -c -n -f -9 $infile > $outfile + +echo "$config_md5" > $outfile.md5 +echo "$this_file_md5" >> $outfile.md5 +echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
PATCH RESEND v2 1/1 Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed
In reply to Elliot BermanIf you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.
Avoid touching config_data.gz by using a script to compare the existing config_data.gz, KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script follows gen_kheaders.sh approach for determing in update is needed. The script intentionally avoids re-compressing KCONFIG_CONFIG.
The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.
For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.
Changes in v2:
- Use md5 checksum to compare .config instead of gzip'ing again
--- kernel/.gitignore | 1 + kernel/Makefile | 4 +++- kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) create mode 100755 kernel/gen_config_data.sh
kernel/.gitignore +1 -0
@@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +config_data.gz.md5 kheaders.md5 timeconst.h hz.bc
kernel/Makefile +3 -1
@@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o $(obj)/configs.o: $(obj)/config_data.gz targets += config_data.gz +quiet_cmd_genicfg = CHK $(obj)/config_data.gz + cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $< $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE - $(call if_changed,gzip) + $(call cmd,genicfg) $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
kernel/gen_config_data.sh new file +31 -0
@@ -0,0 +1,31 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 + +# This script generates a compressed version of .config, if its checksum has changed +set -e + +this_file="$(readlink -f "$0")" +outfile=$1 +infile=$2 + +config_md5="$(md5sum $infile | cut -d ' ' -f1)" +# Any changes to this script will also cause a rebuild of config_data. +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)" +if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi + +if [ -f $outfile.md5 ] && + [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] && + [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] && + [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then + exit +fi + +if [ "${quiet}" != "silent_" ]; then + echo " GEN $outfile" +fi + +${KGZIP} -c -n -f -9 $infile > $outfile + +echo "$config_md5" > $outfile.md5 +echo "$this_file_md5" >> $outfile.md5 +echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
On Sat, Apr 24, 2021 at 2:02 AM Elliot Berman <[email protected]> wrote:
Sorry for the delay.
This patch is over-engineering.
I will apply this.
https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/
The 'cmp' command is not expensive.
md5sum is unneeded.
Quoted 100 lines
If you update the timestamp of
KCONFIG_CONFIGwithout actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.Avoid touching config_data.gz by using a script to compare the existing config_data.gz,
KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script followsgen_kheaders.shapproach for determing in update is needed. The script intentionally avoids re-compressingKCONFIG_CONFIG.The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.
For a simple example on my x86 machine, I modified x86 default defconfig to set
CONFIG_IKCONFIG=yand run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.Changes in v2:
- Use md5 checksum to compare .config instead of gzip'ing againSigned-off-by Elliot Berman <[email protected]>--- kernel/.gitignore | 1 + kernel/Makefile | 4 +++- kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) create mode 100755 kernel/gen_config_data.shkernel/.gitignore +1
-0@@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +config_data.gz.md5 kheaders.md5 timeconst.h hz.bckernel/Makefile +3
-1@@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o $(obj)/configs.o: $(obj)/config_data.gz targets += config_data.gz +quiet_cmd_genicfg = CHK $(obj)/config_data.gz + cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $< $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE - $(call if_changed,gzip) + $(call cmd,genicfg) $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xzkernel/gen_config_data.sh new file +31
-1@@ -0,0 +1,31 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 + +# This script generates a compressed version of .config, if its checksum has changed +set -e + +this_file="$(readlink -f "$0")" +outfile=$1 +infile=$2 + +config_md5="$(md5sum $infile | cut -d ' ' -f1)" +# Any changes to this script will also cause a rebuild of config_data. +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)" +if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi + +if [ -f $outfile.md5 ] && + [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] && + [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] && + [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then + exit +fi + +if [ "${quiet}" != "silent_" ]; then + echo " GEN $outfile" +fi + +${KGZIP} -c -n -f -9 $infile > $outfile + +echo "$config_md5" > $outfile.md5 +echo "$this_file_md5" >> $outfile.md5 +echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5 --The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 4 lines · Masahiro Yamada
On Sat, Apr 24, 2021 at 2:02 AM Elliot Berman <[email protected]> wrote:
Sorry for the delay.
No problem!
Quoted 2 lines
This patch is over-engineering.
I thought so, too.
Quoted 6 lines
I will apply this.
https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/The 'cmp' command is not expensive.
md5sum is unneeded.
Thanks!
Quoted 103 lines
Quoted 101 lines
If you update the timestamp of
KCONFIG_CONFIGwithout actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.Avoid touching config_data.gz by using a script to compare the existing config_data.gz,
KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script followsgen_kheaders.shapproach for determing in update is needed. The script intentionally avoids re-compressingKCONFIG_CONFIG.The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.
For a simple example on my x86 machine, I modified x86 default defconfig to set
CONFIG_IKCONFIG=yand run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.Changes in v2:
- Use md5 checksum to compare .config instead of gzip'ing againSigned-off-by Elliot Berman <[email protected]>--- kernel/.gitignore | 1 + kernel/Makefile | 4 +++- kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-)create mode 100755
kernel/gen_config_data.shkernel/.gitignore +1
-0@@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +config_data.gz.md5 kheaders.md5 timeconst.h hz.bckernel/Makefile +3
-1@@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o $(obj)/configs.o: $(obj)/config_data.gz targets += config_data.gz +quiet_cmd_genicfg = CHK $(obj)/config_data.gz + cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $< $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE - $(call if_changed,gzip) + $(call cmd,genicfg) $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xzkernel/gen_config_data.sh new file +31
-1@@ -0,0 +1,31 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 + +# This script generates a compressed version of .config, if its checksum has changed +set -e + +this_file="$(readlink -f "$0")" +outfile=$1 +infile=$2 + +config_md5="$(md5sum $infile | cut -d ' ' -f1)" +# Any changes to this script will also cause a rebuild of config_data. +this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)" +if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi + +if [ -f $outfile.md5 ] && + [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] && + [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] && + [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then + exit +fi + +if [ "${quiet}" != "silent_" ]; then + echo " GEN $outfile" +fi + +${KGZIP} -c -n -f -9 $infile > $outfile + +echo "$config_md5" > $outfile.md5 +echo "$this_file_md5" >> $outfile.md5 +echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5 --The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project