Skip to contentSkip to conversation

Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed

PATCH Superseded Elliot Berman 2 participants 7 messages Apr 1, 2021 – Apr 26, 2021 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed

If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.

Avoid touching config_data.gz by using filechk to compare the existing config_data.gz and update only if it changed.

The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.

For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.

---
 kernel/Makefile      | 2 +-
 scripts/Makefile.lib | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
kernel/Makefile +1 -1
index 320f1f3..bd4e558 100644
@@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz
 
 targets += config_data.gz
 $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
-	$(call if_changed,gzip)
+	$(call filechk,gzip)
 
 $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
 
scripts/Makefile.lib +2 -0
index a4fbaf8..81d3ec1 100644
@@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
 quiet_cmd_gzip = GZIP    $@
       cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
 
+filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9
+
 # DTC
 # ---------------------------------------------------------------------------
 DTC ?= $(objtree)/scripts/dtc/dtc
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#1 lore raw
Quoted 31 lines · Elliot Berman

If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.

Avoid touching config_data.gz by using filechk to compare the existing config_data.gz and update only if it changed.

The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.

For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.

---
 kernel/Makefile      | 2 +-
 scripts/Makefile.lib | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
kernel/Makefile +0 -0
index 320f1f3..bd4e558 100644

kernel/Makefile $(obj)/configs.o: $(obj)/config_data.gz · line 143

Quoted 5 lines
kernel/Makefile +1 -1
@@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz
 
 targets += config_data.gz
 $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
-       $(call if_changed,gzip)
+       $(call filechk,gzip)

I do not think this is the right approach
because gzip is executed every time, even
if the time stamp is not changed.

Quoted 20 lines
kernel/Makefile +0 -0
 $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
 
scripts/Makefile.lib +2 -1
index a4fbaf8..81d3ec1 100644
@@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
 quiet_cmd_gzip = GZIP    $@
       cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
 
+filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9
+
 # DTC
 # ---------------------------------------------------------------------------
 DTC ?= $(objtree)/scripts/dtc/dtc
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

Signature · 2 lines
Best Regards
Masahiro Yamada
#2 lore raw
Quoted 33 lines · Masahiro Yamada
Quoted 32 lines · Elliot Berman

If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.

Avoid touching config_data.gz by using filechk to compare the existing config_data.gz and update only if it changed.

The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.

For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.

---
  kernel/Makefile      | 2 +-
  scripts/Makefile.lib | 2 ++
  2 files changed, 3 insertions(+), 1 deletion(-)
kernel/Makefile +0 -0
index 320f1f3..bd4e558 100644
@@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz
 
Quoted 10 lines

kernel/Makefile $(obj)/configs.o: $(obj)/config_data.gz · line 143

Quoted 4 lines
kernel/Makefile +1 -1
  targets += config_data.gz
  $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
-       $(call if_changed,gzip)
+       $(call filechk,gzip)

I do not think this is the right approach
because gzip is executed every time, even
if the time stamp is not changed.

Since .config is relatively small, gzip was quickly producing the same binary in multiple runs on my host. I thought about following gen_ikheaders.sh approach of comparing the md5sum, but I felt it was more complex than using filechk and re-compressing every time. I'll send a v2 tomorrow which follows gen_ikheaders.sh approach.

Quoted 21 lines
Quoted 21 lines
kernel/Makefile +0 -0
  $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
 
scripts/Makefile.lib +2 -1
index a4fbaf8..81d3ec1 100644
@@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
  quiet_cmd_gzip = GZIP    $@
        cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
 
+filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9
+
  # DTC
  # ---------------------------------------------------------------------------
  DTC ?= $(objtree)/scripts/dtc/dtc
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

#3 lore raw

PATCH v2 1/1 Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed

In reply to Elliot Berman

If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.

Avoid touching config_data.gz by using a script to compare the existing config_data.gz, KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script follows gen_kheaders.sh approach for determing in update is needed. The script intentionally avoids re-compressing KCONFIG_CONFIG.

The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.

For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.

Changes in v2:
- Use md5 checksum to compare .config instead of gzip'ing again

---
 kernel/.gitignore         |  1 +
 kernel/Makefile           |  4 +++-
 kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100755 kernel/gen_config_data.sh
kernel/.gitignore +1 -0
index 78701ea..a191136 100644
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
+config_data.gz.md5
 kheaders.md5
 timeconst.h
 hz.bc
kernel/Makefile +3 -1
index 320f1f3..0784bf3d 100644
@@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o
 $(obj)/configs.o: $(obj)/config_data.gz
 
 targets += config_data.gz
+quiet_cmd_genicfg = CHK     $(obj)/config_data.gz
+      cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $<
 $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
-	$(call if_changed,gzip)
+	$(call cmd,genicfg)
 
 $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
 
kernel/gen_config_data.sh new file +31 -0
new file mode 100755
index 00000000..e9ff193
@@ -0,0 +1,31 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+# This script generates a compressed version of .config, if its checksum has changed
+set -e
+
+this_file="$(readlink -f "$0")"
+outfile=$1
+infile=$2
+
+config_md5="$(md5sum $infile | cut -d ' ' -f1)"
+# Any changes to this script will also cause a rebuild of config_data.
+this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)"
+if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi
+
+if [ -f $outfile.md5 ] &&
+	[ "$(head -n 1 $outfile.md5)" = "$config_md5" ] &&
+	[ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] &&
+	[ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then
+		exit
+fi
+
+if [ "${quiet}" != "silent_" ]; then
+	echo "  GEN     $outfile"
+fi
+
+${KGZIP} -c -n -f -9 $infile > $outfile
+
+echo "$config_md5" > $outfile.md5
+echo "$this_file_md5" >> $outfile.md5
+echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#4 lore raw

PATCH RESEND v2 1/1 Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed

In reply to Elliot Berman

If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.

Avoid touching config_data.gz by using a script to compare the existing config_data.gz, KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script follows gen_kheaders.sh approach for determing in update is needed. The script intentionally avoids re-compressing KCONFIG_CONFIG.

The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.

For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.

Changes in v2:
- Use md5 checksum to compare .config instead of gzip'ing again

---
 kernel/.gitignore         |  1 +
 kernel/Makefile           |  4 +++-
 kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100755 kernel/gen_config_data.sh
kernel/.gitignore +1 -0
index 78701ea..a191136 100644
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
+config_data.gz.md5
 kheaders.md5
 timeconst.h
 hz.bc
kernel/Makefile +3 -1
index 320f1f3..0784bf3d 100644
@@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o
 $(obj)/configs.o: $(obj)/config_data.gz
 
 targets += config_data.gz
+quiet_cmd_genicfg = CHK     $(obj)/config_data.gz
+      cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $<
 $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
-	$(call if_changed,gzip)
+	$(call cmd,genicfg)
 
 $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
 
kernel/gen_config_data.sh new file +31 -0
new file mode 100755
index 00000000..e9ff193
@@ -0,0 +1,31 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+# This script generates a compressed version of .config, if its checksum has changed
+set -e
+
+this_file="$(readlink -f "$0")"
+outfile=$1
+infile=$2
+
+config_md5="$(md5sum $infile | cut -d ' ' -f1)"
+# Any changes to this script will also cause a rebuild of config_data.
+this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)"
+if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi
+
+if [ -f $outfile.md5 ] &&
+	[ "$(head -n 1 $outfile.md5)" = "$config_md5" ] &&
+	[ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] &&
+	[ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then
+		exit
+fi
+
+if [ "${quiet}" != "silent_" ]; then
+	echo "  GEN     $outfile"
+fi
+
+${KGZIP} -c -n -f -9 $infile > $outfile
+
+echo "$config_md5" > $outfile.md5
+echo "$this_file_md5" >> $outfile.md5
+echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#5 lore raw

On Sat, Apr 24, 2021 at 2:02 AM Elliot Berman <[email protected]> wrote:

Sorry for the delay.

This patch is over-engineering.

I will apply this.
https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/

The 'cmp' command is not expensive.
md5sum is unneeded.

Quoted 100 lines

If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.

Avoid touching config_data.gz by using a script to compare the existing config_data.gz, KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script follows gen_kheaders.sh approach for determing in update is needed. The script intentionally avoids re-compressing KCONFIG_CONFIG.

The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.

For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.

Changes in v2:
- Use md5 checksum to compare .config instead of gzip'ing again

---
 kernel/.gitignore         |  1 +
 kernel/Makefile           |  4 +++-
 kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100755 kernel/gen_config_data.sh
kernel/.gitignore +1 -0
index 78701ea..a191136 100644
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
+config_data.gz.md5
 kheaders.md5
 timeconst.h
 hz.bc
kernel/Makefile +3 -1
index 320f1f3..0784bf3d 100644
@@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o
 $(obj)/configs.o: $(obj)/config_data.gz
 
 targets += config_data.gz
+quiet_cmd_genicfg = CHK     $(obj)/config_data.gz
+      cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $<
 $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
-       $(call if_changed,gzip)
+       $(call cmd,genicfg)
 
 $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
 
kernel/gen_config_data.sh new file +31 -1
new file mode 100755
index 00000000..e9ff193
@@ -0,0 +1,31 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+# This script generates a compressed version of .config, if its checksum has changed
+set -e
+
+this_file="$(readlink -f "$0")"
+outfile=$1
+infile=$2
+
+config_md5="$(md5sum $infile | cut -d ' ' -f1)"
+# Any changes to this script will also cause a rebuild of config_data.
+this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)"
+if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi
+
+if [ -f $outfile.md5 ] &&
+       [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] &&
+       [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] &&
+       [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then
+               exit
+fi
+
+if [ "${quiet}" != "silent_" ]; then
+       echo "  GEN     $outfile"
+fi
+
+${KGZIP} -c -n -f -9 $infile > $outfile
+
+echo "$config_md5" > $outfile.md5
+echo "$this_file_md5" >> $outfile.md5
+echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

Signature · 2 lines
Best Regards
Masahiro Yamada
#6 lore raw
Quoted 4 lines · Masahiro Yamada

On Sat, Apr 24, 2021 at 2:02 AM Elliot Berman <[email protected]> wrote:

Sorry for the delay.

No problem!

Quoted 2 lines

This patch is over-engineering.

I thought so, too.

Quoted 6 lines

I will apply this.
https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/

The 'cmp' command is not expensive.
md5sum is unneeded.

Thanks!

Quoted 103 lines
Quoted 101 lines

If you update the timestamp of KCONFIG_CONFIG without actually changing anything, config_data.gz is re-generated and causes vmlinux to re-link. When Link Time Optimization is enabled, unnecessary re-linking of vmlinux is highly desirable since it adds several minutes to build time.

Avoid touching config_data.gz by using a script to compare the existing config_data.gz, KCONFIG_CONFIG, or script itself to update only if any is mismatched. The script follows gen_kheaders.sh approach for determing in update is needed. The script intentionally avoids re-compressing KCONFIG_CONFIG.

The .config can be touched, for instance, by a build script which installs the default defconfig and then applies a defconfig fragment on top.

For a simple example on my x86 machine, I modified x86 default defconfig to set CONFIG_IKCONFIG=y and run:
make -j50 defconfig tiny.config vmlinux
make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz change.

Changes in v2:
- Use md5 checksum to compare .config instead of gzip'ing again

---
  kernel/.gitignore         |  1 +
  kernel/Makefile           |  4 +++-
  kernel/gen_config_data.sh | 31 +++++++++++++++++++++++++++++++
  3 files changed, 35 insertions(+), 1 deletion(-)

create mode 100755 kernel/gen_config_data.sh

kernel/.gitignore +1 -0
index 78701ea..a191136 100644
@@ -1,4 +1,5 @@
  # SPDX-License-Identifier: GPL-2.0-only
+config_data.gz.md5
  kheaders.md5
  timeconst.h
  hz.bc
kernel/Makefile +3 -1
index 320f1f3..0784bf3d 100644
@@ -139,8 +139,10 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o
  $(obj)/configs.o: $(obj)/config_data.gz
 
  targets += config_data.gz
+quiet_cmd_genicfg = CHK     $(obj)/config_data.gz
+      cmd_genicfg = $(CONFIG_SHELL) $(srctree)/kernel/gen_config_data.sh $@ $<
  $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
-       $(call if_changed,gzip)
+       $(call cmd,genicfg)
 
  $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
 
kernel/gen_config_data.sh new file +31 -1
new file mode 100755
index 00000000..e9ff193
@@ -0,0 +1,31 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+# This script generates a compressed version of .config, if its checksum has changed
+set -e
+
+this_file="$(readlink -f "$0")"
+outfile=$1
+infile=$2
+
+config_md5="$(md5sum $infile | cut -d ' ' -f1)"
+# Any changes to this script will also cause a rebuild of config_data.
+this_file_md5="$(md5sum $sfile | cut -d ' ' -f1)"
+if [ -f $outfile ]; then outfile_md5="$(md5sum $outfile | cut -d ' ' -f1)"; fi
+
+if [ -f $outfile.md5 ] &&
+       [ "$(head -n 1 $outfile.md5)" = "$config_md5" ] &&
+       [ "$(head -n 2 $outfile.md5 | tail -n 1)" = "$this_file_md5" ] &&
+       [ "$(tail -n 1 $outfile.md5)" = "$outfile_md5" ]; then
+               exit
+fi
+
+if [ "${quiet}" != "silent_" ]; then
+       echo "  GEN     $outfile"
+fi
+
+${KGZIP} -c -n -f -9 $infile > $outfile
+
+echo "$config_md5" > $outfile.md5
+echo "$this_file_md5" >> $outfile.md5
+echo "$(md5sum $outfile | cut -d ' ' -f1)" >> $outfile.md5
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project