Skip to contentSkip to conversation

scripts: Fix incremental build header re-generation

PATCH In review Jeevan Shriram 2 participants 4 messages Mar 1, 2021 – Mar 24, 2021 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scripts: Fix incremental build header re-generation

compile.h and autoconf.h are ignored when checking headers sha as they are always re-generated for every kernel compilation. However, these two headers are packaged into kheaders tar. During incremental compilation of kernel, kheaders tar file is always generated and re-packaged irrespective of the changes in headers.

Change-Id I7a64faebb81df44c32230b0fea1d6df09d7ce66f
---
 kernel/gen_kheaders.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/gen_kheaders.sh +1 -2
index c1510f0..5499f72 100755
@@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
 if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
 if [ -f kernel/kheaders.md5 ] &&
 	[ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
-	[ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
-	[ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
+	[ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
 		exit
 fi
 
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#1 lore raw
Quoted 6 lines · Jeevan Shriram

compile.h and autoconf.h are ignored when checking headers sha as they are always re-generated for every kernel compilation. However, these two headers are packaged into kheaders tar. During incremental compilation of kernel, kheaders tar file is always generated and re-packaged irrespective of the changes in headers.

I do not see this problem.
Could you describe the steps to reproduce it, please?

Quoted 24 lines
Change-Id I7a64faebb81df44c32230b0fea1d6df09d7ce66f
---
 kernel/gen_kheaders.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/gen_kheaders.sh +1 -3
index c1510f0..5499f72 100755
@@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
 if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
 if [ -f kernel/kheaders.md5 ] &&
        [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
-       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
-       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
+       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
                exit
 fi
 
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

Signature · 2 lines
Best Regards
Masahiro Yamada
#2 lore raw
Quoted 10 lines · Masahiro Yamada
Quoted 5 lines · Jeevan Shriram

compile.h and autoconf.h are ignored when checking headers sha as they are always re-generated for every kernel compilation. However, these two headers are packaged into kheaders tar. During incremental compilation of kernel, kheaders tar file is always generated and re-packaged irrespective of the changes in headers.

I do not see this problem.
Could you describe the steps to reproduce it, please?

Without making any changes in the kernel or it's headers, re-compile the kernel. i.e.,incremental kernel build without any changes. I have added following log in gen_kheaders.sh script for confirming the hash differences.

kernel/gen_kheaders.sh +4 -0
index b7425a0..ee542a0 100755
@@ -40,6 +40,10 @@ obj_files_md5="$(find $dir_list -name "*.h"                     |
 # Any changes to this script will also cause a rebuild of the archive.
 this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
 if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
+
+echo "Old Tar file $tarfile_md5"
+echo "New Tar file hash $this_file_md5"
+
 if [ -f kernel/kheaders.md5 ] &&
        [ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] &&
        [ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] &&

log output :
89306 19:29:02.109961 CHK kernel/kheaders_data.tar.xz
89307 19:29:02.109971 Old Tar file 2aa6990e4183c31a862951f4bcac037e 89308 19:29:02.109982 New Tar file hash ecf84e700c7cacfe8b35a0905859582d

Quoted 25 lines
Quoted 24 lines
Change-Id I7a64faebb81df44c32230b0fea1d6df09d7ce66f
---
  kernel/gen_kheaders.sh | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
kernel/gen_kheaders.sh +1 -3
index c1510f0..5499f72 100755
@@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
  if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
  if [ -f kernel/kheaders.md5 ] &&
         [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
-       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
-       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
+       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
                 exit
  fi
 
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

#3 lore raw
Quoted 10 lines · Jeevan Shriram
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Jeevan Shriram

compile.h and autoconf.h are ignored when checking headers sha as they are always re-generated for every kernel compilation. However, these two headers are packaged into kheaders tar. During incremental compilation of kernel, kheaders tar file is always generated and re-packaged irrespective of the changes in headers.

Quoted 5 lines
Quoted 3 lines

I do not see this problem.
Could you describe the steps to reproduce it, please?

Without making any changes in the kernel or it's headers, re-compile the kernel. i.e.,incremental kernel build without any changes. I have added following log in gen_kheaders.sh script for confirming the hash differences.

Of course they are different because you are comparing the hashes of two different files.

kernel/gen_kheaders.sh obj_files_md5="$(find $dir_list -name "*.h" | · line 44

Quoted 11 lines
kernel/gen_kheaders.sh +2 -0
index b7425a0..ee542a0 100755
@@ -40,6 +40,10 @@ obj_files_md5="$(find $dir_list -name "*.h"                     |
  # Any changes to this script will also cause a rebuild of the archive.
  this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
  if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
+
+echo "Old Tar file $tarfile_md5"

This is the hash of kernel/kheaders_data.tar.xz

kernel/gen_kheaders.sh obj_files_md5="$(find $dir_list -name "*.h" | · line 45

Quoted 1 line
kernel/gen_kheaders.sh +1 -0
+echo "New Tar file hash $this_file_md5"

This is the hash of kernel/gen_kheaders.sh

Quoted 9 lines
kernel/gen_kheaders.sh +1 -0
+
  if [ -f kernel/kheaders.md5 ] &&
         [ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] &&
         [ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] &&

log output :
89306 19:29:02.109961 CHK kernel/kheaders_data.tar.xz
89307 19:29:02.109971 Old Tar file 2aa6990e4183c31a862951f4bcac037e 89308 19:29:02.109982 New Tar file hash ecf84e700c7cacfe8b35a0905859582d

I do not understand what your claim is.

CHK kernel/kheaders_data.tar.xz

is displayed when it is checking the hash in order to determine if the tarball should be updated.

GEN kernel/kheaders_data.tar.xz

is displayed when the tarball is really updated.

[Incremental build with no change]

masahiro@oscar:~/ref/linux$ make -j24
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
Kernel: arch/x86/boot/bzImage is ready  (#2)

[Touch one header and rebuild]

masahiro@oscar:~/ref/linux$ touch include/uapi/drm/exynos_drm.h
masahiro@oscar:~/ref/linux$ make -j24
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
  GEN     kernel/kheaders_data.tar.xz
  CC      kernel/kheaders.o
  AR      kernel/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  SORTTAB vmlinux
  SYSMAP  System.map
  MODPOST Module.symvers
  CC      arch/x86/boot/version.o
  VOFFSET arch/x86/boot/compressed/../voffset.h
  OBJCOPY arch/x86/boot/compressed/vmlinux.bin
  RELOCS  arch/x86/boot/compressed/vmlinux.relocs
  CC      arch/x86/boot/compressed/kaslr.o
  CC      arch/x86/boot/compressed/misc.o
  GZIP    arch/x86/boot/compressed/vmlinux.bin.gz
  MKPIGGY arch/x86/boot/compressed/piggy.S
  AS      arch/x86/boot/compressed/piggy.o
  LD      arch/x86/boot/compressed/vmlinux
  ZOFFSET arch/x86/boot/zoffset.h
  OBJCOPY arch/x86/boot/vmlinux.bin
  AS      arch/x86/boot/header.o
  LD      arch/x86/boot/setup.elf
  OBJCOPY arch/x86/boot/setup.bin
  BUILD   arch/x86/boot/bzImage
Kernel: arch/x86/boot/bzImage is ready  (#3)
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Change-Id I7a64faebb81df44c32230b0fea1d6df09d7ce66f
---
  kernel/gen_kheaders.sh | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
kernel/gen_kheaders.sh +1 -2
index c1510f0..5499f72 100755
@@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
  if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
  if [ -f kernel/kheaders.md5 ] &&
         [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
-       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
-       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
+       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
                 exit
  fi
 
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

--
Best Regards
Masahiro Yamada