Skip to contentSkip to conversation

kbuild: add a tool to list files ignored by git

PATCH In review Masahiro Yamada 4 participants 26 messages Feb 15, 2023 – Mar 8, 2023 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH v6 1/12 kbuild: add a tool to list files ignored by git

In short, the motivation of this commit is to build a source package without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.

Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.

In fact, there are some ways to achieve this.

The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses it, but I do not like it because it works only when the source tree is managed by git, and all files you want in the tarball must be committed in advance.

I want to make it work without relying on git. We can do this.

Files that are ignored by git are generated files, so should be excluded from the source tarball. We can list them out by parsing the .gitignore files. Of course, .gitignore does not cover all the cases, but it works well enough.

tar(1) claims to support it:

--exclude-vcs-ignores

Exclude files that match patterns read from VCS-specific ignore files. Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.

The best scenario would be to use 'tar --exclude-vcs-ignores', but this option does not work. --exclude-vcs-ignore does not understand any of the negation (!), preceding slash, following slash, etc.. So, this option is just useless.

Hence, I wrote this gitignore parser. The previous version [1], written in Python, was so slow. This version is implemented in C, so it works much faster.

I imported the code from git (commit: 23c56f7bd5f1), so we get the same result.

This tool traverses the source tree, parsing all .gitignore files, and prints file paths that are ignored by git.

The output is similar to 'git ls-files --ignored --directory --others --exclude-per-directory=.gitignore', except

[1] Not sorted
[2] No trailing slash for directories

[2] is intentional because tar's --exclude-from option cannot handle trailing slashes.

[How to test this tool]

  $ git clean -dfx
  $ make -s -j$(nproc) defconfig all                       # or allmodconifg or whatever
  $ git archive -o ../linux1.tar --prefix=./ HEAD
  $ tar tf ../linux1.tar | LANG=C sort > ../file-list1     # files emitted by 'git archive'
  $ make scripts_package
    HOSTCC  scripts/list-gitignored
  $ scripts/list-gitignored  --prefix=./ -o ../exclude-list
  $ tar cf ../linux2.tar --exclude-from=../exclude-list .
  $ tar tf ../linux2.tar | LANG=C sort > ../file-list2     # files emitted by 'tar'
  $ diff  ../file-list1 ../file-list2 | grep -E '^(<|>)'
  < ./Documentation/devicetree/bindings/.yamllint
  < ./drivers/clk/.kunitconfig
  < ./drivers/gpu/drm/tests/.kunitconfig
  < ./drivers/hid/.kunitconfig
  < ./fs/ext4/.kunitconfig
  < ./fs/fat/.kunitconfig
  < ./kernel/kcsan/.kunitconfig
  < ./lib/kunit/.kunitconfig
  < ./mm/kfence/.kunitconfig
  < ./tools/testing/selftests/arm64/tags/
  < ./tools/testing/selftests/arm64/tags/.gitignore
  < ./tools/testing/selftests/arm64/tags/Makefile
  < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
  < ./tools/testing/selftests/arm64/tags/tags_test.c
  < ./tools/testing/selftests/kvm/.gitignore
  < ./tools/testing/selftests/kvm/Makefile
  < ./tools/testing/selftests/kvm/config
  < ./tools/testing/selftests/kvm/settings

The source tarball contains most of files that are tracked by git. You see some diffs, but it is just because some .gitignore files are wrong.

  $ git ls-files -i -c --exclude-per-directory=.gitignore
  Documentation/devicetree/bindings/.yamllint
  drivers/clk/.kunitconfig
  drivers/gpu/drm/tests/.kunitconfig
  drivers/hid/.kunitconfig
  fs/ext4/.kunitconfig
  fs/fat/.kunitconfig
  kernel/kcsan/.kunitconfig
  lib/kunit/.kunitconfig
  mm/kfence/.kunitconfig
  tools/testing/selftests/arm64/tags/.gitignore
  tools/testing/selftests/arm64/tags/Makefile
  tools/testing/selftests/arm64/tags/run_tags_test.sh
  tools/testing/selftests/arm64/tags/tags_test.c
  tools/testing/selftests/kvm/.gitignore
  tools/testing/selftests/kvm/Makefile
  tools/testing/selftests/kvm/config
  tools/testing/selftests/kvm/settings

[1]: https://lore.kernel.org/all/[email protected]/

---

(no changes since v5)

Changes in v5:
- Import a lot of code from GIT
- Fix a bug that the output file is relative to --rootdir
- Check ENOENT when open() fails to open .gitignore
- add more options, --ignore-case, --stat

Changes in v3:
- Various code refactoring: remove struct gitignore, remove next: label etc.
- Support --extra-pattern option

Changes in v2:
- Reimplement in C

 Makefile                  |    7 +-
 scripts/.gitignore        |    1 +
 scripts/Makefile          |    2 +-
 scripts/list-gitignored.c | 1055 +++++++++++++++++++++++++++++++++++++
 4 files changed, 1063 insertions(+), 2 deletions(-)
 create mode 100644 scripts/list-gitignored.c
Makefile +6 -1
index 2faf872b6808..7c726fd26540 100644
@@ -274,7 +274,8 @@ no-dot-config-targets := $(clean-targets) \
 			 cscope gtags TAGS tags help% %docs check% coccicheck \
 			 $(version_h) headers headers_% archheaders archscripts \
 			 %asm-generic kernelversion %src-pkg dt_binding_check \
-			 outputmakefile rustavailable rustfmt rustfmtcheck
+			 outputmakefile rustavailable rustfmt rustfmtcheck \
+			 scripts_package
 # Installation targets should not require compiler. Unfortunately, vdso_install
 # is an exception where build artifacts may be updated. This must be fixed.
 no-compiler-targets := $(no-dot-config-targets) install dtbs_install \
@@ -1652,6 +1653,10 @@ distclean: mrproper
 %pkg: include/config/kernel.release FORCE
 	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.package $@
 
+PHONY += scripts_package
+scripts_package: scripts_basic
+	$(Q)$(MAKE) $(build)=scripts scripts/list-gitignored
+
 # Brief documentation of the typical targets used
 # ---------------------------------------------------------------------------
 
scripts/.gitignore +1 -0
index 6e9ce6720a05..feb43045d1b1 100644
@@ -3,6 +3,7 @@
 /generate_rust_target
 /insert-sys-cert
 /kallsyms
+/list-gitignored
 /module.lds
 /recordmcount
 /sign-file
scripts/Makefile +1 -1
index 32b6ba722728..e8917975905c 100644
@@ -38,7 +38,7 @@ HOSTCFLAGS_sorttable.o += -DMCOUNT_SORT_ENABLED
 endif
 
 # The following programs are only built on demand
-hostprogs += unifdef
+hostprogs += list-gitignored unifdef
 
 # The module linker script is preprocessed on demand
 targets += module.lds
scripts/list-gitignored.c new file +1055 -0

1,055 lines not on this page · raw message

Signature · 1 line
2.34.1
#1 lore raw

PATCH v6 2/12 kbuild: deb-pkg: create source package without cleaning

If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.

Instead of cleaning, pass the exclude list to tar's --exclude-from option.

Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig.

With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the .orig tarball.

linux.tar.gz is rebuilt only when the source files that would go into the tarball are changed.

---

Changes in v6:
- Add more stubs to cmd_tar so I can reuse it for the other rules

Changes in v5:
- Avoid unneeded rebuild of the tarball when nothing in the source tree is changed.

Changes in v4:
- Fix a typo in comment

Changes in v3:
- Add --extra-pattern='*.rej'
- Exclude symlinks at the toplevel
- Add --sort=name tar option

 scripts/Makefile.package | 48 +++++++++++++++++++++++++++++++++++-----
 scripts/package/mkdebian | 24 ++++++++++++++++++++
 2 files changed, 66 insertions(+), 6 deletions(-)
scripts/Makefile.package +42 -6
index dfbf40454a99..f0002ace4156 100644
@@ -43,13 +43,47 @@ if test "$(objtree)" != "$(srctree)"; then \
 	echo >&2 "  ERROR:"; \
 	echo >&2 "  Building source tarball is not possible outside the"; \
 	echo >&2 "  kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \
-	echo >&2 "  binrpm-pkg or bindeb-pkg target instead."; \
+	echo >&2 "  binrpm-pkg target instead."; \
 	echo >&2; \
 	false; \
 fi ; \
 tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
 	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
 
+# .tmp_filelist .tmp_filelist_exclude
+# ---------------------------------------------------------------------------
+
+scripts/list-gitignored: FORCE
+	$(Q)$(MAKE) -f $(srctree)/Makefile scripts_package
+
+# 1f5d3a6b6532e25a5cdf1f311956b2b03d343a48 removed '*.rej' from .gitignore,
+# but it is definitely a generated file.
+filechk_filelist = \
+	$< --exclude='*.rej' --output=$@_exclude --prefix=./ --rootdir=$(srctree) --stat=-
+
+.tmp_filelist: scripts/list-gitignored FORCE
+	$(call filechk,filelist)
+
+# tarball
+# ---------------------------------------------------------------------------
+
+quiet_cmd_tar = TAR     $@
+      cmd_tar = tar -c -f $@ $(tar-compress-opt) $(tar-exclude-opt) \
+                --owner=0 --group=0 --sort=name \
+                --transform 's:^\.:$*:S' -C $(tar-rootdir) .
+
+tar-rootdir := $(srctree)
+
+%.tar.gz: private tar-compress-opt := -I $(KGZIP)
+%.tar.gz:
+	$(call cmd,tar)
+
+# Linux source tarball
+# ---------------------------------------------------------------------------
+
+linux.tar.gz: tar-exclude-opt = --exclude=./$@ --exclude-from=$<_exclude
+linux.tar.gz: .tmp_filelist
+
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
@@ -80,13 +114,12 @@ binrpm-pkg:
 		$(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec
 
 PHONY += deb-pkg
-deb-pkg:
-	$(MAKE) clean
+deb-pkg: linux.tar.gz
 	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-	$(call cmd,src_tar,$(KDEB_SOURCENAME))
 	origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-		mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc
+		cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
+		--build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg:
@@ -174,4 +207,7 @@ help:
 	@echo '  perf-tarxz-src-pkg  - Build $(perf-tar).tar.xz source tarball'
 	@echo '  perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
 
+PHONY += FORCE
+FORCE:
+
 .PHONY: $(PHONY)
scripts/package/mkdebian +24 -0
index c3bbef7a6754..68b13ef590ba 100755
@@ -84,6 +84,8 @@ set_debarch() {
 	fi
 }
 
+rm -rf debian
+
 # Some variables and settings used throughout the script
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
@@ -135,6 +137,28 @@ fi
 mkdir -p debian/source/
 echo "1.0" > debian/source/format
 
+# Ugly: ignore anything except .config or debian/
+cat<<'EOF' > debian/source/local-options
+diff-ignore
+
+extend-diff-ignore = ^[^.d]
+
+extend-diff-ignore = ^\.[^c]
+extend-diff-ignore = ^\.c($|[^o])
+extend-diff-ignore = ^\.co($|[^n])
+extend-diff-ignore = ^\.con($|[^f])
+extend-diff-ignore = ^\.conf($|[^i])
+extend-diff-ignore = ^\.confi($|[^g])
+extend-diff-ignore = ^\.config.
+
+extend-diff-ignore = ^d($|[^e])
+extend-diff-ignore = ^de($|[^b])
+extend-diff-ignore = ^deb($|[^i])
+extend-diff-ignore = ^debi($|[^a])
+extend-diff-ignore = ^debia($|[^n])
+extend-diff-ignore = ^debian[^/]
+EOF
+
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
 extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"
Signature · 1 line
2.34.1
Reviewed-by on v6 2/12
#2 lore raw

Re: [PATCH v6 02/12] kbuild: deb-pkg: create source package without cleaning

Quoted 148 lines · On Wed, Feb 15, 2023 at 10:20:24AM +0900 Masahiro Yamada wrote:

If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.

Instead of cleaning, pass the exclude list to tar's --exclude-from option.

Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig.

With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the .orig tarball.

linux.tar.gz is rebuilt only when the source files that would go into the tarball are changed.

---

Changes in v6:
- Add more stubs to cmd_tar so I can reuse it for the other rules

Changes in v5:
- Avoid unneeded rebuild of the tarball when nothing in the source tree is changed.

Changes in v4:
- Fix a typo in comment

Changes in v3:
- Add --extra-pattern='*.rej'
- Exclude symlinks at the toplevel
- Add --sort=name tar option

 scripts/Makefile.package | 48 +++++++++++++++++++++++++++++++++++-----
 scripts/package/mkdebian | 24 ++++++++++++++++++++
 2 files changed, 66 insertions(+), 6 deletions(-)
scripts/Makefile.package +42 -6
index dfbf40454a99..f0002ace4156 100644
@@ -43,13 +43,47 @@ if test "$(objtree)" != "$(srctree)"; then \
 	echo >&2 "  ERROR:"; \
 	echo >&2 "  Building source tarball is not possible outside the"; \
 	echo >&2 "  kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \
-	echo >&2 "  binrpm-pkg or bindeb-pkg target instead."; \
+	echo >&2 "  binrpm-pkg target instead."; \
 	echo >&2; \
 	false; \
 fi ; \
 tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
 	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
 
+# .tmp_filelist .tmp_filelist_exclude
+# ---------------------------------------------------------------------------
+
+scripts/list-gitignored: FORCE
+	$(Q)$(MAKE) -f $(srctree)/Makefile scripts_package
+
+# 1f5d3a6b6532e25a5cdf1f311956b2b03d343a48 removed '*.rej' from .gitignore,
+# but it is definitely a generated file.
+filechk_filelist = \
+	$< --exclude='*.rej' --output=$@_exclude --prefix=./ --rootdir=$(srctree) --stat=-
+
+.tmp_filelist: scripts/list-gitignored FORCE
+	$(call filechk,filelist)
+
+# tarball
+# ---------------------------------------------------------------------------
+
+quiet_cmd_tar = TAR     $@
+      cmd_tar = tar -c -f $@ $(tar-compress-opt) $(tar-exclude-opt) \
+                --owner=0 --group=0 --sort=name \
+                --transform 's:^\.:$*:S' -C $(tar-rootdir) .
+
+tar-rootdir := $(srctree)
+
+%.tar.gz: private tar-compress-opt := -I $(KGZIP)
+%.tar.gz:
+	$(call cmd,tar)
+
+# Linux source tarball
+# ---------------------------------------------------------------------------
+
+linux.tar.gz: tar-exclude-opt = --exclude=./$@ --exclude-from=$<_exclude
+linux.tar.gz: .tmp_filelist
+
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
@@ -80,13 +114,12 @@ binrpm-pkg:
 		$(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec
 
 PHONY += deb-pkg
-deb-pkg:
-	$(MAKE) clean
+deb-pkg: linux.tar.gz
 	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-	$(call cmd,src_tar,$(KDEB_SOURCENAME))
 	origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-		mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc
+		cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
+		--build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg:
@@ -174,4 +207,7 @@ help:
 	@echo '  perf-tarxz-src-pkg  - Build $(perf-tar).tar.xz source tarball'
 	@echo '  perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
 
+PHONY += FORCE
+FORCE:
+
 .PHONY: $(PHONY)
scripts/package/mkdebian +17 -0
index c3bbef7a6754..68b13ef590ba 100755
@@ -84,6 +84,8 @@ set_debarch() {
 	fi
 }
 
+rm -rf debian
+
 # Some variables and settings used throughout the script
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
@@ -135,6 +137,28 @@ fi
 mkdir -p debian/source/
 echo "1.0" > debian/source/format
 
+# Ugly: ignore anything except .config or debian/
+cat<<'EOF' > debian/source/local-options
+diff-ignore
+
+extend-diff-ignore = ^[^.d]
+
+extend-diff-ignore = ^\.[^c]
+extend-diff-ignore = ^\.c($|[^o])
+extend-diff-ignore = ^\.co($|[^n])
+extend-diff-ignore = ^\.con($|[^f])
+extend-diff-ignore = ^\.conf($|[^i])
+extend-diff-ignore = ^\.confi($|[^g])
+extend-diff-ignore = ^\.config.
+
+extend-diff-ignore = ^d($|[^e])

scripts/package/mkdebian fi · line 164

Quoted 10 lines
scripts/package/mkdebian +7 -0
+extend-diff-ignore = ^de($|[^b])
+extend-diff-ignore = ^deb($|[^i])
+extend-diff-ignore = ^debi($|[^a])
+extend-diff-ignore = ^debia($|[^n])
+extend-diff-ignore = ^debian[^/]
+EOF
+
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
 extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"

ah, nice. When I attempted to build with current Debian amd64 kernel config, python3 was missing as build-depencendy (due to some BPF kconfig).

Quoted 2 lines
Signature · 1 line
2.34.1
Signature · 3 lines
epost|xmpp: [email protected]          irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --
#3 lore raw

PATCH v6 3/12 kbuild: rpm-pkg: build binary packages from source rpm

In reply to Masahiro Yamada

The build rules of rpm-pkg and srcrpm-pkg are almost the same. Remove the code duplication.

Change rpm-pkg to build binary packages from the source package generated by srcrpm-pkg.

This changes the output directory of the srpm generated by 'make rpm-pkg' because srcrpm-pkg overrides _srcrpmdir.

---

(no changes since v3)

Changes in v3:
- Explain that the source package location will be changed.

 scripts/Makefile.package | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
scripts/Makefile.package +3 -5
index f0002ace4156..55441d776d70 100644
@@ -87,11 +87,9 @@ linux.tar.gz: .tmp_filelist
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
-rpm-pkg:
-	$(MAKE) clean
-	$(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec
-	$(call cmd,src_tar,$(KERNELPATH),kernel.spec)
-	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ta $(KERNELPATH).tar.gz \
+rpm-pkg: srpm = $(shell rpmspec --srpm --query --queryformat='%{name}-%{VERSION}-%{RELEASE}.src.rpm' kernel.spec)
+rpm-pkg: srcrpm-pkg
+	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -rb $(srpm) \
 	--define='_smp_mflags %{nil}'
 
 # srcrpm-pkg
Signature · 1 line
2.34.1
#4 lore raw

PATCH v6 4/12 kbuild: srcrpm-pkg: create source package without cleaning

In reply to Masahiro Yamada

If you run 'make (src)rpm-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.

Instead of cleaning, pass the exclude list to tar's --exclude-from option.

Previously, the .config was contained in the source tarball.

With this commit, the source rpm consists of separate linux.tar.gz and .config.

Remove stale comments. Now, 'make (src)rpm-pkg' works with O= option.

---

(no changes since v4)

Changes in v4:
- Do not delete the old tar command because it is still used by snap-pkg although snap-pkg is broken, and it does not work at all.

 scripts/Makefile.package | 32 ++++----------------------------
 scripts/package/mkspec   |  8 ++++----
 2 files changed, 8 insertions(+), 32 deletions(-)
scripts/Makefile.package +4 -28
index 55441d776d70..7266a11855f4 100644
@@ -3,27 +3,6 @@
 
 include $(srctree)/scripts/Kbuild.include
 
-# RPM target
-# ---------------------------------------------------------------------------
-# The rpm target generates two rpm files:
-# /usr/src/packages/SRPMS/kernel-2.6.7rc2-1.src.rpm
-# /usr/src/packages/RPMS/i386/kernel-2.6.7rc2-1.<arch>.rpm
-# The src.rpm files includes all source for the kernel being built
-# The <arch>.rpm includes kernel configuration, modules etc.
-#
-# Process to create the rpm files
-# a) clean the kernel
-# b) Generate .spec file
-# c) Build a tar ball, using symlink to make kernel version
-#    first entry in the path
-# d) and pack the result to a tar.gz file
-# e) generate the rpm files, based on kernel.spec
-# - Use /. to avoid tar packing just the symlink
-
-# Note that the rpm-pkg target cannot be used with KBUILD_OUTPUT,
-# but the binrpm-pkg target can; for some reason O= gets ignored.
-
-# Remove hyphens since they have special meaning in RPM filenames
 KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE))
 KDEB_SOURCENAME ?= linux-upstream
 KBUILD_PKG_ROOTCMD ?="fakeroot -u"
@@ -42,8 +21,7 @@ if test "$(objtree)" != "$(srctree)"; then \
 	echo >&2; \
 	echo >&2 "  ERROR:"; \
 	echo >&2 "  Building source tarball is not possible outside the"; \
-	echo >&2 "  kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \
-	echo >&2 "  binrpm-pkg target instead."; \
+	echo >&2 "  kernel source tree. Don't set KBUILD_OUTPUT"; \
 	echo >&2; \
 	false; \
 fi ; \
@@ -95,12 +73,10 @@ rpm-pkg: srcrpm-pkg
 # srcrpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += srcrpm-pkg
-srcrpm-pkg:
-	$(MAKE) clean
+srcrpm-pkg: linux.tar.gz
 	$(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec
-	$(call cmd,src_tar,$(KERNELPATH),kernel.spec)
-	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ts $(KERNELPATH).tar.gz \
-	--define='_smp_mflags %{nil}' --define='_srcrpmdir $(srctree)'
+	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -bs kernel.spec \
+	--define='_smp_mflags %{nil}' --define='_sourcedir .' --define='_srcrpmdir .'
 
 # binrpm-pkg
 # ---------------------------------------------------------------------------
scripts/package/mkspec +4 -4
index 108c0cb95436..83a64d9d7372 100755
@@ -47,7 +47,8 @@ sed -e '/^DEL/d' -e 's/^\t*//' <<EOF
 	Group: System Environment/Kernel
 	Vendor: The Linux Community
 	URL: https://www.kernel.org
-$S	Source: kernel-$__KERNELRELEASE.tar.gz
+$S	Source0: linux.tar.gz
+$S	Source1: .config
 	Provides: $PROVIDES
 $S	BuildRequires: bc binutils bison dwarves
 $S	BuildRequires: (elfutils-libelf-devel or libelf-devel) flex
@@ -83,9 +84,8 @@ $S$M	This package provides kernel headers and makefiles sufficient to build modu
 $S$M	against the $__KERNELRELEASE kernel package.
 $S$M
 $S	%prep
-$S	%setup -q
-$S	rm -f scripts/basic/fixdep scripts/kconfig/conf
-$S	rm -f tools/objtool/{fixdep,objtool}
+$S	%setup -q -n linux
+$S	cp %{SOURCE1} .
 $S
 $S	%build
 $S	$MAKE %{?_smp_mflags} KERNELRELEASE=$KERNELRELEASE KBUILD_BUILD_VERSION=%{release}
Signature · 1 line
2.34.1
#5 lore raw

PATCH v6 5/12 kbuild: deb-pkg: hide KDEB_SOURCENAME from Makefile

In reply to Masahiro Yamada

scripts/Makefile.package does not need to know the value of KDEB_SOURCENAME because the source name can be taken from debian/changelog by using dpkg-parsechangelog.

Move the default of KDEB_SOURCENAME (i.e. linux-upstream) to scripts/package/mkdebian.

---

(no changes since v5)

Changes in v5:
- Copy linux.tar.gz to .orig tarball

Changes in v3:
- Move cmd_debianize

Changes in v2:
- New patch

 scripts/Makefile.package | 23 +++++++++++++++--------
 scripts/package/mkdebian |  2 +-
 2 files changed, 16 insertions(+), 9 deletions(-)
scripts/Makefile.package +15 -8
index 7266a11855f4..c207f4ec47ac 100644
@@ -4,9 +4,7 @@
 include $(srctree)/scripts/Kbuild.include
 
 KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE))
-KDEB_SOURCENAME ?= linux-upstream
 KBUILD_PKG_ROOTCMD ?="fakeroot -u"
-export KDEB_SOURCENAME
 # Include only those top-level files that are needed by make, plus the GPL copy
 TAR_CONTENT := Documentation LICENSES arch block certs crypto drivers fs \
                include init io_uring ipc kernel lib mm net rust \
@@ -87,17 +85,26 @@ binrpm-pkg:
 	+rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \
 		$(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec
 
+quiet_cmd_debianize = GEN     $@
+      cmd_debianize = $(srctree)/scripts/package/mkdebian
+
+debian: FORCE
+	$(call cmd,debianize)
+
+PHONY += debian-orig
+debian-orig: private source = $(shell dpkg-parsechangelog -S Source)
+debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//')
+debian-orig: private orig-name = $(source)_$(version).orig.tar.gz
+debian-orig: linux.tar.gz debian
+	$(Q)cp $< ../$(orig-name)
+
 PHONY += deb-pkg
-deb-pkg: linux.tar.gz
-	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-	origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-		cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+deb-pkg: debian-orig
 	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
 		--build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
-bindeb-pkg:
-	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
+bindeb-pkg: debian
 	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc
 
 PHONY += intdeb-pkg
scripts/package/mkdebian +1 -1
index 68b13ef590ba..534e9713761a 100755
@@ -95,7 +95,7 @@ else
 	revision=$($srctree/init/build-version)
 	packageversion=$version-$revision
 fi
-sourcename=$KDEB_SOURCENAME
+sourcename=${KDEB_SOURCENAME:-linux-upstream}
 
 if [ "$ARCH" = "um" ] ; then
 	packagename=user-mode-linux
Signature · 1 line
2.34.1
Reviewed-by on v6 5/12
#6 lore raw

Re: [PATCH v6 05/12] kbuild: deb-pkg: hide KDEB_SOURCENAME from Makefile

Quoted 87 lines · On Wed, Feb 15, 2023 at 10:20:27AM +0900 Masahiro Yamada wrote:

scripts/Makefile.package does not need to know the value of KDEB_SOURCENAME because the source name can be taken from debian/changelog by using dpkg-parsechangelog.

Move the default of KDEB_SOURCENAME (i.e. linux-upstream) to scripts/package/mkdebian.

---

(no changes since v5)

Changes in v5:
- Copy linux.tar.gz to .orig tarball

Changes in v3:
- Move cmd_debianize

Changes in v2:
- New patch

 scripts/Makefile.package | 23 +++++++++++++++--------
 scripts/package/mkdebian |  2 +-
 2 files changed, 16 insertions(+), 9 deletions(-)
scripts/Makefile.package +15 -8
index 7266a11855f4..c207f4ec47ac 100644
@@ -4,9 +4,7 @@
 include $(srctree)/scripts/Kbuild.include
 
 KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE))
-KDEB_SOURCENAME ?= linux-upstream
 KBUILD_PKG_ROOTCMD ?="fakeroot -u"
-export KDEB_SOURCENAME
 # Include only those top-level files that are needed by make, plus the GPL copy
 TAR_CONTENT := Documentation LICENSES arch block certs crypto drivers fs \
                include init io_uring ipc kernel lib mm net rust \
@@ -87,17 +85,26 @@ binrpm-pkg:
 	+rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \
 		$(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec
 
+quiet_cmd_debianize = GEN     $@
+      cmd_debianize = $(srctree)/scripts/package/mkdebian
+
+debian: FORCE
+	$(call cmd,debianize)
+
+PHONY += debian-orig
+debian-orig: private source = $(shell dpkg-parsechangelog -S Source)
+debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//')
+debian-orig: private orig-name = $(source)_$(version).orig.tar.gz
+debian-orig: linux.tar.gz debian
+	$(Q)cp $< ../$(orig-name)
+
 PHONY += deb-pkg
-deb-pkg: linux.tar.gz
-	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-	origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-		cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+deb-pkg: debian-orig
 	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
 		--build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
-bindeb-pkg:
-	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
+bindeb-pkg: debian
 	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc
 
 PHONY += intdeb-pkg
scripts/package/mkdebian +1 -1
index 68b13ef590ba..534e9713761a 100755
@@ -95,7 +95,7 @@ else
 	revision=$($srctree/init/build-version)
 	packageversion=$version-$revision
 fi
-sourcename=$KDEB_SOURCENAME
+sourcename=${KDEB_SOURCENAME:-linux-upstream}
 
 if [ "$ARCH" = "um" ] ; then
 	packagename=user-mode-linux
Signature · 1 line
2.34.1
#7 lore raw

PATCH v6 6/12 kbuild: deb-pkg: make .orig tarball a hard link if possible

In reply to Masahiro Yamada

If '..' belongs to the same filesystem, create a hard link instead of a copy. In most cases, you can save disk space.

I do not want to use 'mv' because keeping linux.tar.gz is useful to avoid unneeded rebuilding of the tarball.

---

Changes in v6:
- New patch

 scripts/Makefile.package | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/Makefile.package +5 -1
index c207f4ec47ac..80a96eb61426 100644
@@ -96,7 +96,11 @@ debian-orig: private source = $(shell dpkg-parsechangelog -S Source)
 debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//')
 debian-orig: private orig-name = $(source)_$(version).orig.tar.gz
 debian-orig: linux.tar.gz debian
-	$(Q)cp $< ../$(orig-name)
+	$(Q)if [ "$(df  --output=target .. 2>/dev/null)" = "$(df --output=target $< 2>/dev/null)" ]; then \
+		ln -f $< ../$(orig-name); \
+	else \
+		cp $< ../$(orig-name); \
+	fi
 
 PHONY += deb-pkg
 deb-pkg: debian-orig
Signature · 1 line
2.34.1
Reviewed-by on v6 6/12
#8 lore raw

Re: [PATCH v6 06/12] kbuild: deb-pkg: make .orig tarball a hard link if possible

Quoted 34 lines · On Wed, Feb 15, 2023 at 10:20:28AM +0900 Masahiro Yamada wrote:

If '..' belongs to the same filesystem, create a hard link instead of a copy. In most cases, you can save disk space.

I do not want to use 'mv' because keeping linux.tar.gz is useful to avoid unneeded rebuilding of the tarball.

---

Changes in v6:
- New patch

 scripts/Makefile.package | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/Makefile.package +5 -1
index c207f4ec47ac..80a96eb61426 100644
@@ -96,7 +96,11 @@ debian-orig: private source = $(shell dpkg-parsechangelog -S Source)
 debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//')
 debian-orig: private orig-name = $(source)_$(version).orig.tar.gz
 debian-orig: linux.tar.gz debian
-	$(Q)cp $< ../$(orig-name)
+	$(Q)if [ "$(df  --output=target .. 2>/dev/null)" = "$(df --output=target $< 2>/dev/null)" ]; then \
+		ln -f $< ../$(orig-name); \
+	else \
+		cp $< ../$(orig-name); \
+	fi
 
 PHONY += deb-pkg
 deb-pkg: debian-orig
Signature · 1 line
2.34.1
#9 lore raw

PATCH v6 7/12 kbuild: deb-pkg: switch over to source format 3.0 (quilt)

In reply to Masahiro Yamada

Change the source format from "1.0" to "3.0 (quilt)" because it works more cleanly.

All files except .config and debian/ go into the orig tarball. Add a single patch, debian/patches/config, and delete the ugly extend-diff-ignore patterns.

The debian tarball will be compressed into *.debian.tar.xz by default. If you like to use a different compression mode, you can pass the command line option, DPKG_FLAGS=-Zgzip, for example.

The orig tarball only supports gzip for now. The combination of gzip and xz is somewhat clumsy, but it is not a practical problem.

---

Changes in v6:
- Fix error when KCONFIG_CONFIG is set

Changes in v4:
- New patch

 scripts/Makefile.package |  2 +-
 scripts/package/mkdebian | 41 ++++++++++++++++++----------------------
 2 files changed, 19 insertions(+), 24 deletions(-)
scripts/Makefile.package +1 -1
index 80a96eb61426..5538deacb136 100644
@@ -105,7 +105,7 @@ debian-orig: linux.tar.gz debian
 PHONY += deb-pkg
 deb-pkg: debian-orig
 	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
-		--build=source,binary --source-option=-sP -nc -us -uc
+		--build=source,binary -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg: debian
scripts/package/mkdebian +18 -23

47 lines not on this page · raw message

Signature · 1 line
2.34.1
Reviewed-by on v6 7/12
#10 lore raw

Re: [PATCH v6 07/12] kbuild: deb-pkg: switch over to source format 3.0 (quilt)

Quoted 11 lines · On Wed, Feb 15, 2023 at 10:20:29AM +0900 Masahiro Yamada wrote:

Change the source format from "1.0" to "3.0 (quilt)" because it works more cleanly.

All files except .config and debian/ go into the orig tarball. Add a single patch, debian/patches/config, and delete the ugly extend-diff-ignore patterns.

The debian tarball will be compressed into *.debian.tar.xz by default. If you like to use a different compression mode, you can pass the command line option, DPKG_FLAGS=-Zgzip, for example.

Quoted 5 lines

The orig tarball only supports gzip for now. The combination of gzip and xz is somewhat clumsy, but it is not a practical problem.

---

Thanks, that switch to 3.0/quilt sounds really good to me.

Quoted 60 lines

Changes in v6:
- Fix error when KCONFIG_CONFIG is set

Changes in v4:
- New patch

 scripts/Makefile.package |  2 +-
 scripts/package/mkdebian | 41 ++++++++++++++++++----------------------
 2 files changed, 19 insertions(+), 24 deletions(-)
scripts/Makefile.package +1 -1
index 80a96eb61426..5538deacb136 100644
@@ -105,7 +105,7 @@ debian-orig: linux.tar.gz debian
 PHONY += deb-pkg
 deb-pkg: debian-orig
 	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
-		--build=source,binary --source-option=-sP -nc -us -uc
+		--build=source,binary -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg: debian
scripts/package/mkdebian +5 -23
index 534e9713761a..22fc73a5e4f2 100755
@@ -135,29 +135,24 @@ else
 fi
 
 mkdir -p debian/source/
-echo "1.0" > debian/source/format
-
-# Ugly: ignore anything except .config or debian/
-cat<<'EOF' > debian/source/local-options
-diff-ignore
-
-extend-diff-ignore = ^[^.d]
-
-extend-diff-ignore = ^\.[^c]
-extend-diff-ignore = ^\.c($|[^o])
-extend-diff-ignore = ^\.co($|[^n])
-extend-diff-ignore = ^\.con($|[^f])
-extend-diff-ignore = ^\.conf($|[^i])
-extend-diff-ignore = ^\.confi($|[^g])
-extend-diff-ignore = ^\.config.
-
-extend-diff-ignore = ^d($|[^e])
-extend-diff-ignore = ^de($|[^b])
-extend-diff-ignore = ^deb($|[^i])
-extend-diff-ignore = ^debi($|[^a])
-extend-diff-ignore = ^debia($|[^n])
-extend-diff-ignore = ^debian[^/]
-EOF
+echo "3.0 (quilt)" > debian/source/format
+
+{
+	echo "diff-ignore"
+	echo "extend-diff-ignore = .*"

scripts/package/mkdebian else · line 152

Quoted 10 lines
scripts/package/mkdebian +10 -0
+} > debian/source/local-options
+
+# Add .config as a patch
+mkdir -p debian/patches
+{
+	echo "Subject: Add .config"
+	echo "Author: ${maintainer}"
+	echo
+	echo "--- /dev/null"
+	echo "+++ linux/.config"

bike-shedding:

	cat <<-eof
	Subject: Add .config
	Author: ${maintainer}

	--- /dev/null
	+++ linux/.config
	eof
Quoted 8 lines
scripts/package/mkdebian +3 -0
+	diff -u /dev/null "${KCONFIG_CONFIG}" | tail -n +3
+} > debian/patches/config
+echo config > debian/patches/series
 
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
Signature · 1 line
2.34.1
#11 lore raw

Re: [PATCH v6 07/12] kbuild: deb-pkg: switch over to source format 3.0 (quilt)

Quoted 10 lines · Nicolas Schier

bike-shedding:

        cat <<-eof
        Subject: Add .config
        Author: ${maintainer}

        --- /dev/null
        +++ linux/.config
        eof

I agree this is more readable, but I used echo to
avoid a process fork.

'echo' is a shell built-in for bash and dash but 'cat' is not.

So, this is a trade-off between readability vs
one less process fork.

If the code were much longer, I would prefer 'cat'.

Quoted 11 lines
Quoted 9 lines
scripts/package/mkdebian +3 -1
+     diff -u /dev/null "${KCONFIG_CONFIG}" | tail -n +3
+} > debian/patches/config
+echo config > debian/patches/series
 
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
--

2.34.1

--
Best Regards
Masahiro Yamada

#12 lore raw

PATCH v6 8/12 kbuild: make perf-tar*-src-pkg work without relying on git

In reply to Masahiro Yamada

Currently, perf-tar*-src-pkg only uses 'git archive', but it is better to make it work without relying on git.

The file, HEAD, which saves the commit hash, will be included in the tarball only when the source tree is managed by git. The git tree is more precisely checked; it has been copied from scripts/setlocalversion.

---

(no changes since v5)

Changes in v5:
- New patch

 scripts/Makefile.package | 97 ++++++++++++++++++++++++++--------------
 1 file changed, 64 insertions(+), 33 deletions(-)
scripts/Makefile.package +64 -33
index 5538deacb136..50b86b325d7a 100644
@@ -50,10 +50,25 @@ quiet_cmd_tar = TAR     $@
 
 tar-rootdir := $(srctree)
 
+%.tar:
+	$(call cmd,tar)
+
 %.tar.gz: private tar-compress-opt := -I $(KGZIP)
 %.tar.gz:
 	$(call cmd,tar)
 
+%.tar.bz2: private tar-compress-opt := -I $(KBZIP2)
+%.tar.bz2:
+	$(call cmd,tar)
+
+%.tar.xz: private tar-compress-opt := -I $(XZ)
+%.tar.xz:
+	$(call cmd,tar)
+
+%.tar.zst: private tar-compress-opt := -I $(ZSTD)
+%.tar.zst:
+	$(call cmd,tar)
+
 # Linux source tarball
 # ---------------------------------------------------------------------------
 
@@ -138,36 +153,52 @@ $(tar-pkgs):
 	$(MAKE) -f $(srctree)/Makefile
 	+$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@
 
-# perf-pkg - generate a source tarball with perf source
+# perf-tar*-src-pkg - generate a source tarball with perf source
 # ---------------------------------------------------------------------------
 
-perf-tar=perf-$(KERNELVERSION)
-
-quiet_cmd_perf_tar = TAR
-      cmd_perf_tar = \
-git --git-dir=$(srctree)/.git archive --prefix=$(perf-tar)/         \
-	HEAD^{tree} $$(cd $(srctree);                               \
-		       echo $$(cat tools/perf/MANIFEST)) \
-	-o $(perf-tar).tar;                                         \
-mkdir -p $(perf-tar);                                               \
-git --git-dir=$(srctree)/.git rev-parse HEAD > $(perf-tar)/HEAD;    \
-(cd $(srctree)/tools/perf;                                          \
-util/PERF-VERSION-GEN $(CURDIR)/$(perf-tar)/);              \
-tar rf $(perf-tar).tar $(perf-tar)/HEAD $(perf-tar)/PERF-VERSION-FILE; \
-rm -r $(perf-tar);                                                  \
-$(if $(findstring tar-src,$@),,                                     \
-$(if $(findstring bz2,$@),$(KBZIP2),                                 \
-$(if $(findstring gz,$@),$(KGZIP),                                  \
-$(if $(findstring xz,$@),$(XZ),                                     \
-$(if $(findstring zst,$@),$(ZSTD),                                  \
-$(error unknown target $@)))))                                      \
-	-f -9 $(perf-tar).tar)
-
-perf-tar-pkgs := perf-tar-src-pkg perf-targz-src-pkg perf-tarbz2-src-pkg \
-		 perf-tarxz-src-pkg perf-tarzst-src-pkg
-PHONY += $(perf-tar-pkgs)
-$(perf-tar-pkgs):
-	$(call cmd,perf_tar)
+perf-tar-src-pkg-tarball = perf-$(KERNELVERSION).$(1)
+perf-tar-src-pkg-phony   = perf-$(subst .,,$(1))-src-pkg
+
+quiet_cmd_stage_perf_src = STAGE   $@
+      cmd_stage_perf_src = \
+	rm -rf $@; \
+	mkdir -p $@; \
+	tar -c -f - --exclude-from=$<_exclude -C $(srctree) --files-from=$(srctree)/tools/perf/MANIFEST | \
+	tar -x -f - -C $@
+
+.tmp_perf: .tmp_filelist
+	$(call cmd,stage_perf_src)
+
+filechk_perf_head = \
+	if test -z "$(git -C $(srctree) rev-parse --show-cdup 2>/dev/null)" && \
+	       head=$$(git -C $(srctree) rev-parse --verify HEAD 2>/dev/null); then \
+		echo $$head; \
+	else \
+		echo "not a git tree"; \
+	fi
+
+.tmp_perf/HEAD: .tmp_perf FORCE
+	$(call filechk,perf_head)
+
+quiet_cmd_perf_version_file = GEN     $@
+      cmd_perf_version_file = cd $(srctree)/tools/perf; util/PERF-VERSION-GEN $(dir $(abspath $@))
+
+# PERF-VERSION-FILE and HEAD are independent, but this avoids updating the
+# timestamp of PERF-VERSION-FILE.
+# The best is to fix tools/perf/util/PERF-VERSION-GEN.
+.tmp_perf/PERF-VERSION-FILE: .tmp_perf/HEAD $(srctree)/tools/perf/util/PERF-VERSION-GEN
+	$(call cmd,perf_version_file)
+
+define perf-tar-src-pkg-rule
+PHONY += $(perf-tar-src-pkg-phony)
+$(perf-tar-src-pkg-phony): $(perf-tar-src-pkg-tarball)
+	@:
+
+$(perf-tar-src-pkg-tarball): private tar-rootdir := .tmp_perf
+$(perf-tar-src-pkg-tarball): .tmp_filelist .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE
+endef
+
+$(foreach x, tar tar.gz tar.bz2 tar.xz tar.zst, $(eval $(call perf-tar-src-pkg-rule,$(x))))
 
 # Help text displayed when executing 'make help'
 # ---------------------------------------------------------------------------
@@ -186,11 +217,11 @@ help:
 	@echo '  tarbz2-pkg          - Build the kernel as a bzip2 compressed tarball'
 	@echo '  tarxz-pkg           - Build the kernel as a xz compressed tarball'
 	@echo '  tarzst-pkg          - Build the kernel as a zstd compressed tarball'
-	@echo '  perf-tar-src-pkg    - Build $(perf-tar).tar source tarball'
-	@echo '  perf-targz-src-pkg  - Build $(perf-tar).tar.gz source tarball'
-	@echo '  perf-tarbz2-src-pkg - Build $(perf-tar).tar.bz2 source tarball'
-	@echo '  perf-tarxz-src-pkg  - Build $(perf-tar).tar.xz source tarball'
-	@echo '  perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
+	@echo '  perf-tar-src-pkg    - Build the perf source tarball with no compression'
+	@echo '  perf-targz-src-pkg  - Build the perf source tarball with gzip compression'
+	@echo '  perf-tarbz2-src-pkg - Build the perf source tarball with bz2 compression'
+	@echo '  perf-tarxz-src-pkg  - Build the perf source tarball with xz compression'
+	@echo '  perf-tarzst-src-pkg - Build the perf source tarball with zst compression'
 
 PHONY += FORCE
 FORCE:
Signature · 1 line
2.34.1
#13 lore raw

PATCH v6 9/12 kbuild: tar-pkg: use tar rules in scripts/Makefile.package

In reply to Masahiro Yamada

Use %.tar, %.tar.gz, %.tar.bz2, %.tar.xz, %.tar.zst rules in scripts/Makefile.package.

---

Changes in v6:
- New patch

 scripts/Makefile.package | 29 +++++++++++++++++-----
 scripts/package/buildtar | 52 +---------------------------------------
 2 files changed, 24 insertions(+), 57 deletions(-)
scripts/Makefile.package +23 -6
index 50b86b325d7a..b941e6341b36 100644
@@ -145,13 +145,30 @@ snap-pkg:
 	cd $(objtree)/snap && \
 	snapcraft --target-arch=$(UTS_MACHINE)
 
-# tarball targets
+# dir-pkg tar*-pkg - tarball targets
 # ---------------------------------------------------------------------------
-tar-pkgs := dir-pkg tar-pkg targz-pkg tarbz2-pkg tarxz-pkg tarzst-pkg
-PHONY += $(tar-pkgs)
-$(tar-pkgs):
-	$(MAKE) -f $(srctree)/Makefile
-	+$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@
+
+tar-pkg-tarball = linux-$(KERNELRELEASE)-$(ARCH).$(1)
+tar-pkg-phony = $(subst .,,$(1))-pkg
+
+tar-install: FORCE
+	$(Q)$(MAKE) -f $(srctree)/Makefile
+	+$(Q)$(srctree)/scripts/package/buildtar $@
+
+PHONY += dir-pkg
+dir-pkg: tar-install
+	@echo "Kernel tree successfully created in $<"
+
+define tar-pkg-rule
+PHONY += $(tar-pkg-phony)
+$(tar-pkg-phony): $(tar-pkg-tarball)
+	@:
+
+$(tar-pkg-tarball): private tar-rootdir := tar-install
+$(tar-pkg-tarball): tar-install
+endef
+
+$(foreach x, tar tar.gz tar.bz2 tar.xz tar.zst, $(eval $(call tar-pkg-rule,$(x))))
 
 # perf-tar*-src-pkg - generate a source tarball with perf source
 # ---------------------------------------------------------------------------
scripts/package/buildtar +1 -51

61 lines not on this page · raw message

Signature · 1 line
2.34.1
#14 lore raw

PATCH v6 10/12 kbuild: deb-pkg: fix binary-arch and clean in debian/rules

In reply to Masahiro Yamada

The clean target needs ARCH=${ARCH} to clean up the tree for the correct architecture. 'make (bin)deb-pkg' skips cleaning, but the preclean hook may be executed if dpkg-buildpackage is directly used.

The binary-arch target does not need KERNELRELEASE because it is not updated during the installation. KBUILD_BUILD_VERSION is not needed either because binary-arch does not build vmlinux.

---

Changes in v6:
- New patch

 scripts/package/mkdebian | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
scripts/package/mkdebian +2 -4
index 22fc73a5e4f2..eef20d5dc288 100755
@@ -251,12 +251,10 @@ build: build-arch
 
 binary-indep:
 binary-arch: build-arch
-	\$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \
-	KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile intdeb-pkg
-
+	\$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} intdeb-pkg
 clean:
 	rm -rf debian/files debian/linux-*
-	\$(MAKE) clean
+	\$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} clean
 
 binary: binary-arch
 EOF
Signature · 1 line
2.34.1
#15 lore raw

PATCH v6 11/12 kbuild: deb-pkg: improve the usability of source package

In reply to Masahiro Yamada

Improve the source package support in case the dpkg-buildpackage is directly used to build binary packages.

For cross-compiling, you can set CROSS_COMPILE via the environment variable, but it is better to set it automatically - set it to ${DEB_HOST_GNU_TYPE}- if we are cross-compiling but not from the top Makefile.

The generated source package may be carried to a different build environment, which may have a different compiler installed. Run olddefconfig first to set new CONFIG options to their default values without prompting.

Take KERNELRELEASE and KBUILD_BUILD_VERSION from the version field of debian/changelog in case it is updated afterwards.

---

Changes in v6:
- New patch

 scripts/package/deb-build-option | 16 ++++++++++++++++
 scripts/package/mkdebian         |  9 ++++-----
 2 files changed, 20 insertions(+), 5 deletions(-)
 create mode 100755 scripts/package/deb-build-option
scripts/package/deb-build-option new file +16 -0
new file mode 100755
index 000000000000..b079b0d121d4
@@ -0,0 +1,16 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-only
+
+# Set up CROSS_COMPILE if we are cross-compiling, but not called from the
+# kernel toplevel Makefile
+if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUILD_ARCH}" ]; then
+	echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}-
+fi
+
+version=$(dpkg-parsechangelog -S Version)
+version_upstream="${version%-*}"
+debian_revision="${version#${version_upstream}}"
+debian_revision="${debian_revision#*-}"
+
+echo KERNELRELEASE=${version_upstream}
+echo KBUILD_BUILD_VERSION=${debian_revision}
scripts/package/mkdebian +4 -5
index eef20d5dc288..f74380036bb5 100755
@@ -90,10 +90,8 @@ rm -rf debian
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
 	packageversion=$KDEB_PKGVERSION
-	revision=${packageversion##*-}
 else
-	revision=$($srctree/init/build-version)
-	packageversion=$version-$revision
+	packageversion=$version-$($srctree/init/build-version)
 fi
 sourcename=${KDEB_SOURCENAME:-linux-upstream}
 
@@ -244,8 +242,9 @@ srctree ?= .
 
 build-indep:
 build-arch:
-	\$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \
-	KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile
+	\$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} \
+	\$(shell \$(srctree)/scripts/package/deb-build-option) \
+	olddefconfig all
 
 build: build-arch
 
Signature · 1 line
2.34.1
#16 lore raw

Re: [PATCH v6 11/12] kbuild: deb-pkg: improve the usability of source package

Quoted 8 lines · Masahiro Yamada

Improve the source package support in case the dpkg-buildpackage is directly used to build binary packages.

For cross-compiling, you can set CROSS_COMPILE via the environment variable, but it is better to set it automatically - set it to ${DEB_HOST_GNU_TYPE}- if we are cross-compiling but not from the top Makefile.

Quoted 7 lines

The generated source package may be carried to a different build environment, which may have a different compiler installed. Run olddefconfig first to set new CONFIG options to their default values without prompting.

Take KERNELRELEASE and KBUILD_BUILD_VERSION from the version field of debian/changelog in case it is updated afterwards.

I just arrived to this patch when trying to update the kernel for SOF from 6.2.0 base to 6.3.0-rc1:
https://github.com/thesofproject/linux/pull/4223#issuecomment-1459997676

Our CI is building a kernel deb package from PRs to test with this command:
make bindeb-pkg LOCALVERSION=-$PRIDV-default KDEB_PKGVERSION=$(make kernelversion)-$GITSHA

The files, directories and the name of the deb package follows the rule set up by scripts/setlocalversion, taking into account the LOCALVERSION, CONFIG_LOCALVERSION and CONFIG_LOCALVERSION_AUTO, but the built kernel is using the version derived from KDEB_PKGVERSION and the system is not able to boot up since the modules are in a different directory then they should be.

Something like this:
make bindeb-pkg LOCALVERSION=-testbuild vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty /lib/modules/6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/

strings vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty | grep 6.3 6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty

Then with the KDEB_PKGVERSION:
make bindeb-pkg LOCALVERSION=-testbuild KDEB_PKGVERSION=$(make kernelversion)-notworking vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty /lib/modules/6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/

strings vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty | grep 6.3 6.3.0-rc1

in this case modprobe will not find the modules, it is looking for them under /lib/modules/6.3.0-rc1

Reverting this patch makes the created deb package usable, the system is bootable.

We use the LOCALVERSION to feed in PR inforamtion and the KDEB_PKGVERSION is used fro additional information, like build ID.

Do you have any suggestion on how to get the functionality back or we must stop using the KDEB_PKGVERSION altogether?

Thank you,

Signature · 1 line
Péter
Quoted 62 lines

---

Changes in v6:
- New patch

 scripts/package/deb-build-option | 16 ++++++++++++++++
 scripts/package/mkdebian         |  9 ++++-----
 2 files changed, 20 insertions(+), 5 deletions(-)
 create mode 100755 scripts/package/deb-build-option
scripts/package/deb-build-option new file +16 -0
new file mode 100755
index 000000000000..b079b0d121d4
@@ -0,0 +1,16 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-only
+
+# Set up CROSS_COMPILE if we are cross-compiling, but not called from the
+# kernel toplevel Makefile
+if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUILD_ARCH}" ]; then
+	echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}-
+fi
+
+version=$(dpkg-parsechangelog -S Version)
+version_upstream="${version%-*}"
+debian_revision="${version#${version_upstream}}"
+debian_revision="${debian_revision#*-}"
+
+echo KERNELRELEASE=${version_upstream}
+echo KBUILD_BUILD_VERSION=${debian_revision}
scripts/package/mkdebian +4 -5
index eef20d5dc288..f74380036bb5 100755
@@ -90,10 +90,8 @@ rm -rf debian
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
 	packageversion=$KDEB_PKGVERSION
-	revision=${packageversion##*-}
 else
-	revision=$($srctree/init/build-version)
-	packageversion=$version-$revision
+	packageversion=$version-$($srctree/init/build-version)
 fi
 sourcename=${KDEB_SOURCENAME:-linux-upstream}
 
@@ -244,8 +242,9 @@ srctree ?= .
 
 build-indep:
 build-arch:
-	\$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \
-	KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile
+	\$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} \
+	\$(shell \$(srctree)/scripts/package/deb-build-option) \
+	olddefconfig all
 
 build: build-arch
 
Signature · 1 line
Péter
#17 lore raw

Re: [PATCH v6 11/12] kbuild: deb-pkg: improve the usability of source package

Quoted 55 lines · Péter Ujfalusi
Quoted 8 lines · Masahiro Yamada

Improve the source package support in case the dpkg-buildpackage is directly used to build binary packages.

For cross-compiling, you can set CROSS_COMPILE via the environment variable, but it is better to set it automatically - set it to ${DEB_HOST_GNU_TYPE}- if we are cross-compiling but not from the top Makefile.

Quoted 7 lines

The generated source package may be carried to a different build environment, which may have a different compiler installed. Run olddefconfig first to set new CONFIG options to their default values without prompting.

Take KERNELRELEASE and KBUILD_BUILD_VERSION from the version field of debian/changelog in case it is updated afterwards.

I just arrived to this patch when trying to update the kernel for SOF from 6.2.0 base to 6.3.0-rc1:
https://github.com/thesofproject/linux/pull/4223#issuecomment-1459997676

Our CI is building a kernel deb package from PRs to test with this command:
make bindeb-pkg LOCALVERSION=-$PRIDV-default KDEB_PKGVERSION=$(make kernelversion)-$GITSHA

The files, directories and the name of the deb package follows the rule set up by scripts/setlocalversion, taking into account the LOCALVERSION, CONFIG_LOCALVERSION and CONFIG_LOCALVERSION_AUTO, but the built kernel is using the version derived from KDEB_PKGVERSION and the system is not able to boot up since the modules are in a different directory then they should be.

Something like this:
make bindeb-pkg LOCALVERSION=-testbuild vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty /lib/modules/6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/

strings vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty | grep 6.3 6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty

Then with the KDEB_PKGVERSION:
make bindeb-pkg LOCALVERSION=-testbuild KDEB_PKGVERSION=$(make kernelversion)-notworking vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty /lib/modules/6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/

strings vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty | grep 6.3 6.3.0-rc1

in this case modprobe will not find the modules, it is looking for them under /lib/modules/6.3.0-rc1

Quoted 8 lines

Reverting this patch makes the created deb package usable, the system is bootable.

We use the LOCALVERSION to feed in PR inforamtion and the KDEB_PKGVERSION is used fro additional information, like build ID.

Do you have any suggestion on how to get the functionality back or we must stop using the KDEB_PKGVERSION altogether?

I am aware of this issue.

KERNELVERSION does not need to match the package version.

I should have taken it from the <abiver> in the package name (linux-image-<abiver>_<origver>-<rev>_<arch>.deb) if I had wanted to do this dynamically by debian/control.

I wrongly took it from <origver>.

I will change this when I solve another issue required by Linus Tolvalds.

Please wait.

Quoted 70 lines

Thank you,
--
Péter

Quoted 52 lines

---

Changes in v6:
- New patch

 scripts/package/deb-build-option | 16 ++++++++++++++++
 scripts/package/mkdebian         |  9 ++++-----
 2 files changed, 20 insertions(+), 5 deletions(-)
 create mode 100755 scripts/package/deb-build-option
scripts/package/deb-build-option new file +16 -0
new file mode 100755
index 000000000000..b079b0d121d4
@@ -0,0 +1,16 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-only
+
+# Set up CROSS_COMPILE if we are cross-compiling, but not called from the
+# kernel toplevel Makefile
+if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUILD_ARCH}" ]; then
+     echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}-
+fi
+
+version=$(dpkg-parsechangelog -S Version)
+version_upstream="${version%-*}"
+debian_revision="${version#${version_upstream}}"
+debian_revision="${debian_revision#*-}"
+
+echo KERNELRELEASE=${version_upstream}
+echo KBUILD_BUILD_VERSION=${debian_revision}
scripts/package/mkdebian +1 -3
index eef20d5dc288..f74380036bb5 100755
@@ -90,10 +90,8 @@ rm -rf debian
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
      packageversion=$KDEB_PKGVERSION
-     revision=${packageversion##*-}
 else
-     revision=$($srctree/init/build-version)
-     packageversion=$version-$revision
+     packageversion=$version-$($srctree/init/build-version)
 fi
 sourcename=${KDEB_SOURCENAME:-linux-upstream}
 
@@ -244,8 +242,9 @@ srctree ?= .
 

scripts/package/mkdebian srctree ?= . · line 250

Quoted 10 lines
scripts/package/mkdebian +3 -2
 build-indep:
 build-arch:
-     \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \
-     KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile
+     \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} \
+     \$(shell \$(srctree)/scripts/package/deb-build-option) \
+     olddefconfig all
 
 build: build-arch
 

--
Péter

Signature · 2 lines
Best Regards
Masahiro Yamada
#18 lore raw

PATCH v6 12/12 kbuild: add srcdeb-pkg target

In reply to Masahiro Yamada

This new target builds only the debian source package.

Factor similar code out to the 'build-deb-pkg' variable.

---

Changes in v6:
- New patch

 scripts/Makefile.package | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
scripts/Makefile.package +20 -3
index b941e6341b36..8650d9432c26 100644
@@ -100,6 +100,9 @@ binrpm-pkg:
 	+rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \
 		$(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec
 
+# deb-pkg srcdeb-pkg bindeb-pkg
+# ---------------------------------------------------------------------------
+
 quiet_cmd_debianize = GEN     $@
       cmd_debianize = $(srctree)/scripts/package/mkdebian
 
@@ -117,14 +120,27 @@ debian-orig: linux.tar.gz debian
 		cp $< ../$(orig-name); \
 	fi
 
+build-deb-pkg = \
+        dpkg-buildpackage \
+        --build=$(build-type) --no-pre-clean --unsigned-changes \
+        $(if $(findstring source, $(build-type)), --unsigned-source) \
+        $(if $(findstring binary, $(build-type)), -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch)) \
+        $(DPKG_FLAGS)
+
 PHONY += deb-pkg
+deb-pkg: private build-type := source,binary
 deb-pkg: debian-orig
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
-		--build=source,binary -nc -us -uc
+	+$(build-deb-pkg)
+
+PHONY += srcdeb-pkg
+srcdeb-pkg: private build-type := source
+srcdeb-pkg: debian-orig
+	+$(build-deb-pkg)
 
 PHONY += bindeb-pkg
+bindeb-pkg: private build-type := binary
 bindeb-pkg: debian
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc
+	+$(build-deb-pkg)
 
 PHONY += intdeb-pkg
 intdeb-pkg:
@@ -225,6 +241,7 @@ help:
 	@echo '  srcrpm-pkg          - Build only the source kernel RPM package'
 	@echo '  binrpm-pkg          - Build only the binary kernel RPM package'
 	@echo '  deb-pkg             - Build both source and binary deb kernel packages'
+	@echo '  binsrc-pkg          - Build only the source kernel deb package'
 	@echo '  bindeb-pkg          - Build only the binary kernel deb package'
 	@echo '  snap-pkg            - Build only the binary kernel snap package'
 	@echo '                        (will connect to external hosts)'
Signature · 1 line
2.34.1
Reviewed-by on v6 12/12
#19 lore raw

Re: [PATCH v6 12/12] kbuild: add srcdeb-pkg target

Quoted 59 lines · On Wed, Feb 15, 2023 at 10:20:34AM +0900 Masahiro Yamada wrote:

This new target builds only the debian source package.

Factor similar code out to the 'build-deb-pkg' variable.

---

Changes in v6:
- New patch

 scripts/Makefile.package | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
scripts/Makefile.package +19 -3
index b941e6341b36..8650d9432c26 100644
@@ -100,6 +100,9 @@ binrpm-pkg:
 	+rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \
 		$(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec
 
+# deb-pkg srcdeb-pkg bindeb-pkg
+# ---------------------------------------------------------------------------
+
 quiet_cmd_debianize = GEN     $@
       cmd_debianize = $(srctree)/scripts/package/mkdebian
 
@@ -117,14 +120,27 @@ debian-orig: linux.tar.gz debian
 		cp $< ../$(orig-name); \
 	fi
 
+build-deb-pkg = \
+        dpkg-buildpackage \
+        --build=$(build-type) --no-pre-clean --unsigned-changes \
+        $(if $(findstring source, $(build-type)), --unsigned-source) \
+        $(if $(findstring binary, $(build-type)), -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch)) \
+        $(DPKG_FLAGS)
+
 PHONY += deb-pkg
+deb-pkg: private build-type := source,binary
 deb-pkg: debian-orig
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
-		--build=source,binary -nc -us -uc
+	+$(build-deb-pkg)
+
+PHONY += srcdeb-pkg
+srcdeb-pkg: private build-type := source
+srcdeb-pkg: debian-orig
+	+$(build-deb-pkg)
 
 PHONY += bindeb-pkg
+bindeb-pkg: private build-type := binary
 bindeb-pkg: debian
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc
+	+$(build-deb-pkg)
 
 PHONY += intdeb-pkg
 intdeb-pkg:

scripts/Makefile.package help: · line 244

Quoted 4 lines
scripts/Makefile.package +1 -0
@@ -225,6 +241,7 @@ help:
 	@echo '  srcrpm-pkg          - Build only the source kernel RPM package'
 	@echo '  binrpm-pkg          - Build only the binary kernel RPM package'
 	@echo '  deb-pkg             - Build both source and binary deb kernel packages'
+	@echo '  binsrc-pkg          - Build only the source kernel deb package'

typo: srcdeb-dpkg

With this typo fixed:

Quoted 5 lines
scripts/Makefile.package +0 -0
 	@echo '  bindeb-pkg          - Build only the binary kernel deb package'
 	@echo '  snap-pkg            - Build only the binary kernel snap package'
 	@echo '                        (will connect to external hosts)'
Signature · 1 line
2.34.1

Thanks for the patch (and the whole patch set), I appreciate it!

While testing, I stumbled over one thing, not really related to this patch:

As tar complains about any kind of tree changes while it creates an archive, 'olddefconfig' and 'srcdeb-pkg' must not be build at the same time. Is it sensible to add another 'mixed-build' check in top-level Makefile against all target depending on linux.tar.gz?

    $ make olddefconfig srcdeb-pkg -j8
    #
    # No change to .config
    #
      GEN     debian
      UPD     .tmp_filelist
      TAR     linux.tar.gz
    tar: .: file changed as we read it
    make[2]: *** [scripts/Makefile.package:58: linux.tar.gz] Error 1
    make[2]: *** Deleting file 'linux.tar.gz'
    make[1]: *** [Makefile:1654: srcdeb-pkg] Error 2
    make: *** [Makefile:358: __build_one_by_one] Error 2

Kind regards,
Nicolas

#20 lore raw

Re: [PATCH v6 12/12] kbuild: add srcdeb-pkg target

Quoted 58 lines · Nicolas Schier
Quoted 56 lines · On Wed, Feb 15, 2023 at 10:20:34AM +0900 Masahiro Yamada wrote:

This new target builds only the debian source package.

Factor similar code out to the 'build-deb-pkg' variable.

---

Changes in v6:
- New patch

 scripts/Makefile.package | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
scripts/Makefile.package +19 -3
index b941e6341b36..8650d9432c26 100644
@@ -100,6 +100,9 @@ binrpm-pkg:
      +rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \
              $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec
 
+# deb-pkg srcdeb-pkg bindeb-pkg
+# ---------------------------------------------------------------------------
+
 quiet_cmd_debianize = GEN     $@
       cmd_debianize = $(srctree)/scripts/package/mkdebian
 
@@ -117,14 +120,27 @@ debian-orig: linux.tar.gz debian
              cp $< ../$(orig-name); \
      fi
 
+build-deb-pkg = \
+        dpkg-buildpackage \
+        --build=$(build-type) --no-pre-clean --unsigned-changes \
+        $(if $(findstring source, $(build-type)), --unsigned-source) \
+        $(if $(findstring binary, $(build-type)), -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch)) \
+        $(DPKG_FLAGS)
+
 PHONY += deb-pkg
+deb-pkg: private build-type := source,binary
 deb-pkg: debian-orig
-     +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
-             --build=source,binary -nc -us -uc
+     +$(build-deb-pkg)
+
+PHONY += srcdeb-pkg
+srcdeb-pkg: private build-type := source
+srcdeb-pkg: debian-orig
+     +$(build-deb-pkg)
 
 PHONY += bindeb-pkg
+bindeb-pkg: private build-type := binary
 bindeb-pkg: debian
-     +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc
+     +$(build-deb-pkg)
 
Quoted 9 lines

scripts/Makefile.package help: · line 244

Quoted 7 lines
scripts/Makefile.package +1 -0
 PHONY += intdeb-pkg
 intdeb-pkg:
@@ -225,6 +241,7 @@ help:
      @echo '  srcrpm-pkg          - Build only the source kernel RPM package'
      @echo '  binrpm-pkg          - Build only the binary kernel RPM package'
      @echo '  deb-pkg             - Build both source and binary deb kernel packages'
+     @echo '  binsrc-pkg          - Build only the source kernel deb package'

typo: srcdeb-dpkg

I will fix. Thanks.

Quoted 12 lines

With this typo fixed:

Quoted 5 lines
scripts/Makefile.package +0 -1
      @echo '  bindeb-pkg          - Build only the binary kernel deb package'
      @echo '  snap-pkg            - Build only the binary kernel snap package'
      @echo '                        (will connect to external hosts)'
--

2.34.1

Thanks for the patch (and the whole patch set), I appreciate it!

Quoted 8 lines

While testing, I stumbled over one thing, not really related to this patch:

As tar complains about any kind of tree changes while it creates an archive, 'olddefconfig' and 'srcdeb-pkg' must not be build at the same time. Is it sensible to add another 'mixed-build' check in top-level Makefile against all target depending on linux.tar.gz?

    $ make olddefconfig srcdeb-pkg -j8

This is already done.

The mixture of config targets and non-config targets are treated as mixed-build. [1]
So, Kbuild internally divides it into 'make olddefconfig -j8' followed by 'make srcdeb-pkg -j8'.

[1] https://github.com/torvalds/linux/blob/v6.2/Makefile#L335

Quoted 11 lines
    #
    # No change to .config
    #
      GEN     debian
      UPD     .tmp_filelist
      TAR     linux.tar.gz
    tar: .: file changed as we read it
    make[2]: *** [scripts/Makefile.package:58: linux.tar.gz] Error 1
    make[2]: *** Deleting file 'linux.tar.gz'
    make[1]: *** [Makefile:1654: srcdeb-pkg] Error 2
    make: *** [Makefile:358: __build_one_by_one] Error 2

I ran the same command, but I could not reproduce this.

Quoted 3 lines

Kind regards,
Nicolas

Signature · 2 lines
Best Regards
Masahiro Yamada
#21 lore raw

Re: [PATCH v6 12/12] kbuild: add srcdeb-pkg target

Quoted 10 lines · Masahiro Yamada
Quoted 8 lines · Nicolas Schier
Quoted 6 lines · On Wed, Feb 15, 2023 at 10:20:34AM +0900 Masahiro Yamada wrote:

This new target builds only the debian source package.

Factor similar code out to the 'build-deb-pkg' variable.

---

[...]

Quoted 8 lines
Quoted 8 lines

While testing, I stumbled over one thing, not really related to this patch:

As tar complains about any kind of tree changes while it creates an archive, 'olddefconfig' and 'srcdeb-pkg' must not be build at the same time. Is it sensible to add another 'mixed-build' check in top-level Makefile against all target depending on linux.tar.gz?

Quoted 8 lines
Quoted 1 line
    $ make olddefconfig srcdeb-pkg -j8

This is already done.

The mixture of config targets and non-config targets are treated as mixed-build. [1]
So, Kbuild internally divides it into 'make olddefconfig -j8' followed by 'make srcdeb-pkg -j8'.

oh yes, and already for quite some time. I didn't check carefully enough.

Quoted 20 lines

[1] https://github.com/torvalds/linux/blob/v6.2/Makefile#L335

Quoted 11 lines
    #
    # No change to .config
    #
      GEN     debian
      UPD     .tmp_filelist
      TAR     linux.tar.gz
    tar: .: file changed as we read it
    make[2]: *** [scripts/Makefile.package:58: linux.tar.gz] Error 1
    make[2]: *** Deleting file 'linux.tar.gz'
    make[1]: *** [Makefile:1654: srcdeb-pkg] Error 2
    make: *** [Makefile:358: __build_one_by_one] Error 2
Quoted 1 line

I ran the same command, but I could not reproduce this.

I cannot reproduce it any more; I don't know what might had been the problem on my site.

Kind regards,
Nicolas

#22 lore raw
In reply to Masahiro Yamada
Quoted 112 lines · Masahiro Yamada

In short, the motivation of this commit is to build a source package without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.

Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.

In fact, there are some ways to achieve this.

The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses it, but I do not like it because it works only when the source tree is managed by git, and all files you want in the tarball must be committed in advance.

I want to make it work without relying on git. We can do this.

Files that are ignored by git are generated files, so should be excluded from the source tarball. We can list them out by parsing the .gitignore files. Of course, .gitignore does not cover all the cases, but it works well enough.

tar(1) claims to support it:

--exclude-vcs-ignores

Exclude files that match patterns read from VCS-specific ignore files. Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.

The best scenario would be to use 'tar --exclude-vcs-ignores', but this option does not work. --exclude-vcs-ignore does not understand any of the negation (!), preceding slash, following slash, etc.. So, this option is just useless.

Hence, I wrote this gitignore parser. The previous version [1], written in Python, was so slow. This version is implemented in C, so it works much faster.

I imported the code from git (commit: 23c56f7bd5f1), so we get the same result.

This tool traverses the source tree, parsing all .gitignore files, and prints file paths that are ignored by git.

The output is similar to 'git ls-files --ignored --directory --others --exclude-per-directory=.gitignore', except

[1] Not sorted
[2] No trailing slash for directories

[2] is intentional because tar's --exclude-from option cannot handle trailing slashes.

[How to test this tool]

  $ git clean -dfx
  $ make -s -j$(nproc) defconfig all                       # or allmodconifg or whatever
  $ git archive -o ../linux1.tar --prefix=./ HEAD
  $ tar tf ../linux1.tar | LANG=C sort > ../file-list1     # files emitted by 'git archive'
  $ make scripts_package
    HOSTCC  scripts/list-gitignored
  $ scripts/list-gitignored  --prefix=./ -o ../exclude-list
  $ tar cf ../linux2.tar --exclude-from=../exclude-list .
  $ tar tf ../linux2.tar | LANG=C sort > ../file-list2     # files emitted by 'tar'
  $ diff  ../file-list1 ../file-list2 | grep -E '^(<|>)'
  < ./Documentation/devicetree/bindings/.yamllint
  < ./drivers/clk/.kunitconfig
  < ./drivers/gpu/drm/tests/.kunitconfig
  < ./drivers/hid/.kunitconfig
  < ./fs/ext4/.kunitconfig
  < ./fs/fat/.kunitconfig
  < ./kernel/kcsan/.kunitconfig
  < ./lib/kunit/.kunitconfig
  < ./mm/kfence/.kunitconfig
  < ./tools/testing/selftests/arm64/tags/
  < ./tools/testing/selftests/arm64/tags/.gitignore
  < ./tools/testing/selftests/arm64/tags/Makefile
  < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
  < ./tools/testing/selftests/arm64/tags/tags_test.c
  < ./tools/testing/selftests/kvm/.gitignore
  < ./tools/testing/selftests/kvm/Makefile
  < ./tools/testing/selftests/kvm/config
  < ./tools/testing/selftests/kvm/settings

The source tarball contains most of files that are tracked by git. You see some diffs, but it is just because some .gitignore files are wrong.

  $ git ls-files -i -c --exclude-per-directory=.gitignore
  Documentation/devicetree/bindings/.yamllint
  drivers/clk/.kunitconfig
  drivers/gpu/drm/tests/.kunitconfig
  drivers/hid/.kunitconfig
  fs/ext4/.kunitconfig
  fs/fat/.kunitconfig
  kernel/kcsan/.kunitconfig
  lib/kunit/.kunitconfig
  mm/kfence/.kunitconfig
  tools/testing/selftests/arm64/tags/.gitignore
  tools/testing/selftests/arm64/tags/Makefile
  tools/testing/selftests/arm64/tags/run_tags_test.sh
  tools/testing/selftests/arm64/tags/tags_test.c
  tools/testing/selftests/kvm/.gitignore
  tools/testing/selftests/kvm/Makefile
  tools/testing/selftests/kvm/config
  tools/testing/selftests/kvm/settings
Quoted 4 lines

[1]: https://lore.kernel.org/all/[email protected]/

---

01 - 11 applied to linux-kbuild.

I will update 12/12 (postponed to v6.4)

Signature · 2 lines
Best Regards
Masahiro Yamada
#23 lore raw
Quoted 115 lines · Masahiro Yamada
Quoted 114 lines · Masahiro Yamada

In short, the motivation of this commit is to build a source package without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.

Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.

In fact, there are some ways to achieve this.

The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses it, but I do not like it because it works only when the source tree is managed by git, and all files you want in the tarball must be committed in advance.

I want to make it work without relying on git. We can do this.

Files that are ignored by git are generated files, so should be excluded from the source tarball. We can list them out by parsing the .gitignore files. Of course, .gitignore does not cover all the cases, but it works well enough.

tar(1) claims to support it:

--exclude-vcs-ignores

Exclude files that match patterns read from VCS-specific ignore files. Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.

The best scenario would be to use 'tar --exclude-vcs-ignores', but this option does not work. --exclude-vcs-ignore does not understand any of the negation (!), preceding slash, following slash, etc.. So, this option is just useless.

Hence, I wrote this gitignore parser. The previous version [1], written in Python, was so slow. This version is implemented in C, so it works much faster.

I imported the code from git (commit: 23c56f7bd5f1), so we get the same result.

This tool traverses the source tree, parsing all .gitignore files, and prints file paths that are ignored by git.

The output is similar to 'git ls-files --ignored --directory --others --exclude-per-directory=.gitignore', except

[1] Not sorted
[2] No trailing slash for directories

[2] is intentional because tar's --exclude-from option cannot handle trailing slashes.

[How to test this tool]

  $ git clean -dfx
  $ make -s -j$(nproc) defconfig all                       # or allmodconifg or whatever
  $ git archive -o ../linux1.tar --prefix=./ HEAD
  $ tar tf ../linux1.tar | LANG=C sort > ../file-list1     # files emitted by 'git archive'
  $ make scripts_package
    HOSTCC  scripts/list-gitignored
  $ scripts/list-gitignored  --prefix=./ -o ../exclude-list
  $ tar cf ../linux2.tar --exclude-from=../exclude-list .
  $ tar tf ../linux2.tar | LANG=C sort > ../file-list2     # files emitted by 'tar'
  $ diff  ../file-list1 ../file-list2 | grep -E '^(<|>)'
  < ./Documentation/devicetree/bindings/.yamllint
  < ./drivers/clk/.kunitconfig
  < ./drivers/gpu/drm/tests/.kunitconfig
  < ./drivers/hid/.kunitconfig
  < ./fs/ext4/.kunitconfig
  < ./fs/fat/.kunitconfig
  < ./kernel/kcsan/.kunitconfig
  < ./lib/kunit/.kunitconfig
  < ./mm/kfence/.kunitconfig
  < ./tools/testing/selftests/arm64/tags/
  < ./tools/testing/selftests/arm64/tags/.gitignore
  < ./tools/testing/selftests/arm64/tags/Makefile
  < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
  < ./tools/testing/selftests/arm64/tags/tags_test.c
  < ./tools/testing/selftests/kvm/.gitignore
  < ./tools/testing/selftests/kvm/Makefile
  < ./tools/testing/selftests/kvm/config
  < ./tools/testing/selftests/kvm/settings

The source tarball contains most of files that are tracked by git. You see some diffs, but it is just because some .gitignore files are wrong.

  $ git ls-files -i -c --exclude-per-directory=.gitignore
  Documentation/devicetree/bindings/.yamllint
  drivers/clk/.kunitconfig
  drivers/gpu/drm/tests/.kunitconfig
  drivers/hid/.kunitconfig
  fs/ext4/.kunitconfig
  fs/fat/.kunitconfig
  kernel/kcsan/.kunitconfig
  lib/kunit/.kunitconfig
  mm/kfence/.kunitconfig
  tools/testing/selftests/arm64/tags/.gitignore
  tools/testing/selftests/arm64/tags/Makefile
  tools/testing/selftests/arm64/tags/run_tags_test.sh
  tools/testing/selftests/arm64/tags/tags_test.c
  tools/testing/selftests/kvm/.gitignore
  tools/testing/selftests/kvm/Makefile
  tools/testing/selftests/kvm/config
  tools/testing/selftests/kvm/settings

[1]: https://lore.kernel.org/all/[email protected]/

Quoted 8 lines
Quoted 2 lines

---

01 - 11 applied to linux-kbuild.

I will update 12/12 (postponed to v6.4)

Might you want to re-consider to include this patch already in v6.3? Bastian seems to be unhappy with the current state of v6.3:
https://lore.kernel.org/linux-kbuild/[email protected]/T/#t

Kind regards,
Nicolas

#24 lore raw
In reply to Masahiro Yamada
Quoted 9 lines · Masahiro Yamada

In short, the motivation of this commit is to build a source package without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.

Quoted 7 lines

Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.

In fact, there are some ways to achieve this.

The easiest solution is 'git archive'.

Eh, no, the easiest solution is to just don't build in-tree? Can you explain why 'make O=/over/there' isn't sufficient for whatever it is you really want to do here?

Rasmus

#25 lore raw
Quoted 17 lines · Rasmus Villemoes
Quoted 14 lines · Masahiro Yamada

In short, the motivation of this commit is to build a source package without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.

Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.

In fact, there are some ways to achieve this.

Quoted 7 lines
Quoted 1 line

The easiest solution is 'git archive'.

Eh, no, the easiest solution is to just don't build in-tree? Can you explain why 'make O=/over/there' isn't sufficient for whatever it is you really want to do here?

Rasmus

Right.

If you have a strict rule "I never build the kernel in-tree", your source tree is kept clean.
So, tar will never include generated files.

But, how would you know if the source tree is really clean?

--
Best Regards
Masahiro Yamada