kbuild: add a tool to list files ignored by git
PATCH v6 1/12 kbuild: add a tool to list files ignored by git
In short, the motivation of this commit is to build a source package without cleaning the source tree.
The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.
Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.
In fact, there are some ways to achieve this.
The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses it, but I do not like it because it works only when the source tree is managed by git, and all files you want in the tarball must be committed in advance.
I want to make it work without relying on git. We can do this.
Files that are ignored by git are generated files, so should be excluded from the source tarball. We can list them out by parsing the .gitignore files. Of course, .gitignore does not cover all the cases, but it works well enough.
tar(1) claims to support it:
--exclude-vcs-ignores
Exclude files that match patterns read from VCS-specific ignore files. Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.
The best scenario would be to use 'tar --exclude-vcs-ignores', but this option does not work. --exclude-vcs-ignore does not understand any of the negation (!), preceding slash, following slash, etc.. So, this option is just useless.
Hence, I wrote this gitignore parser. The previous version [1], written in Python, was so slow. This version is implemented in C, so it works much faster.
I imported the code from git (commit: 23c56f7bd5f1), so we get the same result.
This tool traverses the source tree, parsing all .gitignore files, and prints file paths that are ignored by git.
The output is similar to 'git ls-files --ignored --directory --others --exclude-per-directory=.gitignore', except
[1] Not sorted
[2] No trailing slash for directories
[2] is intentional because tar's --exclude-from option cannot handle trailing slashes.
[How to test this tool]
$ git clean -dfx
$ make -s -j$(nproc) defconfig all # or allmodconifg or whatever
$ git archive -o ../linux1.tar --prefix=./ HEAD
$ tar tf ../linux1.tar | LANG=C sort > ../file-list1 # files emitted by 'git archive'
$ make scripts_package
HOSTCC scripts/list-gitignored
$ scripts/list-gitignored --prefix=./ -o ../exclude-list
$ tar cf ../linux2.tar --exclude-from=../exclude-list .
$ tar tf ../linux2.tar | LANG=C sort > ../file-list2 # files emitted by 'tar'
$ diff ../file-list1 ../file-list2 | grep -E '^(<|>)'
< ./Documentation/devicetree/bindings/.yamllint
< ./drivers/clk/.kunitconfig
< ./drivers/gpu/drm/tests/.kunitconfig
< ./drivers/hid/.kunitconfig
< ./fs/ext4/.kunitconfig
< ./fs/fat/.kunitconfig
< ./kernel/kcsan/.kunitconfig
< ./lib/kunit/.kunitconfig
< ./mm/kfence/.kunitconfig
< ./tools/testing/selftests/arm64/tags/
< ./tools/testing/selftests/arm64/tags/.gitignore
< ./tools/testing/selftests/arm64/tags/Makefile
< ./tools/testing/selftests/arm64/tags/run_tags_test.sh
< ./tools/testing/selftests/arm64/tags/tags_test.c
< ./tools/testing/selftests/kvm/.gitignore
< ./tools/testing/selftests/kvm/Makefile
< ./tools/testing/selftests/kvm/config
< ./tools/testing/selftests/kvm/settingsThe source tarball contains most of files that are tracked by git. You see some diffs, but it is just because some .gitignore files are wrong.
$ git ls-files -i -c --exclude-per-directory=.gitignore Documentation/devicetree/bindings/.yamllint drivers/clk/.kunitconfig drivers/gpu/drm/tests/.kunitconfig drivers/hid/.kunitconfig fs/ext4/.kunitconfig fs/fat/.kunitconfig kernel/kcsan/.kunitconfig lib/kunit/.kunitconfig mm/kfence/.kunitconfig tools/testing/selftests/arm64/tags/.gitignore tools/testing/selftests/arm64/tags/Makefile tools/testing/selftests/arm64/tags/run_tags_test.sh tools/testing/selftests/arm64/tags/tags_test.c tools/testing/selftests/kvm/.gitignore tools/testing/selftests/kvm/Makefile tools/testing/selftests/kvm/config tools/testing/selftests/kvm/settings
[1]: https://lore.kernel.org/all/[email protected]/
---
(no changes since v5)
Changes in v5:
- Import a lot of code from GIT
- Fix a bug that the output file is relative to --rootdir
- Check ENOENT when open() fails to open .gitignore
- add more options, --ignore-case, --stat
Changes in v3:
- Various code refactoring: remove struct gitignore, remove next: label etc.
- Support --extra-pattern option
Changes in v2:
- Reimplement in C
Makefile | 7 +- scripts/.gitignore | 1 + scripts/Makefile | 2 +- scripts/list-gitignored.c | 1055 +++++++++++++++++++++++++++++++++++++ 4 files changed, 1063 insertions(+), 2 deletions(-) create mode 100644 scripts/list-gitignored.c
Makefile +6 -1
@@ -274,7 +274,8 @@ no-dot-config-targets := $(clean-targets) \ cscope gtags TAGS tags help% %docs check% coccicheck \ $(version_h) headers headers_% archheaders archscripts \ %asm-generic kernelversion %src-pkg dt_binding_check \ - outputmakefile rustavailable rustfmt rustfmtcheck + outputmakefile rustavailable rustfmt rustfmtcheck \ + scripts_package # Installation targets should not require compiler. Unfortunately, vdso_install # is an exception where build artifacts may be updated. This must be fixed. no-compiler-targets := $(no-dot-config-targets) install dtbs_install \
@@ -1652,6 +1653,10 @@ distclean: mrproper %pkg: include/config/kernel.release FORCE $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.package $@ +PHONY += scripts_package +scripts_package: scripts_basic + $(Q)$(MAKE) $(build)=scripts scripts/list-gitignored + # Brief documentation of the typical targets used # ---------------------------------------------------------------------------
scripts/.gitignore +1 -0
@@ -3,6 +3,7 @@ /generate_rust_target /insert-sys-cert /kallsyms +/list-gitignored /module.lds /recordmcount /sign-file
scripts/Makefile +1 -1
@@ -38,7 +38,7 @@ HOSTCFLAGS_sorttable.o += -DMCOUNT_SORT_ENABLED endif # The following programs are only built on demand -hostprogs += unifdef +hostprogs += list-gitignored unifdef # The module linker script is preprocessed on demand targets += module.lds
scripts/list-gitignored.c new file +1055 -0
1,055 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v6 2/12 kbuild: deb-pkg: create source package without cleaning
If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.
Instead of cleaning, pass the exclude list to tar's --exclude-from option.
Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig.
With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the .orig tarball.
linux.tar.gz is rebuilt only when the source files that would go into the tarball are changed.
---
Changes in v6:
- Add more stubs to cmd_tar so I can reuse it for the other rules
Changes in v5:
- Avoid unneeded rebuild of the tarball when nothing in the source tree is changed.
Changes in v4:
- Fix a typo in comment
Changes in v3:
- Add --extra-pattern='*.rej'
- Exclude symlinks at the toplevel
- Add --sort=name tar option
scripts/Makefile.package | 48 +++++++++++++++++++++++++++++++++++----- scripts/package/mkdebian | 24 ++++++++++++++++++++ 2 files changed, 66 insertions(+), 6 deletions(-)
scripts/Makefile.package +42 -6
@@ -43,13 +43,47 @@ if test "$(objtree)" != "$(srctree)"; then \ echo >&2 " ERROR:"; \ echo >&2 " Building source tarball is not possible outside the"; \ echo >&2 " kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \ - echo >&2 " binrpm-pkg or bindeb-pkg target instead."; \ + echo >&2 " binrpm-pkg target instead."; \ echo >&2; \ false; \ fi ; \ tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \ --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3) +# .tmp_filelist .tmp_filelist_exclude +# --------------------------------------------------------------------------- + +scripts/list-gitignored: FORCE + $(Q)$(MAKE) -f $(srctree)/Makefile scripts_package + +# 1f5d3a6b6532e25a5cdf1f311956b2b03d343a48 removed '*.rej' from .gitignore, +# but it is definitely a generated file. +filechk_filelist = \ + $< --exclude='*.rej' --output=$@_exclude --prefix=./ --rootdir=$(srctree) --stat=- + +.tmp_filelist: scripts/list-gitignored FORCE + $(call filechk,filelist) + +# tarball +# --------------------------------------------------------------------------- + +quiet_cmd_tar = TAR $@ + cmd_tar = tar -c -f $@ $(tar-compress-opt) $(tar-exclude-opt) \ + --owner=0 --group=0 --sort=name \ + --transform 's:^\.:$*:S' -C $(tar-rootdir) . + +tar-rootdir := $(srctree) + +%.tar.gz: private tar-compress-opt := -I $(KGZIP) +%.tar.gz: + $(call cmd,tar) + +# Linux source tarball +# --------------------------------------------------------------------------- + +linux.tar.gz: tar-exclude-opt = --exclude=./$@ --exclude-from=$<_exclude +linux.tar.gz: .tmp_filelist + # rpm-pkg # --------------------------------------------------------------------------- PHONY += rpm-pkg
@@ -80,13 +114,12 @@ binrpm-pkg: $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec PHONY += deb-pkg -deb-pkg: - $(MAKE) clean +deb-pkg: linux.tar.gz $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian - $(call cmd,src_tar,$(KDEB_SOURCENAME)) origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\ - mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc + cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz + +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ + --build=source,binary --source-option=-sP -nc -us -uc PHONY += bindeb-pkg bindeb-pkg:
@@ -174,4 +207,7 @@ help: @echo ' perf-tarxz-src-pkg - Build $(perf-tar).tar.xz source tarball' @echo ' perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball' +PHONY += FORCE +FORCE: + .PHONY: $(PHONY)
scripts/package/mkdebian +24 -0
@@ -84,6 +84,8 @@ set_debarch() { fi } +rm -rf debian + # Some variables and settings used throughout the script version=$KERNELRELEASE if [ -n "$KDEB_PKGVERSION" ]; then
@@ -135,6 +137,28 @@ fi mkdir -p debian/source/ echo "1.0" > debian/source/format +# Ugly: ignore anything except .config or debian/ +cat<<'EOF' > debian/source/local-options +diff-ignore + +extend-diff-ignore = ^[^.d] + +extend-diff-ignore = ^\.[^c] +extend-diff-ignore = ^\.c($|[^o]) +extend-diff-ignore = ^\.co($|[^n]) +extend-diff-ignore = ^\.con($|[^f]) +extend-diff-ignore = ^\.conf($|[^i]) +extend-diff-ignore = ^\.confi($|[^g]) +extend-diff-ignore = ^\.config. + +extend-diff-ignore = ^d($|[^e]) +extend-diff-ignore = ^de($|[^b]) +extend-diff-ignore = ^deb($|[^i]) +extend-diff-ignore = ^debi($|[^a]) +extend-diff-ignore = ^debia($|[^n]) +extend-diff-ignore = ^debian[^/] +EOF + echo $debarch > debian/arch extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)" extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"
Signature · 1 line
2.34.1
Re: [PATCH v6 02/12] kbuild: deb-pkg: create source package without cleaning
Quoted 148 lines · On Wed, Feb 15, 2023 at 10:20:24AM +0900 Masahiro Yamada wrote:
If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.
Instead of cleaning, pass the exclude list to tar's --exclude-from option.
Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig.
With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the .orig tarball.
linux.tar.gz is rebuilt only when the source files that would go into the tarball are changed.
Signed-off-by Masahiro Yamada <[email protected]>---
Changes in v6:
- Add more stubs tocmd_tarso I can reuse it for the other rulesChanges in v5:
- Avoid unneeded rebuild of the tarball when nothing in the source tree is changed.Changes in v4:
- Fix a typo in commentChanges in v3:
- Add --extra-pattern='*.rej'
- Exclude symlinks at the toplevel
- Add --sort=name tar optionscripts/Makefile.package | 48 +++++++++++++++++++++++++++++++++++----- scripts/package/mkdebian | 24 ++++++++++++++++++++ 2 files changed, 66 insertions(+), 6 deletions(-)scripts/Makefile.package +42
-6@@ -43,13 +43,47 @@ if test "$(objtree)" != "$(srctree)"; then \ echo >&2 " ERROR:"; \ echo >&2 " Building source tarball is not possible outside the"; \ echo >&2 " kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \ - echo >&2 " binrpm-pkg or bindeb-pkg target instead."; \ + echo >&2 " binrpm-pkg target instead."; \ echo >&2; \ false; \ fi ; \ tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \ --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3) +# .tmp_filelist .tmp_filelist_exclude +# --------------------------------------------------------------------------- + +scripts/list-gitignored: FORCE + $(Q)$(MAKE) -f $(srctree)/Makefile scripts_package + +# 1f5d3a6b6532e25a5cdf1f311956b2b03d343a48 removed '*.rej' from .gitignore, +# but it is definitely a generated file. +filechk_filelist = \ + $< --exclude='*.rej' --output=$@_exclude --prefix=./ --rootdir=$(srctree) --stat=- + +.tmp_filelist: scripts/list-gitignored FORCE + $(call filechk,filelist) + +# tarball +# --------------------------------------------------------------------------- + +quiet_cmd_tar = TAR $@ + cmd_tar = tar -c -f $@ $(tar-compress-opt) $(tar-exclude-opt) \ + --owner=0 --group=0 --sort=name \ + --transform 's:^\.:$*:S' -C $(tar-rootdir) . + +tar-rootdir := $(srctree) + +%.tar.gz: private tar-compress-opt := -I $(KGZIP) +%.tar.gz: + $(call cmd,tar) + +# Linux source tarball +# --------------------------------------------------------------------------- + +linux.tar.gz: tar-exclude-opt = --exclude=./$@ --exclude-from=$<_exclude +linux.tar.gz: .tmp_filelist + # rpm-pkg # --------------------------------------------------------------------------- PHONY += rpm-pkg@@ -80,13 +114,12 @@ binrpm-pkg: $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec PHONY += deb-pkg -deb-pkg: - $(MAKE) clean +deb-pkg: linux.tar.gz $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian - $(call cmd,src_tar,$(KDEB_SOURCENAME)) origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\ - mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc + cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz + +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ + --build=source,binary --source-option=-sP -nc -us -uc PHONY += bindeb-pkg bindeb-pkg:@@ -174,4 +207,7 @@ help: @echo ' perf-tarxz-src-pkg - Build $(perf-tar).tar.xz source tarball' @echo ' perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball' +PHONY += FORCE +FORCE: + .PHONY: $(PHONY)scripts/package/mkdebian +17
-0@@ -84,6 +84,8 @@ set_debarch() { fi } +rm -rf debian + # Some variables and settings used throughout the script version=$KERNELRELEASE if [ -n "$KDEB_PKGVERSION" ]; then@@ -135,6 +137,28 @@ fi mkdir -p debian/source/ echo "1.0" > debian/source/format +# Ugly: ignore anything except .config or debian/ +cat<<'EOF' > debian/source/local-options +diff-ignore + +extend-diff-ignore = ^[^.d] + +extend-diff-ignore = ^\.[^c] +extend-diff-ignore = ^\.c($|[^o]) +extend-diff-ignore = ^\.co($|[^n]) +extend-diff-ignore = ^\.con($|[^f]) +extend-diff-ignore = ^\.conf($|[^i]) +extend-diff-ignore = ^\.confi($|[^g]) +extend-diff-ignore = ^\.config. + +extend-diff-ignore = ^d($|[^e])
scripts/package/mkdebian fi · line 164
Quoted 10 lines
scripts/package/mkdebian +7
-0+extend-diff-ignore = ^de($|[^b]) +extend-diff-ignore = ^deb($|[^i]) +extend-diff-ignore = ^debi($|[^a]) +extend-diff-ignore = ^debia($|[^n]) +extend-diff-ignore = ^debian[^/] +EOF + echo $debarch > debian/arch extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)" extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"
ah, nice. When I attempted to build with current Debian amd64 kernel config, python3 was missing as build-depencendy (due to some BPF kconfig).
Quoted 2 lines
Signature · 1 line
2.34.1
Signature · 3 lines
epost|xmpp: [email protected] irc://oftc.net/nsc ↳ gpg: 18ed 52db e34f 860e e9fb c82b 7d97 0932 55a0 ce7f -- frykten for herren er opphav til kunnskap --
PATCH v6 3/12 kbuild: rpm-pkg: build binary packages from source rpm
In reply to Masahiro YamadaThe build rules of rpm-pkg and srcrpm-pkg are almost the same. Remove the code duplication.
Change rpm-pkg to build binary packages from the source package generated by srcrpm-pkg.
This changes the output directory of the srpm generated by 'make rpm-pkg' because srcrpm-pkg overrides _srcrpmdir.
---
(no changes since v3)
Changes in v3:
- Explain that the source package location will be changed.
scripts/Makefile.package | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
scripts/Makefile.package +3 -5
@@ -87,11 +87,9 @@ linux.tar.gz: .tmp_filelist # rpm-pkg # --------------------------------------------------------------------------- PHONY += rpm-pkg -rpm-pkg: - $(MAKE) clean - $(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec - $(call cmd,src_tar,$(KERNELPATH),kernel.spec) - +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ta $(KERNELPATH).tar.gz \ +rpm-pkg: srpm = $(shell rpmspec --srpm --query --queryformat='%{name}-%{VERSION}-%{RELEASE}.src.rpm' kernel.spec) +rpm-pkg: srcrpm-pkg + +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -rb $(srpm) \ --define='_smp_mflags %{nil}' # srcrpm-pkg
Signature · 1 line
2.34.1
PATCH v6 4/12 kbuild: srcrpm-pkg: create source package without cleaning
In reply to Masahiro YamadaIf you run 'make (src)rpm-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.
Instead of cleaning, pass the exclude list to tar's --exclude-from option.
Previously, the .config was contained in the source tarball.
With this commit, the source rpm consists of separate linux.tar.gz and .config.
Remove stale comments. Now, 'make (src)rpm-pkg' works with O= option.
---
(no changes since v4)
Changes in v4:
- Do not delete the old tar command because it is still used by snap-pkg although snap-pkg is broken, and it does not work at all.
scripts/Makefile.package | 32 ++++---------------------------- scripts/package/mkspec | 8 ++++---- 2 files changed, 8 insertions(+), 32 deletions(-)
scripts/Makefile.package +4 -28
@@ -3,27 +3,6 @@ include $(srctree)/scripts/Kbuild.include -# RPM target -# --------------------------------------------------------------------------- -# The rpm target generates two rpm files: -# /usr/src/packages/SRPMS/kernel-2.6.7rc2-1.src.rpm -# /usr/src/packages/RPMS/i386/kernel-2.6.7rc2-1.<arch>.rpm -# The src.rpm files includes all source for the kernel being built -# The <arch>.rpm includes kernel configuration, modules etc. -# -# Process to create the rpm files -# a) clean the kernel -# b) Generate .spec file -# c) Build a tar ball, using symlink to make kernel version -# first entry in the path -# d) and pack the result to a tar.gz file -# e) generate the rpm files, based on kernel.spec -# - Use /. to avoid tar packing just the symlink - -# Note that the rpm-pkg target cannot be used with KBUILD_OUTPUT, -# but the binrpm-pkg target can; for some reason O= gets ignored. - -# Remove hyphens since they have special meaning in RPM filenames KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE)) KDEB_SOURCENAME ?= linux-upstream KBUILD_PKG_ROOTCMD ?="fakeroot -u"
@@ -42,8 +21,7 @@ if test "$(objtree)" != "$(srctree)"; then \ echo >&2; \ echo >&2 " ERROR:"; \ echo >&2 " Building source tarball is not possible outside the"; \ - echo >&2 " kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \ - echo >&2 " binrpm-pkg target instead."; \ + echo >&2 " kernel source tree. Don't set KBUILD_OUTPUT"; \ echo >&2; \ false; \ fi ; \
@@ -95,12 +73,10 @@ rpm-pkg: srcrpm-pkg # srcrpm-pkg # --------------------------------------------------------------------------- PHONY += srcrpm-pkg -srcrpm-pkg: - $(MAKE) clean +srcrpm-pkg: linux.tar.gz $(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec - $(call cmd,src_tar,$(KERNELPATH),kernel.spec) - +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ts $(KERNELPATH).tar.gz \ - --define='_smp_mflags %{nil}' --define='_srcrpmdir $(srctree)' + +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -bs kernel.spec \ + --define='_smp_mflags %{nil}' --define='_sourcedir .' --define='_srcrpmdir .' # binrpm-pkg # ---------------------------------------------------------------------------
scripts/package/mkspec +4 -4
@@ -47,7 +47,8 @@ sed -e '/^DEL/d' -e 's/^\t*//' <<EOF Group: System Environment/Kernel Vendor: The Linux Community URL: https://www.kernel.org -$S Source: kernel-$__KERNELRELEASE.tar.gz +$S Source0: linux.tar.gz +$S Source1: .config Provides: $PROVIDES $S BuildRequires: bc binutils bison dwarves $S BuildRequires: (elfutils-libelf-devel or libelf-devel) flex
@@ -83,9 +84,8 @@ $S$M This package provides kernel headers and makefiles sufficient to build modu $S$M against the $__KERNELRELEASE kernel package. $S$M $S %prep -$S %setup -q -$S rm -f scripts/basic/fixdep scripts/kconfig/conf -$S rm -f tools/objtool/{fixdep,objtool} +$S %setup -q -n linux +$S cp %{SOURCE1} . $S $S %build $S $MAKE %{?_smp_mflags} KERNELRELEASE=$KERNELRELEASE KBUILD_BUILD_VERSION=%{release}
Signature · 1 line
2.34.1
PATCH v6 5/12 kbuild: deb-pkg: hide KDEB_SOURCENAME from Makefile
In reply to Masahiro Yamadascripts/Makefile.package does not need to know the value of KDEB_SOURCENAME because the source name can be taken from debian/changelog by using dpkg-parsechangelog.
Move the default of KDEB_SOURCENAME (i.e. linux-upstream) to scripts/package/mkdebian.
---
(no changes since v5)
Changes in v5:
- Copy linux.tar.gz to .orig tarball
Changes in v3:
- Move cmd_debianize
Changes in v2:
- New patch
scripts/Makefile.package | 23 +++++++++++++++-------- scripts/package/mkdebian | 2 +- 2 files changed, 16 insertions(+), 9 deletions(-)
scripts/Makefile.package +15 -8
@@ -4,9 +4,7 @@ include $(srctree)/scripts/Kbuild.include KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE)) -KDEB_SOURCENAME ?= linux-upstream KBUILD_PKG_ROOTCMD ?="fakeroot -u" -export KDEB_SOURCENAME # Include only those top-level files that are needed by make, plus the GPL copy TAR_CONTENT := Documentation LICENSES arch block certs crypto drivers fs \ include init io_uring ipc kernel lib mm net rust \
@@ -87,17 +85,26 @@ binrpm-pkg: +rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \ $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec +quiet_cmd_debianize = GEN $@ + cmd_debianize = $(srctree)/scripts/package/mkdebian + +debian: FORCE + $(call cmd,debianize) + +PHONY += debian-orig +debian-orig: private source = $(shell dpkg-parsechangelog -S Source) +debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//') +debian-orig: private orig-name = $(source)_$(version).orig.tar.gz +debian-orig: linux.tar.gz debian + $(Q)cp $< ../$(orig-name) + PHONY += deb-pkg -deb-pkg: linux.tar.gz - $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian - origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\ - cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz +deb-pkg: debian-orig +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ --build=source,binary --source-option=-sP -nc -us -uc PHONY += bindeb-pkg -bindeb-pkg: - $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian +bindeb-pkg: debian +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc PHONY += intdeb-pkg
scripts/package/mkdebian +1 -1
@@ -95,7 +95,7 @@ else revision=$($srctree/init/build-version) packageversion=$version-$revision fi -sourcename=$KDEB_SOURCENAME +sourcename=${KDEB_SOURCENAME:-linux-upstream} if [ "$ARCH" = "um" ] ; then packagename=user-mode-linux
Signature · 1 line
2.34.1
Re: [PATCH v6 05/12] kbuild: deb-pkg: hide KDEB_SOURCENAME from Makefile
Quoted 87 lines · On Wed, Feb 15, 2023 at 10:20:27AM +0900 Masahiro Yamada wrote:
scripts/Makefile.packagedoes not need to know the value ofKDEB_SOURCENAMEbecause the source name can be taken from debian/changelog by using dpkg-parsechangelog.Move the default of
KDEB_SOURCENAME(i.e. linux-upstream) toscripts/package/mkdebian.Signed-off-by Masahiro Yamada <[email protected]>---
(no changes since v5)
Changes in v5:
- Copy linux.tar.gz to .orig tarballChanges in v3:
- Movecmd_debianizeChanges in v2:
- New patchscripts/Makefile.package | 23 +++++++++++++++-------- scripts/package/mkdebian | 2 +- 2 files changed, 16 insertions(+), 9 deletions(-)scripts/Makefile.package +15
-8@@ -4,9 +4,7 @@ include $(srctree)/scripts/Kbuild.include KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE)) -KDEB_SOURCENAME ?= linux-upstream KBUILD_PKG_ROOTCMD ?="fakeroot -u" -export KDEB_SOURCENAME # Include only those top-level files that are needed by make, plus the GPL copy TAR_CONTENT := Documentation LICENSES arch block certs crypto drivers fs \ include init io_uring ipc kernel lib mm net rust \@@ -87,17 +85,26 @@ binrpm-pkg: +rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \ $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec +quiet_cmd_debianize = GEN $@ + cmd_debianize = $(srctree)/scripts/package/mkdebian + +debian: FORCE + $(call cmd,debianize) + +PHONY += debian-orig +debian-orig: private source = $(shell dpkg-parsechangelog -S Source) +debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//') +debian-orig: private orig-name = $(source)_$(version).orig.tar.gz +debian-orig: linux.tar.gz debian + $(Q)cp $< ../$(orig-name) + PHONY += deb-pkg -deb-pkg: linux.tar.gz - $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian - origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\ - cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz +deb-pkg: debian-orig +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ --build=source,binary --source-option=-sP -nc -us -uc PHONY += bindeb-pkg -bindeb-pkg: - $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian +bindeb-pkg: debian +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc PHONY += intdeb-pkgscripts/package/mkdebian +1
-1@@ -95,7 +95,7 @@ else revision=$($srctree/init/build-version) packageversion=$version-$revision fi -sourcename=$KDEB_SOURCENAME +sourcename=${KDEB_SOURCENAME:-linux-upstream} if [ "$ARCH" = "um" ] ; then packagename=user-mode-linuxSignature · 1 line
2.34.1
PATCH v6 6/12 kbuild: deb-pkg: make .orig tarball a hard link if possible
In reply to Masahiro YamadaIf '..' belongs to the same filesystem, create a hard link instead of a copy. In most cases, you can save disk space.
I do not want to use 'mv' because keeping linux.tar.gz is useful to avoid unneeded rebuilding of the tarball.
---
Changes in v6:
- New patch
scripts/Makefile.package | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/Makefile.package +5 -1
@@ -96,7 +96,11 @@ debian-orig: private source = $(shell dpkg-parsechangelog -S Source) debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//') debian-orig: private orig-name = $(source)_$(version).orig.tar.gz debian-orig: linux.tar.gz debian - $(Q)cp $< ../$(orig-name) + $(Q)if [ "$(df --output=target .. 2>/dev/null)" = "$(df --output=target $< 2>/dev/null)" ]; then \ + ln -f $< ../$(orig-name); \ + else \ + cp $< ../$(orig-name); \ + fi PHONY += deb-pkg deb-pkg: debian-orig
Signature · 1 line
2.34.1
Re: [PATCH v6 06/12] kbuild: deb-pkg: make .orig tarball a hard link if possible
Quoted 34 lines · On Wed, Feb 15, 2023 at 10:20:28AM +0900 Masahiro Yamada wrote:
If '..' belongs to the same filesystem, create a hard link instead of a copy. In most cases, you can save disk space.
I do not want to use 'mv' because keeping linux.tar.gz is useful to avoid unneeded rebuilding of the tarball.
Signed-off-by Masahiro Yamada <[email protected]>---
Changes in v6:
- New patchscripts/Makefile.package | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)scripts/Makefile.package +5
-1@@ -96,7 +96,11 @@ debian-orig: private source = $(shell dpkg-parsechangelog -S Source) debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//') debian-orig: private orig-name = $(source)_$(version).orig.tar.gz debian-orig: linux.tar.gz debian - $(Q)cp $< ../$(orig-name) + $(Q)if [ "$(df --output=target .. 2>/dev/null)" = "$(df --output=target $< 2>/dev/null)" ]; then \ + ln -f $< ../$(orig-name); \ + else \ + cp $< ../$(orig-name); \ + fi PHONY += deb-pkg deb-pkg: debian-origSignature · 1 line
2.34.1
PATCH v6 7/12 kbuild: deb-pkg: switch over to source format 3.0 (quilt)
In reply to Masahiro YamadaChange the source format from "1.0" to "3.0 (quilt)" because it works more cleanly.
All files except .config and debian/ go into the orig tarball. Add a single patch, debian/patches/config, and delete the ugly extend-diff-ignore patterns.
The debian tarball will be compressed into *.debian.tar.xz by default. If you like to use a different compression mode, you can pass the command line option, DPKG_FLAGS=-Zgzip, for example.
The orig tarball only supports gzip for now. The combination of gzip and xz is somewhat clumsy, but it is not a practical problem.
---
Changes in v6:
- Fix error when KCONFIG_CONFIG is set
Changes in v4:
- New patch
scripts/Makefile.package | 2 +- scripts/package/mkdebian | 41 ++++++++++++++++++---------------------- 2 files changed, 19 insertions(+), 24 deletions(-)
scripts/Makefile.package +1 -1
@@ -105,7 +105,7 @@ debian-orig: linux.tar.gz debian PHONY += deb-pkg deb-pkg: debian-orig +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ - --build=source,binary --source-option=-sP -nc -us -uc + --build=source,binary -nc -us -uc PHONY += bindeb-pkg bindeb-pkg: debian
scripts/package/mkdebian +18 -23
47 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v6 07/12] kbuild: deb-pkg: switch over to source format 3.0 (quilt)
Quoted 11 lines · On Wed, Feb 15, 2023 at 10:20:29AM +0900 Masahiro Yamada wrote:
Change the source format from "1.0" to "3.0 (quilt)" because it works more cleanly.
All files except .config and debian/ go into the orig tarball. Add a single patch, debian/patches/config, and delete the ugly extend-diff-ignore patterns.
The debian tarball will be compressed into *.debian.tar.xz by default. If you like to use a different compression mode, you can pass the command line option, DPKG_FLAGS=-Zgzip, for example.
Quoted 5 lines
The orig tarball only supports gzip for now. The combination of gzip and xz is somewhat clumsy, but it is not a practical problem.
Signed-off-by Masahiro Yamada <[email protected]>---
Thanks, that switch to 3.0/quilt sounds really good to me.
Quoted 60 lines
Changes in v6:
- Fix error whenKCONFIG_CONFIGis setChanges in v4:
- New patchscripts/Makefile.package | 2 +- scripts/package/mkdebian | 41 ++++++++++++++++++---------------------- 2 files changed, 19 insertions(+), 24 deletions(-)scripts/Makefile.package +1
-1@@ -105,7 +105,7 @@ debian-orig: linux.tar.gz debian PHONY += deb-pkg deb-pkg: debian-orig +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ - --build=source,binary --source-option=-sP -nc -us -uc + --build=source,binary -nc -us -uc PHONY += bindeb-pkg bindeb-pkg: debianscripts/package/mkdebian +5
-23@@ -135,29 +135,24 @@ else fi mkdir -p debian/source/ -echo "1.0" > debian/source/format - -# Ugly: ignore anything except .config or debian/ -cat<<'EOF' > debian/source/local-options -diff-ignore - -extend-diff-ignore = ^[^.d] - -extend-diff-ignore = ^\.[^c] -extend-diff-ignore = ^\.c($|[^o]) -extend-diff-ignore = ^\.co($|[^n]) -extend-diff-ignore = ^\.con($|[^f]) -extend-diff-ignore = ^\.conf($|[^i]) -extend-diff-ignore = ^\.confi($|[^g]) -extend-diff-ignore = ^\.config. - -extend-diff-ignore = ^d($|[^e]) -extend-diff-ignore = ^de($|[^b]) -extend-diff-ignore = ^deb($|[^i]) -extend-diff-ignore = ^debi($|[^a]) -extend-diff-ignore = ^debia($|[^n]) -extend-diff-ignore = ^debian[^/] -EOF +echo "3.0 (quilt)" > debian/source/format + +{ + echo "diff-ignore" + echo "extend-diff-ignore = .*"
scripts/package/mkdebian else · line 152
Quoted 10 lines
scripts/package/mkdebian +10
-0+} > debian/source/local-options + +# Add .config as a patch +mkdir -p debian/patches +{ + echo "Subject: Add .config" + echo "Author: ${maintainer}" + echo + echo "--- /dev/null" + echo "+++ linux/.config"
bike-shedding:
cat <<-eof
Subject: Add .config
Author: ${maintainer}
--- /dev/null
+++ linux/.config
eofQuoted 8 lines
scripts/package/mkdebian +3
-0+ diff -u /dev/null "${KCONFIG_CONFIG}" | tail -n +3 +} > debian/patches/config +echo config > debian/patches/series echo $debarch > debian/arch extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"Signature · 1 line
2.34.1
Re: [PATCH v6 07/12] kbuild: deb-pkg: switch over to source format 3.0 (quilt)
Quoted 10 lines · Nicolas Schier
bike-shedding:
cat <<-eof Subject: Add .config Author: ${maintainer} --- /dev/null +++ linux/.config eof
I agree this is more readable, but I used echo to
avoid a process fork.
'echo' is a shell built-in for bash and dash but 'cat' is not.
So, this is a trade-off between readability vs
one less process fork.
If the code were much longer, I would prefer 'cat'.
Quoted 11 lines
Quoted 9 lines
scripts/package/mkdebian +3
-1+ diff -u /dev/null "${KCONFIG_CONFIG}" | tail -n +3 +} > debian/patches/config +echo config > debian/patches/series echo $debarch > debian/arch extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)" --2.34.1
Reviewed-by Nicolas Schier <[email protected]>
--
Best Regards
Masahiro Yamada
PATCH v6 8/12 kbuild: make perf-tar*-src-pkg work without relying on git
In reply to Masahiro YamadaCurrently, perf-tar*-src-pkg only uses 'git archive', but it is better to make it work without relying on git.
The file, HEAD, which saves the commit hash, will be included in the tarball only when the source tree is managed by git. The git tree is more precisely checked; it has been copied from scripts/setlocalversion.
---
(no changes since v5)
Changes in v5:
- New patch
scripts/Makefile.package | 97 ++++++++++++++++++++++++++-------------- 1 file changed, 64 insertions(+), 33 deletions(-)
scripts/Makefile.package +64 -33
@@ -50,10 +50,25 @@ quiet_cmd_tar = TAR $@ tar-rootdir := $(srctree) +%.tar: + $(call cmd,tar) + %.tar.gz: private tar-compress-opt := -I $(KGZIP) %.tar.gz: $(call cmd,tar) +%.tar.bz2: private tar-compress-opt := -I $(KBZIP2) +%.tar.bz2: + $(call cmd,tar) + +%.tar.xz: private tar-compress-opt := -I $(XZ) +%.tar.xz: + $(call cmd,tar) + +%.tar.zst: private tar-compress-opt := -I $(ZSTD) +%.tar.zst: + $(call cmd,tar) + # Linux source tarball # ---------------------------------------------------------------------------
@@ -138,36 +153,52 @@ $(tar-pkgs): $(MAKE) -f $(srctree)/Makefile +$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@ -# perf-pkg - generate a source tarball with perf source +# perf-tar*-src-pkg - generate a source tarball with perf source # --------------------------------------------------------------------------- -perf-tar=perf-$(KERNELVERSION) - -quiet_cmd_perf_tar = TAR - cmd_perf_tar = \ -git --git-dir=$(srctree)/.git archive --prefix=$(perf-tar)/ \ - HEAD^{tree} $$(cd $(srctree); \ - echo $$(cat tools/perf/MANIFEST)) \ - -o $(perf-tar).tar; \ -mkdir -p $(perf-tar); \ -git --git-dir=$(srctree)/.git rev-parse HEAD > $(perf-tar)/HEAD; \ -(cd $(srctree)/tools/perf; \ -util/PERF-VERSION-GEN $(CURDIR)/$(perf-tar)/); \ -tar rf $(perf-tar).tar $(perf-tar)/HEAD $(perf-tar)/PERF-VERSION-FILE; \ -rm -r $(perf-tar); \ -$(if $(findstring tar-src,$@),, \ -$(if $(findstring bz2,$@),$(KBZIP2), \ -$(if $(findstring gz,$@),$(KGZIP), \ -$(if $(findstring xz,$@),$(XZ), \ -$(if $(findstring zst,$@),$(ZSTD), \ -$(error unknown target $@))))) \ - -f -9 $(perf-tar).tar) - -perf-tar-pkgs := perf-tar-src-pkg perf-targz-src-pkg perf-tarbz2-src-pkg \ - perf-tarxz-src-pkg perf-tarzst-src-pkg -PHONY += $(perf-tar-pkgs) -$(perf-tar-pkgs): - $(call cmd,perf_tar) +perf-tar-src-pkg-tarball = perf-$(KERNELVERSION).$(1) +perf-tar-src-pkg-phony = perf-$(subst .,,$(1))-src-pkg + +quiet_cmd_stage_perf_src = STAGE $@ + cmd_stage_perf_src = \ + rm -rf $@; \ + mkdir -p $@; \ + tar -c -f - --exclude-from=$<_exclude -C $(srctree) --files-from=$(srctree)/tools/perf/MANIFEST | \ + tar -x -f - -C $@ + +.tmp_perf: .tmp_filelist + $(call cmd,stage_perf_src) + +filechk_perf_head = \ + if test -z "$(git -C $(srctree) rev-parse --show-cdup 2>/dev/null)" && \ + head=$$(git -C $(srctree) rev-parse --verify HEAD 2>/dev/null); then \ + echo $$head; \ + else \ + echo "not a git tree"; \ + fi + +.tmp_perf/HEAD: .tmp_perf FORCE + $(call filechk,perf_head) + +quiet_cmd_perf_version_file = GEN $@ + cmd_perf_version_file = cd $(srctree)/tools/perf; util/PERF-VERSION-GEN $(dir $(abspath $@)) + +# PERF-VERSION-FILE and HEAD are independent, but this avoids updating the +# timestamp of PERF-VERSION-FILE. +# The best is to fix tools/perf/util/PERF-VERSION-GEN. +.tmp_perf/PERF-VERSION-FILE: .tmp_perf/HEAD $(srctree)/tools/perf/util/PERF-VERSION-GEN + $(call cmd,perf_version_file) + +define perf-tar-src-pkg-rule +PHONY += $(perf-tar-src-pkg-phony) +$(perf-tar-src-pkg-phony): $(perf-tar-src-pkg-tarball) + @: + +$(perf-tar-src-pkg-tarball): private tar-rootdir := .tmp_perf +$(perf-tar-src-pkg-tarball): .tmp_filelist .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE +endef + +$(foreach x, tar tar.gz tar.bz2 tar.xz tar.zst, $(eval $(call perf-tar-src-pkg-rule,$(x)))) # Help text displayed when executing 'make help' # ---------------------------------------------------------------------------
@@ -186,11 +217,11 @@ help: @echo ' tarbz2-pkg - Build the kernel as a bzip2 compressed tarball' @echo ' tarxz-pkg - Build the kernel as a xz compressed tarball' @echo ' tarzst-pkg - Build the kernel as a zstd compressed tarball' - @echo ' perf-tar-src-pkg - Build $(perf-tar).tar source tarball' - @echo ' perf-targz-src-pkg - Build $(perf-tar).tar.gz source tarball' - @echo ' perf-tarbz2-src-pkg - Build $(perf-tar).tar.bz2 source tarball' - @echo ' perf-tarxz-src-pkg - Build $(perf-tar).tar.xz source tarball' - @echo ' perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball' + @echo ' perf-tar-src-pkg - Build the perf source tarball with no compression' + @echo ' perf-targz-src-pkg - Build the perf source tarball with gzip compression' + @echo ' perf-tarbz2-src-pkg - Build the perf source tarball with bz2 compression' + @echo ' perf-tarxz-src-pkg - Build the perf source tarball with xz compression' + @echo ' perf-tarzst-src-pkg - Build the perf source tarball with zst compression' PHONY += FORCE FORCE:
Signature · 1 line
2.34.1
PATCH v6 9/12 kbuild: tar-pkg: use tar rules in scripts/Makefile.package
In reply to Masahiro YamadaUse %.tar, %.tar.gz, %.tar.bz2, %.tar.xz, %.tar.zst rules in scripts/Makefile.package.
---
Changes in v6:
- New patch
scripts/Makefile.package | 29 +++++++++++++++++----- scripts/package/buildtar | 52 +--------------------------------------- 2 files changed, 24 insertions(+), 57 deletions(-)
scripts/Makefile.package +23 -6
@@ -145,13 +145,30 @@ snap-pkg: cd $(objtree)/snap && \ snapcraft --target-arch=$(UTS_MACHINE) -# tarball targets +# dir-pkg tar*-pkg - tarball targets # --------------------------------------------------------------------------- -tar-pkgs := dir-pkg tar-pkg targz-pkg tarbz2-pkg tarxz-pkg tarzst-pkg -PHONY += $(tar-pkgs) -$(tar-pkgs): - $(MAKE) -f $(srctree)/Makefile - +$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@ + +tar-pkg-tarball = linux-$(KERNELRELEASE)-$(ARCH).$(1) +tar-pkg-phony = $(subst .,,$(1))-pkg + +tar-install: FORCE + $(Q)$(MAKE) -f $(srctree)/Makefile + +$(Q)$(srctree)/scripts/package/buildtar $@ + +PHONY += dir-pkg +dir-pkg: tar-install + @echo "Kernel tree successfully created in $<" + +define tar-pkg-rule +PHONY += $(tar-pkg-phony) +$(tar-pkg-phony): $(tar-pkg-tarball) + @: + +$(tar-pkg-tarball): private tar-rootdir := tar-install +$(tar-pkg-tarball): tar-install +endef + +$(foreach x, tar tar.gz tar.bz2 tar.xz tar.zst, $(eval $(call tar-pkg-rule,$(x)))) # perf-tar*-src-pkg - generate a source tarball with perf source # ---------------------------------------------------------------------------
scripts/package/buildtar +1 -51
61 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v6 10/12 kbuild: deb-pkg: fix binary-arch and clean in debian/rules
In reply to Masahiro YamadaThe clean target needs ARCH=${ARCH} to clean up the tree for the correct architecture. 'make (bin)deb-pkg' skips cleaning, but the preclean hook may be executed if dpkg-buildpackage is directly used.
The binary-arch target does not need KERNELRELEASE because it is not updated during the installation. KBUILD_BUILD_VERSION is not needed either because binary-arch does not build vmlinux.
---
Changes in v6:
- New patch
scripts/package/mkdebian | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
scripts/package/mkdebian +2 -4
@@ -251,12 +251,10 @@ build: build-arch binary-indep: binary-arch: build-arch - \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \ - KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile intdeb-pkg - + \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} intdeb-pkg clean: rm -rf debian/files debian/linux-* - \$(MAKE) clean + \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} clean binary: binary-arch EOF
Signature · 1 line
2.34.1
PATCH v6 11/12 kbuild: deb-pkg: improve the usability of source package
In reply to Masahiro YamadaImprove the source package support in case the dpkg-buildpackage is directly used to build binary packages.
For cross-compiling, you can set CROSS_COMPILE via the environment variable, but it is better to set it automatically - set it to ${DEB_HOST_GNU_TYPE}- if we are cross-compiling but not from the top Makefile.
The generated source package may be carried to a different build environment, which may have a different compiler installed. Run olddefconfig first to set new CONFIG options to their default values without prompting.
Take KERNELRELEASE and KBUILD_BUILD_VERSION from the version field of debian/changelog in case it is updated afterwards.
---
Changes in v6:
- New patch
scripts/package/deb-build-option | 16 ++++++++++++++++ scripts/package/mkdebian | 9 ++++----- 2 files changed, 20 insertions(+), 5 deletions(-) create mode 100755 scripts/package/deb-build-option
scripts/package/deb-build-option new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only + +# Set up CROSS_COMPILE if we are cross-compiling, but not called from the +# kernel toplevel Makefile +if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUILD_ARCH}" ]; then + echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}- +fi + +version=$(dpkg-parsechangelog -S Version) +version_upstream="${version%-*}" +debian_revision="${version#${version_upstream}}" +debian_revision="${debian_revision#*-}" + +echo KERNELRELEASE=${version_upstream} +echo KBUILD_BUILD_VERSION=${debian_revision}
scripts/package/mkdebian +4 -5
@@ -90,10 +90,8 @@ rm -rf debian version=$KERNELRELEASE if [ -n "$KDEB_PKGVERSION" ]; then packageversion=$KDEB_PKGVERSION - revision=${packageversion##*-} else - revision=$($srctree/init/build-version) - packageversion=$version-$revision + packageversion=$version-$($srctree/init/build-version) fi sourcename=${KDEB_SOURCENAME:-linux-upstream}
@@ -244,8 +242,9 @@ srctree ?= . build-indep: build-arch: - \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \ - KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile + \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} \ + \$(shell \$(srctree)/scripts/package/deb-build-option) \ + olddefconfig all build: build-arch
Signature · 1 line
2.34.1
Re: [PATCH v6 11/12] kbuild: deb-pkg: improve the usability of source package
Quoted 8 lines · Masahiro Yamada
Improve the source package support in case the dpkg-buildpackage is directly used to build binary packages.
For cross-compiling, you can set
CROSS_COMPILEvia the environment variable, but it is better to set it automatically - set it to ${DEB_HOST_GNU_TYPE}- if we are cross-compiling but not from the topMakefile.
Quoted 7 lines
The generated source package may be carried to a different build environment, which may have a different compiler installed. Run olddefconfig first to set new CONFIG options to their default values without prompting.
Take KERNELRELEASE and
KBUILD_BUILD_VERSIONfrom the version field of debian/changelog in case it is updated afterwards.
I just arrived to this patch when trying to update the kernel for SOF from 6.2.0 base to 6.3.0-rc1:
https://github.com/thesofproject/linux/pull/4223#issuecomment-1459997676
Our CI is building a kernel deb package from PRs to test with this command:
make bindeb-pkg LOCALVERSION=-$PRIDV-default KDEB_PKGVERSION=$(make kernelversion)-$GITSHA
The files, directories and the name of the deb package follows the rule set up by scripts/setlocalversion, taking into account the LOCALVERSION, CONFIG_LOCALVERSION and CONFIG_LOCALVERSION_AUTO, but the built kernel is using the version derived from KDEB_PKGVERSION and the system is not able to boot up since the modules are in a different directory then they should be.
Something like this:
make bindeb-pkg LOCALVERSION=-testbuild vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty /lib/modules/6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/
strings vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty | grep 6.3 6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty
Then with the KDEB_PKGVERSION:
make bindeb-pkg LOCALVERSION=-testbuild KDEB_PKGVERSION=$(make kernelversion)-notworking vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty /lib/modules/6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/
strings vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty | grep 6.3 6.3.0-rc1
in this case modprobe will not find the modules, it is looking for them under /lib/modules/6.3.0-rc1
Reverting this patch makes the created deb package usable, the system is bootable.
We use the LOCALVERSION to feed in PR inforamtion and the KDEB_PKGVERSION is used fro additional information, like build ID.
Do you have any suggestion on how to get the functionality back or we must stop using the KDEB_PKGVERSION altogether?
Thank you,
Signature · 1 line
Péter
Quoted 62 lines
Signed-off-by Masahiro Yamada <[email protected]>---
Changes in v6:
- New patchscripts/package/deb-build-option | 16 ++++++++++++++++ scripts/package/mkdebian | 9 ++++----- 2 files changed, 20 insertions(+), 5 deletions(-) create mode 100755 scripts/package/deb-build-optionscripts/package/deb-build-option new file +16
-0@@ -0,0 +1,16 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only + +# Set up CROSS_COMPILE if we are cross-compiling, but not called from the +# kernel toplevel Makefile +if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUILD_ARCH}" ]; then + echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}- +fi + +version=$(dpkg-parsechangelog -S Version) +version_upstream="${version%-*}" +debian_revision="${version#${version_upstream}}" +debian_revision="${debian_revision#*-}" + +echo KERNELRELEASE=${version_upstream} +echo KBUILD_BUILD_VERSION=${debian_revision}scripts/package/mkdebian +4
-5@@ -90,10 +90,8 @@ rm -rf debian version=$KERNELRELEASE if [ -n "$KDEB_PKGVERSION" ]; then packageversion=$KDEB_PKGVERSION - revision=${packageversion##*-} else - revision=$($srctree/init/build-version) - packageversion=$version-$revision + packageversion=$version-$($srctree/init/build-version) fi sourcename=${KDEB_SOURCENAME:-linux-upstream}@@ -244,8 +242,9 @@ srctree ?= . build-indep: build-arch: - \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \ - KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile + \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} \ + \$(shell \$(srctree)/scripts/package/deb-build-option) \ + olddefconfig all build: build-arch
Signature · 1 line
Péter
Re: [PATCH v6 11/12] kbuild: deb-pkg: improve the usability of source package
Quoted 55 lines · Péter Ujfalusi
Quoted 8 lines · Masahiro Yamada
Improve the source package support in case the dpkg-buildpackage is directly used to build binary packages.
For cross-compiling, you can set
CROSS_COMPILEvia the environment variable, but it is better to set it automatically - set it to ${DEB_HOST_GNU_TYPE}- if we are cross-compiling but not from the topMakefile.Quoted 7 lines
The generated source package may be carried to a different build environment, which may have a different compiler installed. Run olddefconfig first to set new CONFIG options to their default values without prompting.
Take KERNELRELEASE and
KBUILD_BUILD_VERSIONfrom the version field of debian/changelog in case it is updated afterwards.I just arrived to this patch when trying to update the kernel for SOF from 6.2.0 base to 6.3.0-rc1:
https://github.com/thesofproject/linux/pull/4223#issuecomment-1459997676Our CI is building a kernel deb package from PRs to test with this command:
make bindeb-pkg LOCALVERSION=-$PRIDV-default KDEB_PKGVERSION=$(make kernelversion)-$GITSHAThe files, directories and the name of the deb package follows the rule set up by
scripts/setlocalversion, taking into account the LOCALVERSION,CONFIG_LOCALVERSIONandCONFIG_LOCALVERSION_AUTO, but the built kernel is using the version derived fromKDEB_PKGVERSIONand the system is not able to boot up since the modules are in a different directory then they should be.Something like this:
make bindeb-pkg LOCALVERSION=-testbuild vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/lib/modules/6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/strings vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty | grep 6.3 6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty
Then with the
KDEB_PKGVERSION:
make bindeb-pkg LOCALVERSION=-testbuild KDEB_PKGVERSION=$(make kernelversion)-notworking vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/lib/modules/6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty/strings vmlinuz-6.3.0-rc1-testbuild-05827-g863ad81a6aa9-dirty | grep 6.3 6.3.0-rc1
in this case modprobe will not find the modules, it is looking for them under
/lib/modules/6.3.0-rc1
Quoted 8 lines
Reverting this patch makes the created deb package usable, the system is bootable.
We use the LOCALVERSION to feed in PR inforamtion and the
KDEB_PKGVERSIONis used fro additional information, like build ID.Do you have any suggestion on how to get the functionality back or we must stop using the
KDEB_PKGVERSIONaltogether?
I am aware of this issue.
KERNELVERSION does not need to match the package version.
I should have taken it from the <abiver> in the package name (linux-image-<abiver>_<origver>-<rev>_<arch>.deb) if I had wanted to do this dynamically by debian/control.
I wrongly took it from <origver>.
I will change this when I solve another issue required by Linus Tolvalds.
Please wait.
Quoted 70 lines
Thank you,
--
PéterQuoted 52 lines
Signed-off-by Masahiro Yamada <[email protected]>---
Changes in v6:
- New patchscripts/package/deb-build-option | 16 ++++++++++++++++ scripts/package/mkdebian | 9 ++++----- 2 files changed, 20 insertions(+), 5 deletions(-) create mode 100755 scripts/package/deb-build-optionscripts/package/deb-build-option new file +16
-0@@ -0,0 +1,16 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only + +# Set up CROSS_COMPILE if we are cross-compiling, but not called from the +# kernel toplevel Makefile +if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUILD_ARCH}" ]; then + echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}- +fi + +version=$(dpkg-parsechangelog -S Version) +version_upstream="${version%-*}" +debian_revision="${version#${version_upstream}}" +debian_revision="${debian_revision#*-}" + +echo KERNELRELEASE=${version_upstream} +echo KBUILD_BUILD_VERSION=${debian_revision}scripts/package/mkdebian +1
-3@@ -90,10 +90,8 @@ rm -rf debian version=$KERNELRELEASE if [ -n "$KDEB_PKGVERSION" ]; then packageversion=$KDEB_PKGVERSION - revision=${packageversion##*-} else - revision=$($srctree/init/build-version) - packageversion=$version-$revision + packageversion=$version-$($srctree/init/build-version) fi sourcename=${KDEB_SOURCENAME:-linux-upstream}@@ -244,8 +242,9 @@ srctree ?= .scripts/package/mkdebian srctree ?= . · line 250
Quoted 10 lines
scripts/package/mkdebian +3
-2build-indep: build-arch: - \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \ - KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile + \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} \ + \$(shell \$(srctree)/scripts/package/deb-build-option) \ + olddefconfig all build: build-arch--
Péter
Signature · 2 lines
Best Regards Masahiro Yamada
PATCH v6 12/12 kbuild: add srcdeb-pkg target
In reply to Masahiro YamadaThis new target builds only the debian source package.
Factor similar code out to the 'build-deb-pkg' variable.
---
Changes in v6:
- New patch
scripts/Makefile.package | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
scripts/Makefile.package +20 -3
@@ -100,6 +100,9 @@ binrpm-pkg: +rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \ $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec +# deb-pkg srcdeb-pkg bindeb-pkg +# --------------------------------------------------------------------------- + quiet_cmd_debianize = GEN $@ cmd_debianize = $(srctree)/scripts/package/mkdebian
@@ -117,14 +120,27 @@ debian-orig: linux.tar.gz debian cp $< ../$(orig-name); \ fi +build-deb-pkg = \ + dpkg-buildpackage \ + --build=$(build-type) --no-pre-clean --unsigned-changes \ + $(if $(findstring source, $(build-type)), --unsigned-source) \ + $(if $(findstring binary, $(build-type)), -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch)) \ + $(DPKG_FLAGS) + PHONY += deb-pkg +deb-pkg: private build-type := source,binary deb-pkg: debian-orig - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ - --build=source,binary -nc -us -uc + +$(build-deb-pkg) + +PHONY += srcdeb-pkg +srcdeb-pkg: private build-type := source +srcdeb-pkg: debian-orig + +$(build-deb-pkg) PHONY += bindeb-pkg +bindeb-pkg: private build-type := binary bindeb-pkg: debian - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc + +$(build-deb-pkg) PHONY += intdeb-pkg intdeb-pkg:
@@ -225,6 +241,7 @@ help: @echo ' srcrpm-pkg - Build only the source kernel RPM package' @echo ' binrpm-pkg - Build only the binary kernel RPM package' @echo ' deb-pkg - Build both source and binary deb kernel packages' + @echo ' binsrc-pkg - Build only the source kernel deb package' @echo ' bindeb-pkg - Build only the binary kernel deb package' @echo ' snap-pkg - Build only the binary kernel snap package' @echo ' (will connect to external hosts)'
Signature · 1 line
2.34.1
Re: [PATCH v6 12/12] kbuild: add srcdeb-pkg target
Quoted 59 lines · On Wed, Feb 15, 2023 at 10:20:34AM +0900 Masahiro Yamada wrote:
This new target builds only the debian source package.
Factor similar code out to the 'build-deb-pkg' variable.
Signed-off-by Masahiro Yamada <[email protected]>---
Changes in v6:
- New patchscripts/Makefile.package | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)scripts/Makefile.package +19
-3@@ -100,6 +100,9 @@ binrpm-pkg: +rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \ $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec +# deb-pkg srcdeb-pkg bindeb-pkg +# --------------------------------------------------------------------------- + quiet_cmd_debianize = GEN $@ cmd_debianize = $(srctree)/scripts/package/mkdebian@@ -117,14 +120,27 @@ debian-orig: linux.tar.gz debian cp $< ../$(orig-name); \ fi +build-deb-pkg = \ + dpkg-buildpackage \ + --build=$(build-type) --no-pre-clean --unsigned-changes \ + $(if $(findstring source, $(build-type)), --unsigned-source) \ + $(if $(findstring binary, $(build-type)), -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch)) \ + $(DPKG_FLAGS) + PHONY += deb-pkg +deb-pkg: private build-type := source,binary deb-pkg: debian-orig - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ - --build=source,binary -nc -us -uc + +$(build-deb-pkg) + +PHONY += srcdeb-pkg +srcdeb-pkg: private build-type := source +srcdeb-pkg: debian-orig + +$(build-deb-pkg) PHONY += bindeb-pkg +bindeb-pkg: private build-type := binary bindeb-pkg: debian - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc + +$(build-deb-pkg) PHONY += intdeb-pkg intdeb-pkg:
scripts/Makefile.package help: · line 244
Quoted 4 lines
scripts/Makefile.package +1
-0@@ -225,6 +241,7 @@ help: @echo ' srcrpm-pkg - Build only the source kernel RPM package' @echo ' binrpm-pkg - Build only the binary kernel RPM package' @echo ' deb-pkg - Build both source and binary deb kernel packages' + @echo ' binsrc-pkg - Build only the source kernel deb package'
typo: srcdeb-dpkg
With this typo fixed:
Quoted 5 lines
scripts/Makefile.package +0
-0@echo ' bindeb-pkg - Build only the binary kernel deb package' @echo ' snap-pkg - Build only the binary kernel snap package' @echo ' (will connect to external hosts)'Signature · 1 line
2.34.1
Thanks for the patch (and the whole patch set), I appreciate it!
While testing, I stumbled over one thing, not really related to this patch:
As tar complains about any kind of tree changes while it creates an archive, 'olddefconfig' and 'srcdeb-pkg' must not be build at the same time. Is it sensible to add another 'mixed-build' check in top-level Makefile against all target depending on linux.tar.gz?
$ make olddefconfig srcdeb-pkg -j8
#
# No change to .config
#
GEN debian
UPD .tmp_filelist
TAR linux.tar.gz
tar: .: file changed as we read it
make[2]: *** [scripts/Makefile.package:58: linux.tar.gz] Error 1
make[2]: *** Deleting file 'linux.tar.gz'
make[1]: *** [Makefile:1654: srcdeb-pkg] Error 2
make: *** [Makefile:358: __build_one_by_one] Error 2Kind regards,
Nicolas
Re: [PATCH v6 12/12] kbuild: add srcdeb-pkg target
Quoted 58 lines · Nicolas Schier
Quoted 56 lines · On Wed, Feb 15, 2023 at 10:20:34AM +0900 Masahiro Yamada wrote:
This new target builds only the debian source package.
Factor similar code out to the 'build-deb-pkg' variable.
Signed-off-by Masahiro Yamada <[email protected]>---
Changes in v6:
- New patchscripts/Makefile.package | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)scripts/Makefile.package +19
-3@@ -100,6 +100,9 @@ binrpm-pkg: +rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \ $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec +# deb-pkg srcdeb-pkg bindeb-pkg +# --------------------------------------------------------------------------- + quiet_cmd_debianize = GEN $@ cmd_debianize = $(srctree)/scripts/package/mkdebian@@ -117,14 +120,27 @@ debian-orig: linux.tar.gz debian cp $< ../$(orig-name); \ fi +build-deb-pkg = \ + dpkg-buildpackage \ + --build=$(build-type) --no-pre-clean --unsigned-changes \ + $(if $(findstring source, $(build-type)), --unsigned-source) \ + $(if $(findstring binary, $(build-type)), -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch)) \ + $(DPKG_FLAGS) + PHONY += deb-pkg +deb-pkg: private build-type := source,binary deb-pkg: debian-orig - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ - --build=source,binary -nc -us -uc + +$(build-deb-pkg) + +PHONY += srcdeb-pkg +srcdeb-pkg: private build-type := source +srcdeb-pkg: debian-orig + +$(build-deb-pkg) PHONY += bindeb-pkg +bindeb-pkg: private build-type := binary bindeb-pkg: debian - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc + +$(build-deb-pkg)
Quoted 9 lines
scripts/Makefile.package help: · line 244
Quoted 7 lines
scripts/Makefile.package +1
-0PHONY += intdeb-pkg intdeb-pkg:@@ -225,6 +241,7 @@ help: @echo ' srcrpm-pkg - Build only the source kernel RPM package' @echo ' binrpm-pkg - Build only the binary kernel RPM package' @echo ' deb-pkg - Build both source and binary deb kernel packages' + @echo ' binsrc-pkg - Build only the source kernel deb package'typo: srcdeb-dpkg
I will fix. Thanks.
Quoted 12 lines
With this typo fixed:
Reviewed-by Nicolas Schier <[email protected]>Quoted 5 lines
scripts/Makefile.package +0
-1@echo ' bindeb-pkg - Build only the binary kernel deb package' @echo ' snap-pkg - Build only the binary kernel snap package' @echo ' (will connect to external hosts)' --2.34.1
Thanks for the patch (and the whole patch set), I appreciate it!
Quoted 8 lines
While testing, I stumbled over one thing, not really related to this patch:
As tar complains about any kind of tree changes while it creates an archive, 'olddefconfig' and 'srcdeb-pkg' must not be build at the same time. Is it sensible to add another 'mixed-build' check in top-level
Makefileagainst all target depending on linux.tar.gz?$ make olddefconfig srcdeb-pkg -j8
This is already done.
The mixture of config targets and non-config targets are treated as mixed-build. [1]
So, Kbuild internally divides it into 'make olddefconfig -j8' followed by 'make srcdeb-pkg -j8'.
[1] https://github.com/torvalds/linux/blob/v6.2/Makefile#L335
Quoted 11 lines
#
# No change to .config
#
GEN debian
UPD .tmp_filelist
TAR linux.tar.gz
tar: .: file changed as we read it
make[2]: *** [scripts/Makefile.package:58: linux.tar.gz] Error 1
make[2]: *** Deleting file 'linux.tar.gz'
make[1]: *** [Makefile:1654: srcdeb-pkg] Error 2
make: *** [Makefile:358: __build_one_by_one] Error 2I ran the same command, but I could not reproduce this.
Quoted 3 lines
Kind regards,
Nicolas
Signature · 2 lines
Best Regards Masahiro Yamada
Re: [PATCH v6 12/12] kbuild: add srcdeb-pkg target
Quoted 10 lines · Masahiro Yamada
Quoted 8 lines · Nicolas Schier
Quoted 6 lines · On Wed, Feb 15, 2023 at 10:20:34AM +0900 Masahiro Yamada wrote:
This new target builds only the debian source package.
Factor similar code out to the 'build-deb-pkg' variable.
Signed-off-by Masahiro Yamada <[email protected]>---
[...]
Quoted 8 lines
Quoted 8 lines
While testing, I stumbled over one thing, not really related to this patch:
As tar complains about any kind of tree changes while it creates an archive, 'olddefconfig' and 'srcdeb-pkg' must not be build at the same time. Is it sensible to add another 'mixed-build' check in top-level
Makefileagainst all target depending on linux.tar.gz?
Quoted 8 lines
Quoted 1 line
$ make olddefconfig srcdeb-pkg -j8This is already done.
The mixture of config targets and non-config targets are treated as mixed-build. [1]
So,Kbuildinternally divides it into 'make olddefconfig -j8' followed by 'make srcdeb-pkg -j8'.
oh yes, and already for quite some time. I didn't check carefully enough.
Quoted 20 lines
[1] https://github.com/torvalds/linux/blob/v6.2/Makefile#L335
Quoted 11 lines
# # No change to .config # GEN debian UPD .tmp_filelist TAR linux.tar.gz tar: .: file changed as we read it make[2]: *** [scripts/Makefile.package:58: linux.tar.gz] Error 1 make[2]: *** Deleting file 'linux.tar.gz' make[1]: *** [Makefile:1654: srcdeb-pkg] Error 2 make: *** [Makefile:358: __build_one_by_one] Error 2
Quoted 1 line
I ran the same command, but I could not reproduce this.
I cannot reproduce it any more; I don't know what might had been the problem on my site.
Kind regards,
Nicolas
Quoted 112 lines · Masahiro Yamada
In short, the motivation of this commit is to build a source package without cleaning the source tree.
The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.
Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.
In fact, there are some ways to achieve this.
The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses it, but I do not like it because it works only when the source tree is managed by git, and all files you want in the tarball must be committed in advance.
I want to make it work without relying on git. We can do this.
Files that are ignored by git are generated files, so should be excluded from the source tarball. We can list them out by parsing the .gitignore files. Of course, .gitignore does not cover all the cases, but it works well enough.
tar(1) claims to support it:
--exclude-vcs-ignores
Exclude files that match patterns read from VCS-specific ignore files. Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.
The best scenario would be to use 'tar --exclude-vcs-ignores', but this option does not work. --exclude-vcs-ignore does not understand any of the negation (!), preceding slash, following slash, etc.. So, this option is just useless.
Hence, I wrote this gitignore parser. The previous version [1], written in Python, was so slow. This version is implemented in C, so it works much faster.
I imported the code from git (commit:
23c56f7bd5f1), so we get the same result.This tool traverses the source tree, parsing all .gitignore files, and prints file paths that are ignored by git.
The output is similar to 'git ls-files --ignored --directory --others --exclude-per-directory=.gitignore', except
[1] Not sorted
[2] No trailing slash for directories[2] is intentional because tar's --exclude-from option cannot handle trailing slashes.
[How to test this tool]
$ git clean -dfx $ make -s -j$(nproc) defconfig all # or allmodconifg or whatever $ git archive -o ../linux1.tar --prefix=./ HEAD $ tar tf ../linux1.tar | LANG=C sort > ../file-list1 # files emitted by 'git archive' $ make scripts_package HOSTCC scripts/list-gitignored $ scripts/list-gitignored --prefix=./ -o ../exclude-list $ tar cf ../linux2.tar --exclude-from=../exclude-list . $ tar tf ../linux2.tar | LANG=C sort > ../file-list2 # files emitted by 'tar' $ diff ../file-list1 ../file-list2 | grep -E '^(<|>)' < ./Documentation/devicetree/bindings/.yamllint < ./drivers/clk/.kunitconfig < ./drivers/gpu/drm/tests/.kunitconfig < ./drivers/hid/.kunitconfig < ./fs/ext4/.kunitconfig < ./fs/fat/.kunitconfig < ./kernel/kcsan/.kunitconfig < ./lib/kunit/.kunitconfig < ./mm/kfence/.kunitconfig < ./tools/testing/selftests/arm64/tags/ < ./tools/testing/selftests/arm64/tags/.gitignore < ./tools/testing/selftests/arm64/tags/Makefile < ./tools/testing/selftests/arm64/tags/run_tags_test.sh < ./tools/testing/selftests/arm64/tags/tags_test.c < ./tools/testing/selftests/kvm/.gitignore < ./tools/testing/selftests/kvm/Makefile < ./tools/testing/selftests/kvm/config < ./tools/testing/selftests/kvm/settingsThe source tarball contains most of files that are tracked by git. You see some diffs, but it is just because some .gitignore files are wrong.
$ git ls-files -i -c --exclude-per-directory=.gitignore Documentation/devicetree/bindings/.yamllint drivers/clk/.kunitconfig drivers/gpu/drm/tests/.kunitconfig drivers/hid/.kunitconfig fs/ext4/.kunitconfig fs/fat/.kunitconfig kernel/kcsan/.kunitconfig lib/kunit/.kunitconfig mm/kfence/.kunitconfig tools/testing/selftests/arm64/tags/.gitignore tools/testing/selftests/arm64/tags/Makefile tools/testing/selftests/arm64/tags/run_tags_test.sh tools/testing/selftests/arm64/tags/tags_test.c tools/testing/selftests/kvm/.gitignore tools/testing/selftests/kvm/Makefile tools/testing/selftests/kvm/config tools/testing/selftests/kvm/settings
Quoted 4 lines
[1]: https://lore.kernel.org/all/[email protected]/
Signed-off-by Masahiro Yamada <[email protected]>---
01 - 11 applied to linux-kbuild.
I will update 12/12 (postponed to v6.4)
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 115 lines · Masahiro Yamada
Quoted 114 lines · Masahiro Yamada
In short, the motivation of this commit is to build a source package without cleaning the source tree.
The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.
Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.
In fact, there are some ways to achieve this.
The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses it, but I do not like it because it works only when the source tree is managed by git, and all files you want in the tarball must be committed in advance.
I want to make it work without relying on git. We can do this.
Files that are ignored by git are generated files, so should be excluded from the source tarball. We can list them out by parsing the .gitignore files. Of course, .gitignore does not cover all the cases, but it works well enough.
tar(1) claims to support it:
--exclude-vcs-ignores
Exclude files that match patterns read from VCS-specific ignore files. Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.
The best scenario would be to use 'tar --exclude-vcs-ignores', but this option does not work. --exclude-vcs-ignore does not understand any of the negation (!), preceding slash, following slash, etc.. So, this option is just useless.
Hence, I wrote this gitignore parser. The previous version [1], written in Python, was so slow. This version is implemented in C, so it works much faster.
I imported the code from git (commit:
23c56f7bd5f1), so we get the same result.This tool traverses the source tree, parsing all .gitignore files, and prints file paths that are ignored by git.
The output is similar to 'git ls-files --ignored --directory --others --exclude-per-directory=.gitignore', except
[1] Not sorted
[2] No trailing slash for directories[2] is intentional because tar's --exclude-from option cannot handle trailing slashes.
[How to test this tool]
$ git clean -dfx $ make -s -j$(nproc) defconfig all # or allmodconifg or whatever $ git archive -o ../linux1.tar --prefix=./ HEAD $ tar tf ../linux1.tar | LANG=C sort > ../file-list1 # files emitted by 'git archive' $ make scripts_package HOSTCC scripts/list-gitignored $ scripts/list-gitignored --prefix=./ -o ../exclude-list $ tar cf ../linux2.tar --exclude-from=../exclude-list . $ tar tf ../linux2.tar | LANG=C sort > ../file-list2 # files emitted by 'tar' $ diff ../file-list1 ../file-list2 | grep -E '^(<|>)' < ./Documentation/devicetree/bindings/.yamllint < ./drivers/clk/.kunitconfig < ./drivers/gpu/drm/tests/.kunitconfig < ./drivers/hid/.kunitconfig < ./fs/ext4/.kunitconfig < ./fs/fat/.kunitconfig < ./kernel/kcsan/.kunitconfig < ./lib/kunit/.kunitconfig < ./mm/kfence/.kunitconfig < ./tools/testing/selftests/arm64/tags/ < ./tools/testing/selftests/arm64/tags/.gitignore < ./tools/testing/selftests/arm64/tags/Makefile < ./tools/testing/selftests/arm64/tags/run_tags_test.sh < ./tools/testing/selftests/arm64/tags/tags_test.c < ./tools/testing/selftests/kvm/.gitignore < ./tools/testing/selftests/kvm/Makefile < ./tools/testing/selftests/kvm/config < ./tools/testing/selftests/kvm/settingsThe source tarball contains most of files that are tracked by git. You see some diffs, but it is just because some .gitignore files are wrong.
$ git ls-files -i -c --exclude-per-directory=.gitignore Documentation/devicetree/bindings/.yamllint drivers/clk/.kunitconfig drivers/gpu/drm/tests/.kunitconfig drivers/hid/.kunitconfig fs/ext4/.kunitconfig fs/fat/.kunitconfig kernel/kcsan/.kunitconfig lib/kunit/.kunitconfig mm/kfence/.kunitconfig tools/testing/selftests/arm64/tags/.gitignore tools/testing/selftests/arm64/tags/Makefile tools/testing/selftests/arm64/tags/run_tags_test.sh tools/testing/selftests/arm64/tags/tags_test.c tools/testing/selftests/kvm/.gitignore tools/testing/selftests/kvm/Makefile tools/testing/selftests/kvm/config tools/testing/selftests/kvm/settings
Quoted 8 lines
Quoted 2 lines
Signed-off-by Masahiro Yamada <[email protected]>---
01 - 11 applied to linux-kbuild.
I will update 12/12 (postponed to v6.4)
Might you want to re-consider to include this patch already in v6.3? Bastian seems to be unhappy with the current state of v6.3:
https://lore.kernel.org/linux-kbuild/[email protected]/T/#t
Kind regards,
Nicolas
Quoted 9 lines · Masahiro Yamada
In short, the motivation of this commit is to build a source package without cleaning the source tree.
The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.
Quoted 7 lines
Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.
In fact, there are some ways to achieve this.
The easiest solution is 'git archive'.
Eh, no, the easiest solution is to just don't build in-tree? Can you explain why 'make O=/over/there' isn't sufficient for whatever it is you really want to do here?
Rasmus
Quoted 17 lines · Rasmus Villemoes
Quoted 14 lines · Masahiro Yamada
In short, the motivation of this commit is to build a source package without cleaning the source tree.
The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.
Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree.
In fact, there are some ways to achieve this.
Quoted 7 lines
Quoted 1 line
The easiest solution is 'git archive'.
Eh, no, the easiest solution is to just don't build in-tree? Can you explain why 'make O=/over/there' isn't sufficient for whatever it is you really want to do here?
Rasmus
Right.
If you have a strict rule "I never build the kernel in-tree", your source tree is kept clean.
So, tar will never include generated files.
But, how would you know if the source tree is really clean?
--
Best Regards
Masahiro Yamada