Skip to contentSkip to conversation

kbuild: add a script to generate a list of files ignored by git

PATCH Superseded Masahiro Yamada 3 participants 8 messages Jan 28, 2023 – Jan 29, 2023

Subsystems (2)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/4 kbuild: add a script to generate a list of files ignored by git

In short, the motivation of this commit is to build a source package without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything.

Clearning the tree everytime is annoying since it makes the incremental build impossible. It is desireable to create a source tarball without cleaning the tree.

In fact, there are some ways to archive this.

The easiest way is "git archive". Actually, "make perf-tar*-src-pkg" does this way, but I do not like it because it works only when the source tree is managed by git, and all files you want in the tarball must be committed in advance.

I want to make it work without relying on git. We can do this.

Files that are not tracked by git are generated files. We can list them out by parsing the .gitignore files. Of course, .gitignore does not cover all the cases, but it works well mostly.

tar(1) claims to support it:

--exclude-vcs-ignores

Exclude files that match patterns read from VCS-specific ignore files. Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.

The best scenario would be to use "tar --exclude-vcs-ignores", but this option does not work. --exclude-vcs-ignore does not understand any of the negation (!), preceding slash, following slash. So, this option is just useless.

I explored some libraries.

- libgit2

libgit2 supports git_ignore_path_is_ignored(), but it only works in a git-managed tree. So, this does not fit to my purpose.

- Python's gitignore-parser

This library does not work. For example, if .gitignore is like this:

    $ cat .gitignore
    foo/

You will get this:

Quoted 1 line
Quoted 1 line
Quoted 1 line

gitignore_parser.parse_gitignore('.gitignore')('foo.a')

True

I do not understand why 'foo.a' matches 'foo/'. This library is buggy.

- Perl's Parse::Gitignore

It does not understand glob patterns at all. 'foo.o' does not match '*.o'. Hence, this library is also useless.

After all, I decided to write the code by myself. It is much faster. Hence, this tool.

The new script, scripts/gen-exclude.py, tarverses the source tree, parsing the .gitignore files. It prints the file paths that are not tracked by git. This is suitable to be passed to tar's --exclude-from= option.

A drawback is this script is so slow. It takes more than a minutes to traverse the kernel tree. It is better to rewrite it in C or something, but this is useful at least for proof of concept.

[How to test this script]

  $ git clean -dfx
  $ make -s -j$(nproc) defconfig all                       # or allmodconifg or whatever
  $ git archive -o ../linux1.tar --prefix=./ HEAD
  $ tar tf ../linux1.tar | LANG=C sort > ../file-list1     # files emitted by 'git archive'
  $ scripts/gen-exclude.py --prefix=./ > ../exclude-list
  $ tar cf ../linux2.tar --exclude-from=../exclude-list .
  $ tar tf ../linux2.tar | LANG=C sort > ../file-list2     # files emitted by 'tar'
  $ diff  ../file-list1 ../file-list2 | grep -E '^(<|>)'
  < ./Documentation/devicetree/bindings/.yamllint
  < ./drivers/clk/.kunitconfig
  < ./drivers/gpu/drm/tests/.kunitconfig
  < ./drivers/hid/.kunitconfig
  < ./fs/ext4/.kunitconfig
  < ./fs/fat/.kunitconfig
  < ./kernel/kcsan/.kunitconfig
  < ./lib/kunit/.kunitconfig
  < ./mm/kfence/.kunitconfig
  < ./tools/testing/selftests/arm64/tags/
  < ./tools/testing/selftests/arm64/tags/.gitignore
  < ./tools/testing/selftests/arm64/tags/Makefile
  < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
  < ./tools/testing/selftests/arm64/tags/tags_test.c
  < ./tools/testing/selftests/kvm/.gitignore
  < ./tools/testing/selftests/kvm/Makefile
  < ./tools/testing/selftests/kvm/config
  < ./tools/testing/selftests/kvm/settings

The source tarball contains most of files that are tracked by git. You see some diffs, but it is just because some .gitignore files are wrong.

  $ git ls-files -i -c --exclude-per-directory=.gitignore
  Documentation/devicetree/bindings/.yamllint
  drivers/clk/.kunitconfig
  drivers/gpu/drm/tests/.kunitconfig
  drivers/hid/.kunitconfig
  fs/ext4/.kunitconfig
  fs/fat/.kunitconfig
  kernel/kcsan/.kunitconfig
  lib/kunit/.kunitconfig
  mm/kfence/.kunitconfig
  tools/testing/selftests/arm64/tags/.gitignore
  tools/testing/selftests/arm64/tags/Makefile
  tools/testing/selftests/arm64/tags/run_tags_test.sh
  tools/testing/selftests/arm64/tags/tags_test.c
  tools/testing/selftests/kvm/.gitignore
  tools/testing/selftests/kvm/Makefile
  tools/testing/selftests/kvm/config
  tools/testing/selftests/kvm/settings

---

 scripts/gen-exclude.py | 201 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 201 insertions(+)
 create mode 100755 scripts/gen-exclude.py
scripts/gen-exclude.py new file +201 -0
new file mode 100755
index 000000000000..308c30779c79
@@ -0,0 +1,201 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0 only
+#
+# Copyright: Masahiro Yamada <[email protected]>
+
+import argparse
+import collections
+import logging
+import os
+import pathlib
+
+
+class IgnorePattern:
+
+    def __init__(self, pattern, dirpath):
+
+        logging.debug(f"    Add pattern: {pattern}")
+
+        self.orig = pattern
+        self.negate = False
+        self.dir_only = False
+        self.double_asterisk = False
+
+        # The prefix '!' negates the pattern.
+        if pattern[0] == '!':
+            self.negate = True
+            pattern = pattern[1:]
+
+        # a pattern with a trailing slash only matches to directories
+        if pattern[-1] == '/':
+            self.dir_only = True
+            pattern = pattern[:-1]
+
+        # FIXME:
+        # Escape sequence ('\') does not work correctly.
+        # Just strip a backslash at the beginning to support '\#*#'.
+        if pattern[0] == '\\':
+            pattern = pattern[1:]
+
+        # It is tricky to handle double asterisks ("**").
+        # pathlib's match() cannot handle it but glob() can.
+        if '**' in pattern:
+            self.double_asterisk = True
+            if pattern[0] == '/':
+                pattern = pattern[1:]
+
+            self.pattern = []
+
+            for f in dirpath.glob(pattern):
+                self.pattern.append(f.absolute())
+
+            return
+        # If a slash appears at the beginning or middle of the pattern,
+        # (e.g. "/a", "a/b", etc.) it is relative to the directory level.
+        elif '/' in pattern:
+            if pattern[0] == '/':
+                pattern = pattern[1:]
+            pattern = str(dirpath.joinpath(pattern).absolute())
+
+        self.pattern = pattern
+
+    def is_ignored(self, path):
+
+        if path.is_dir() and path.name == '.git':
+            logging.debug(f'  Ignore: {path}')
+            return True
+
+        if self.dir_only and not path.is_dir():
+            return None
+
+        if self.double_asterisk:
+            # Special handling for double asterisks
+            if not path.absolute() in self.pattern:
+                return None
+        else:
+            # make the path abosolute before matching, so it works for patterns
+            # that contain slash(es) such as "/a", "a/b".
+            if not path.absolute().match(self.pattern):
+                return None
+
+        logging.debug(f'  {"Not" if self.negate else ""}Ignore: {path} (pattern: {self.orig})')
+
+        return not self.negate
+
+
+class GitIgnore:
+    def __init__(self, path):
+        # If a path matches multiple patterns, the last one wins.
+        # So, patterns must be checked in the reverse order.
+        # Use deque's appendleft() to add a new pattern.
+        self.patterns = collections.deque()
+        if path is not None:
+            logging.debug(f"  Add gitignore: {path}")
+            self._parse(path)
+
+    def _parse(self, path):
+        with open(path) as f:
+            for line in f:
+                if line.startswith('#'):
+                    continue
+
+                line = line.strip()
+                if not line:
+                    continue
+
+                self.patterns.appendleft(IgnorePattern(line, path.parent))
+
+    def is_ignored(self, path):
+        for pattern in self.patterns:
+            ret = pattern.is_ignored(path)
+            if ret is not None:
+                return ret
+
+        return None
+
+
+class GitIgnores:
+    def __init__(self):
+        self.ignore_files = collections.deque()
+
+    def append(self, path):
+        self.ignore_files.appendleft(GitIgnore(path))
+
+    def pop(self):
+        self.ignore_files.popleft()
+
+    def is_ignored(self, path):
+
+        for gitignore in self.ignore_files:
+            ret = gitignore.is_ignored(path)
+            if ret is not None:
+                return ret
+
+        logging.debug(f"  NoMatch: {path}")
+
+        return False
+
+
+def parse_arguments():
+
+    parser = argparse.ArgumentParser(description='Print files that are not ignored by .gitignore')
+
+    parser.add_argument('-d', '--debug', action='store_true', help='enable debug log')
+
+    parser.add_argument('--prefix', default='', help='prefix added to each path')
+
+    parser.add_argument('--rootdir', default='.', help='root of the source tree (default: current working directory)')
+
+    args = parser.parse_args()
+
+    return args
+
+
+def traverse_directory(path, gitignores, printer):
+
+    logging.debug(f"Enter: {path}")
+
+    gitignore_file = None
+    all_ignored = True
+
+    for f in path.iterdir():
+        if f.is_file() and f.name == '.gitignore':
+            gitignore_file = f
+
+    gitignores.append(gitignore_file)
+
+    for f in path.iterdir():
+        logging.debug(f"  Check: {f}")
+        if gitignores.is_ignored(f):
+            printer(f)
+        else:
+            if f.is_dir() and not f.is_symlink():
+                ret = traverse_directory(f, gitignores, printer)
+                all_ignored = all_ignored and ret
+            else:
+                all_ignored = False
+
+    if all_ignored:
+        printer(path)
+
+    gitignores.pop()
+    logging.debug(f"Leave: {path}")
+
+    return all_ignored
+
+
+def main():
+
+    args = parse_arguments()
+
+    logging.basicConfig(format='%(levelname)s: %(message)s',
+                        level='DEBUG' if args.debug else 'WARNING')
+
+    os.chdir(args.rootdir)
+
+    traverse_directory(pathlib.Path('.'), GitIgnores(),
+                       lambda path: print(f'{args.prefix}{str(path)}'))
+
+
+if __name__ == '__main__':
+    main()
Signature · 1 line
2.34.1
#1 lore raw

PATCH 2/4 kbuild: deb-pkg: create source package without cleaning

If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.

Instead of cleaning, pass the exclude list to tar's --exclude-from option.

Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig.

With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the tarball.

---

 scripts/Makefile.package | 27 ++++++++++++++++++++++-----
 scripts/package/mkdebian | 27 +++++++++++++++++++++++++++
 2 files changed, 49 insertions(+), 5 deletions(-)
scripts/Makefile.package +22 -5
index dfbf40454a99..cb135c99a273 100644
@@ -50,6 +50,21 @@ fi ; \
 tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
 	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
 
+# Source Tarball
+# ---------------------------------------------------------------------------
+
+quiet_cmd_exclude_list = GEN     $@
+      cmd_exclude_list = $(srctree)/scripts/gen-exclude.py --prefix=./ --rootdir=$(srctree) > $@; echo "./$@" >> $@
+
+.exclude-list: FORCE
+	$(call cmd,exclude_list)
+
+quiet_cmd_tar = TAR     $@
+      cmd_tar = tar -I $(KGZIP) -c -f $@ -C $(srctree) --exclude-from=$< --exclude=./$@ --transform 's:^\.:linux:S' .
+
+%.tar.gz: .exclude-list
+	$(call cmd,tar)
+
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
@@ -81,12 +96,11 @@ binrpm-pkg:
 
 PHONY += deb-pkg
 deb-pkg:
-	$(MAKE) clean
 	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-	$(call cmd,src_tar,$(KDEB_SOURCENAME))
-	origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-		mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc
+	$(Q)origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
+		$(MAKE) -f $(srctree)/scripts/Makefile.package ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
+		--build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg:
@@ -174,4 +188,7 @@ help:
 	@echo '  perf-tarxz-src-pkg  - Build $(perf-tar).tar.xz source tarball'
 	@echo '  perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
 
+PHONY += FORCE
+FORCE:
+
 .PHONY: $(PHONY)
scripts/package/mkdebian +27 -0
index c3bbef7a6754..12c057ffbe6e 100755
@@ -84,6 +84,8 @@ set_debarch() {
 	fi
 }
 
+rm -rf debian
+
 # Some variables and settings used throughout the script
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
@@ -135,6 +137,31 @@ fi
 mkdir -p debian/source/
 echo "1.0" > debian/source/format
 
+cat<<'EOF' > debian/source/local-options
+#
+# Ugly: ignore anything except .config or debian/
+# (is there a cleaner way to do this?)
+#
+diff-ignore
+
+extend-diff-ignore = ^[^.d]
+
+extend-diff-ignore = ^\.[^c]
+extend-diff-ignore = ^\.c($|[^o])
+extend-diff-ignore = ^\.co($|[^n])
+extend-diff-ignore = ^\.con($|[^f])
+extend-diff-ignore = ^\.conf($|[^i])
+extend-diff-ignore = ^\.confi($|[^g])
+extend-diff-ignore = ^\.config.
+
+extend-diff-ignore = ^d($|[^e])
+extend-diff-ignore = ^de($|[^b])
+extend-diff-ignore = ^deb($|[^i])
+extend-diff-ignore = ^debi($|[^a])
+extend-diff-ignore = ^debia($|[^n])
+extend-diff-ignore = ^debian[^/]
+EOF
+
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
 extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 2/4] kbuild: deb-pkg: create source package without cleaning

Quoted 4 lines · Masahiro Yamada

If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.

Quoted 9 lines

Instead of cleaning, pass the exclude list to tar's --exclude-from option.

Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig.

With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the tarball.

Thanks for the patch.

While at this...

...why not switch over to Debian's packaging default XZ compressor:
*.orig.xz and *.diff.xz (or *.debian.tar.xz)?

EXAMPLE binutils:

DSC: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40-2.dsc
TAR: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40.orig.tar.xz
DIFF: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40-2.debian.tar.xz

-Sedat-

[1] https://packages.debian.org/sid/binutils

Quoted 106 lines

---

 scripts/Makefile.package | 27 ++++++++++++++++++++++-----
 scripts/package/mkdebian | 27 +++++++++++++++++++++++++++
 2 files changed, 49 insertions(+), 5 deletions(-)
scripts/Makefile.package +22 -5
index dfbf40454a99..cb135c99a273 100644
@@ -50,6 +50,21 @@ fi ; \
 tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
        --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
 
+# Source Tarball
+# ---------------------------------------------------------------------------
+
+quiet_cmd_exclude_list = GEN     $@
+      cmd_exclude_list = $(srctree)/scripts/gen-exclude.py --prefix=./ --rootdir=$(srctree) > $@; echo "./$@" >> $@
+
+.exclude-list: FORCE
+       $(call cmd,exclude_list)
+
+quiet_cmd_tar = TAR     $@
+      cmd_tar = tar -I $(KGZIP) -c -f $@ -C $(srctree) --exclude-from=$< --exclude=./$@ --transform 's:^\.:linux:S' .
+
+%.tar.gz: .exclude-list
+       $(call cmd,tar)
+
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
@@ -81,12 +96,11 @@ binrpm-pkg:
 
 PHONY += deb-pkg
 deb-pkg:
-       $(MAKE) clean
        $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-       $(call cmd,src_tar,$(KDEB_SOURCENAME))
-       origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-               mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
-       +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc
+       $(Q)origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
+               $(MAKE) -f $(srctree)/scripts/Makefile.package ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+       +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
+               --build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg:
@@ -174,4 +188,7 @@ help:
        @echo '  perf-tarxz-src-pkg  - Build $(perf-tar).tar.xz source tarball'
        @echo '  perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
 
+PHONY += FORCE
+FORCE:
+
 .PHONY: $(PHONY)
scripts/package/mkdebian +27 -1
index c3bbef7a6754..12c057ffbe6e 100755
@@ -84,6 +84,8 @@ set_debarch() {
        fi
 }
 
+rm -rf debian
+
 # Some variables and settings used throughout the script
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
@@ -135,6 +137,31 @@ fi
 mkdir -p debian/source/
 echo "1.0" > debian/source/format
 
+cat<<'EOF' > debian/source/local-options
+#
+# Ugly: ignore anything except .config or debian/
+# (is there a cleaner way to do this?)
+#
+diff-ignore
+
+extend-diff-ignore = ^[^.d]
+
+extend-diff-ignore = ^\.[^c]
+extend-diff-ignore = ^\.c($|[^o])
+extend-diff-ignore = ^\.co($|[^n])
+extend-diff-ignore = ^\.con($|[^f])
+extend-diff-ignore = ^\.conf($|[^i])
+extend-diff-ignore = ^\.confi($|[^g])
+extend-diff-ignore = ^\.config.
+
+extend-diff-ignore = ^d($|[^e])
+extend-diff-ignore = ^de($|[^b])
+extend-diff-ignore = ^deb($|[^i])
+extend-diff-ignore = ^debi($|[^a])
+extend-diff-ignore = ^debia($|[^n])
+extend-diff-ignore = ^debian[^/]
+EOF
+
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
 extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"
--

2.34.1

#3 lore raw

Re: [PATCH 2/4] kbuild: deb-pkg: create source package without cleaning

Quoted 16 lines · Sedat Dilek
Quoted 4 lines · Masahiro Yamada

If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.

Quoted 9 lines

Instead of cleaning, pass the exclude list to tar's --exclude-from option.

Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig.

With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the tarball.

Quoted 6 lines

Thanks for the patch.

While at this...

...why not switch over to Debian's packaging default XZ compressor:
*.orig.xz and *.diff.xz (or *.debian.tar.xz)?

Does debian support *.diff.xz?
I do not think so.

*.debian.tar.xz requires "Format: 3.0 (quilt)" migration.

See scripts/package/mkdebian.
We use "Format: 1.0", which only supports gzip.

It is true the Debian kernel uses "Format: 3.0 (quilt)", but I do not think it will fit to the upstream kernel.

Rather, I want to use the Native format since I do not see much sense in the *.orig.tar.gz / *.diff.gz split for the upstream project.

My plan is to stop cleaning first,
then change the source format if it is desirable.

Quoted 4 lines

EXAMPLE binutils:

DSC: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40-2.dsc

This is "Format: 3.0 (quilt)".

Quoted 2 lines

TAR: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40.orig.tar.xz
DIFF: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40-2.debian.tar.xz

This is not a diff. It is a tarball of the debian/ directory. Real diffs are stored in debian/patches/.

Quoted 103 lines

-Sedat-

[1] https://packages.debian.org/sid/binutils

Quoted 98 lines

---

 scripts/Makefile.package | 27 ++++++++++++++++++++++-----
 scripts/package/mkdebian | 27 +++++++++++++++++++++++++++
 2 files changed, 49 insertions(+), 5 deletions(-)
scripts/Makefile.package +22 -5
index dfbf40454a99..cb135c99a273 100644
@@ -50,6 +50,21 @@ fi ; \
 tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
        --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
 
+# Source Tarball
+# ---------------------------------------------------------------------------
+
+quiet_cmd_exclude_list = GEN     $@
+      cmd_exclude_list = $(srctree)/scripts/gen-exclude.py --prefix=./ --rootdir=$(srctree) > $@; echo "./$@" >> $@
+
+.exclude-list: FORCE
+       $(call cmd,exclude_list)
+
+quiet_cmd_tar = TAR     $@
+      cmd_tar = tar -I $(KGZIP) -c -f $@ -C $(srctree) --exclude-from=$< --exclude=./$@ --transform 's:^\.:linux:S' .
+
+%.tar.gz: .exclude-list
+       $(call cmd,tar)
+
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
@@ -81,12 +96,11 @@ binrpm-pkg:
 
 PHONY += deb-pkg
 deb-pkg:
-       $(MAKE) clean
        $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-       $(call cmd,src_tar,$(KDEB_SOURCENAME))
-       origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-               mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
-       +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc
+       $(Q)origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
+               $(MAKE) -f $(srctree)/scripts/Makefile.package ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+       +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
+               --build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg:
@@ -174,4 +188,7 @@ help:
        @echo '  perf-tarxz-src-pkg  - Build $(perf-tar).tar.xz source tarball'
        @echo '  perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
 
+PHONY += FORCE
+FORCE:
+
 .PHONY: $(PHONY)
scripts/package/mkdebian +25 -0
index c3bbef7a6754..12c057ffbe6e 100755
@@ -84,6 +84,8 @@ set_debarch() {
        fi
 }
 
+rm -rf debian
+
 # Some variables and settings used throughout the script
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
@@ -135,6 +137,31 @@ fi
 mkdir -p debian/source/
 echo "1.0" > debian/source/format
 
+cat<<'EOF' > debian/source/local-options
+#
+# Ugly: ignore anything except .config or debian/
+# (is there a cleaner way to do this?)
+#
+diff-ignore
+
+extend-diff-ignore = ^[^.d]
+
+extend-diff-ignore = ^\.[^c]
+extend-diff-ignore = ^\.c($|[^o])
+extend-diff-ignore = ^\.co($|[^n])
+extend-diff-ignore = ^\.con($|[^f])
+extend-diff-ignore = ^\.conf($|[^i])
+extend-diff-ignore = ^\.confi($|[^g])
+extend-diff-ignore = ^\.config.
+
+extend-diff-ignore = ^d($|[^e])
+extend-diff-ignore = ^de($|[^b])
+extend-diff-ignore = ^deb($|[^i])
+extend-diff-ignore = ^debi($|[^a])
+extend-diff-ignore = ^debia($|[^n])
+extend-diff-ignore = ^debian[^/]
Quoted 8 lines
Quoted 8 lines
scripts/package/mkdebian +2 -1
+EOF
+
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
 extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"
--

2.34.1

--
Best Regards
Masahiro Yamada

#4 lore raw

Re: [PATCH 2/4] kbuild: deb-pkg: create source package without cleaning

Quoted 26 lines · Masahiro Yamada
Quoted 16 lines · Sedat Dilek
Quoted 4 lines · Masahiro Yamada

If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.

Quoted 9 lines

Instead of cleaning, pass the exclude list to tar's --exclude-from option.

Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig.

With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the tarball.

Quoted 6 lines

Thanks for the patch.

While at this...

...why not switch over to Debian's packaging default XZ compressor:
*.orig.xz and *.diff.xz (or *.debian.tar.xz)?

Quoted 9 lines

Does debian support *.diff.xz?
I do not think so.

*.debian.tar.xz requires "Format: 3.0 (quilt)" migration.

See scripts/package/mkdebian.
We use "Format: 1.0", which only supports gzip.

Ah OK.

Quoted 4 lines

It is true the Debian kernel uses "Format: 3.0 (quilt)", but I do not think it will fit to the upstream kernel.

Quoted 8 lines

Rather, I want to use the Native format since I do not see much sense in the *.orig.tar.gz / *.diff.gz split for the upstream project.

My plan is to stop cleaning first,
then change the source format if it is desirable.

Yes, makes sense.

-Sedat-

Quoted 140 lines
Quoted 8 lines

EXAMPLE binutils:

DSC: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40-2.dsc

This is "Format: 3.0 (quilt)".

Quoted 2 lines

TAR: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40.orig.tar.xz
DIFF: http://deb.debian.org/debian/pool/main/b/binutils/binutils_2.40-2.debian.tar.xz

This is not a diff. It is a tarball of the debian/ directory. Real diffs are stored in debian/patches/.

Quoted 103 lines

-Sedat-

[1] https://packages.debian.org/sid/binutils

Quoted 98 lines

---

 scripts/Makefile.package | 27 ++++++++++++++++++++++-----
 scripts/package/mkdebian | 27 +++++++++++++++++++++++++++
 2 files changed, 49 insertions(+), 5 deletions(-)
scripts/Makefile.package +22 -5
index dfbf40454a99..cb135c99a273 100644
@@ -50,6 +50,21 @@ fi ; \
 tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
        --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
 
+# Source Tarball
+# ---------------------------------------------------------------------------
+
+quiet_cmd_exclude_list = GEN     $@
+      cmd_exclude_list = $(srctree)/scripts/gen-exclude.py --prefix=./ --rootdir=$(srctree) > $@; echo "./$@" >> $@
+
+.exclude-list: FORCE
+       $(call cmd,exclude_list)
+
+quiet_cmd_tar = TAR     $@
+      cmd_tar = tar -I $(KGZIP) -c -f $@ -C $(srctree) --exclude-from=$< --exclude=./$@ --transform 's:^\.:linux:S' .
+
+%.tar.gz: .exclude-list
+       $(call cmd,tar)
+
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
@@ -81,12 +96,11 @@ binrpm-pkg:
 
 PHONY += deb-pkg
 deb-pkg:
-       $(MAKE) clean
        $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
-       $(call cmd,src_tar,$(KDEB_SOURCENAME))
-       origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
-               mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
-       +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc
+       $(Q)origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
+               $(MAKE) -f $(srctree)/scripts/Makefile.package ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
+       +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \
+               --build=source,binary --source-option=-sP -nc -us -uc
 
 PHONY += bindeb-pkg
 bindeb-pkg:
@@ -174,4 +188,7 @@ help:
        @echo '  perf-tarxz-src-pkg  - Build $(perf-tar).tar.xz source tarball'
        @echo '  perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
 
+PHONY += FORCE
+FORCE:
+
 .PHONY: $(PHONY)
scripts/package/mkdebian +25 -0
index c3bbef7a6754..12c057ffbe6e 100755
@@ -84,6 +84,8 @@ set_debarch() {
        fi
 }
 
+rm -rf debian
+
 # Some variables and settings used throughout the script
 version=$KERNELRELEASE
 if [ -n "$KDEB_PKGVERSION" ]; then
@@ -135,6 +137,31 @@ fi
 mkdir -p debian/source/
 echo "1.0" > debian/source/format
 
+cat<<'EOF' > debian/source/local-options
+#
+# Ugly: ignore anything except .config or debian/
+# (is there a cleaner way to do this?)
+#
+diff-ignore
+
+extend-diff-ignore = ^[^.d]
+
+extend-diff-ignore = ^\.[^c]
+extend-diff-ignore = ^\.c($|[^o])
+extend-diff-ignore = ^\.co($|[^n])
+extend-diff-ignore = ^\.con($|[^f])
+extend-diff-ignore = ^\.conf($|[^i])
+extend-diff-ignore = ^\.confi($|[^g])
+extend-diff-ignore = ^\.config.
+
+extend-diff-ignore = ^d($|[^e])
+extend-diff-ignore = ^de($|[^b])
+extend-diff-ignore = ^deb($|[^i])
+extend-diff-ignore = ^debi($|[^a])
+extend-diff-ignore = ^debia($|[^n])
+extend-diff-ignore = ^debian[^/]
Quoted 8 lines
Quoted 8 lines
scripts/package/mkdebian +2 -1
+EOF
+
 echo $debarch > debian/arch
 extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)"
 extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)"
--

2.34.1

--
Best Regards
Masahiro Yamada

#5 lore raw

PATCH 3/4 kbuild: rpm-pkg: build binary packages from source rpm

In reply to Masahiro Yamada

The build rules of rpm-pkg and srcrpm-pkg are almost the same. Remove the code duplication.

Make rpm-pkg build binary packages from the source package built by srcrpm-pkg.

---

 scripts/Makefile.package | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
scripts/Makefile.package +3 -5
index cb135c99a273..8fa6f05967aa 100644
@@ -68,11 +68,9 @@ quiet_cmd_tar = TAR     $@
 # rpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += rpm-pkg
-rpm-pkg:
-	$(MAKE) clean
-	$(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec
-	$(call cmd,src_tar,$(KERNELPATH),kernel.spec)
-	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ta $(KERNELPATH).tar.gz \
+rpm-pkg: srpm = $(shell rpmspec --srpm --query --queryformat='%{name}-%{VERSION}-%{RELEASE}.src.rpm' kernel.spec)
+rpm-pkg: srcrpm-pkg
+	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -rb $(srpm) \
 	--define='_smp_mflags %{nil}'
 
 # srcrpm-pkg
Signature · 1 line
2.34.1
#6 lore raw

PATCH 4/4 kbuild: srcrpm-pkg: create source package without cleaning

In reply to Masahiro Yamada

If you run 'make (src)rpm-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible.

Instead of cleaning, pass the exclude list to tar's --exclude-from option.

Previously, the .config was contained in the source tarball.

With this commit, the source rpm consists of separate linux.tar.gz and .config.

Remove stale comments. Now, 'make (src)rpm-pkg' works with O= option.

---

 scripts/Makefile.package | 50 +++-------------------------------------
 scripts/package/mkspec   |  8 +++----
 2 files changed, 7 insertions(+), 51 deletions(-)
scripts/Makefile.package +3 -47
index 8fa6f05967aa..b899fe747bc9 100644
@@ -3,53 +3,11 @@
 
 include $(srctree)/scripts/Kbuild.include
 
-# RPM target
-# ---------------------------------------------------------------------------
-# The rpm target generates two rpm files:
-# /usr/src/packages/SRPMS/kernel-2.6.7rc2-1.src.rpm
-# /usr/src/packages/RPMS/i386/kernel-2.6.7rc2-1.<arch>.rpm
-# The src.rpm files includes all source for the kernel being built
-# The <arch>.rpm includes kernel configuration, modules etc.
-#
-# Process to create the rpm files
-# a) clean the kernel
-# b) Generate .spec file
-# c) Build a tar ball, using symlink to make kernel version
-#    first entry in the path
-# d) and pack the result to a tar.gz file
-# e) generate the rpm files, based on kernel.spec
-# - Use /. to avoid tar packing just the symlink
-
-# Note that the rpm-pkg target cannot be used with KBUILD_OUTPUT,
-# but the binrpm-pkg target can; for some reason O= gets ignored.
-
-# Remove hyphens since they have special meaning in RPM filenames
-KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE))
 KDEB_SOURCENAME ?= linux-upstream
 KBUILD_PKG_ROOTCMD ?="fakeroot -u"
 export KDEB_SOURCENAME
-# Include only those top-level files that are needed by make, plus the GPL copy
-TAR_CONTENT := Documentation LICENSES arch block certs crypto drivers fs \
-               include init io_uring ipc kernel lib mm net rust \
-               samples scripts security sound tools usr virt \
-               .config Makefile \
-               Kbuild Kconfig COPYING $(wildcard localversion*)
 MKSPEC     := $(srctree)/scripts/package/mkspec
 
-quiet_cmd_src_tar = TAR     $(2).tar.gz
-      cmd_src_tar = \
-if test "$(objtree)" != "$(srctree)"; then \
-	echo >&2; \
-	echo >&2 "  ERROR:"; \
-	echo >&2 "  Building source tarball is not possible outside the"; \
-	echo >&2 "  kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \
-	echo >&2 "  binrpm-pkg or bindeb-pkg target instead."; \
-	echo >&2; \
-	false; \
-fi ; \
-tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
-	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
-
 # Source Tarball
 # ---------------------------------------------------------------------------
 
@@ -76,12 +34,10 @@ rpm-pkg: srcrpm-pkg
 # srcrpm-pkg
 # ---------------------------------------------------------------------------
 PHONY += srcrpm-pkg
-srcrpm-pkg:
-	$(MAKE) clean
+srcrpm-pkg: linux.tar.gz
 	$(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec
-	$(call cmd,src_tar,$(KERNELPATH),kernel.spec)
-	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ts $(KERNELPATH).tar.gz \
-	--define='_smp_mflags %{nil}' --define='_srcrpmdir $(srctree)'
+	+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -bs kernel.spec \
+	--define='_smp_mflags %{nil}' --define='_sourcedir .' --define='_srcrpmdir .'
 
 # binrpm-pkg
 # ---------------------------------------------------------------------------
scripts/package/mkspec +4 -4
index 108c0cb95436..83a64d9d7372 100755
@@ -47,7 +47,8 @@ sed -e '/^DEL/d' -e 's/^\t*//' <<EOF
 	Group: System Environment/Kernel
 	Vendor: The Linux Community
 	URL: https://www.kernel.org
-$S	Source: kernel-$__KERNELRELEASE.tar.gz
+$S	Source0: linux.tar.gz
+$S	Source1: .config
 	Provides: $PROVIDES
 $S	BuildRequires: bc binutils bison dwarves
 $S	BuildRequires: (elfutils-libelf-devel or libelf-devel) flex
@@ -83,9 +84,8 @@ $S$M	This package provides kernel headers and makefiles sufficient to build modu
 $S$M	against the $__KERNELRELEASE kernel package.
 $S$M
 $S	%prep
-$S	%setup -q
-$S	rm -f scripts/basic/fixdep scripts/kconfig/conf
-$S	rm -f tools/objtool/{fixdep,objtool}
+$S	%setup -q -n linux
+$S	cp %{SOURCE1} .
 $S
 $S	%build
 $S	$MAKE %{?_smp_mflags} KERNELRELEASE=$KERNELRELEASE KBUILD_BUILD_VERSION=%{release}
Signature · 1 line
2.34.1
#7 lore raw
In reply to Masahiro Yamada
I love your patch! Perhaps something to improve:

Hi Masahiro,

I love your patch! Perhaps something to improve:

[auto build test WARNING on masahiroy-kbuild/for-next]
[also build test WARNING on masahiroy-kbuild/fixes linus/master v6.2-rc5 next-20230127] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Masahiro-Yamada/kbuild-deb-pkg-create-source-package-without-cleaning/20230129-014015
base:   https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git for-next
patch link:    https://lore.kernel.org/r/20230128173843.765212-1-masahiroy%40kernel.org
patch subject: [PATCH 1/4] kbuild: add a script to generate a list of files ignored by git

reproduce:
scripts/spdxcheck.py

If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot <[email protected]>

spdxcheck warnings: (new ones prefixed by >>)

>> scripts/gen-exclude.py: 2:35 Invalid License ID: only
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests