kbuild: deb-pkg: Allow parallel build
cover kbuild: deb-pkg: Allow parallel build
Make use of DEB_BUILD_OPTIONS' parallel option.
Bastian Germann (1):
kbuild: deb-pkg: Allow parallel build
scripts/package/deb-build-option | 6 ++++++ 1 file changed, 6 insertions(+)
Signature · 1 line
2.39.2
PATCH 1/1 kbuild: deb-pkg: Allow parallel build
Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.
Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable that ends up being set by Debian's build tools.
--- scripts/package/deb-build-option | 6 ++++++ 1 file changed, 6 insertions(+)
scripts/package/deb-build-option +6 -0
@@ -7,6 +7,12 @@ if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUI echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}- fi +for build_opt in $DEB_BUILD_OPTIONS; do + if [ "${build_opt#parallel=}" != "$build_opt" ]; then + echo -j${build_opt#parallel=} + fi +done + version=$(dpkg-parsechangelog -S Version) version_upstream="${version%-*}" debian_revision="${version#${version_upstream}}"
Signature · 1 line
2.39.2
Quoted 4 lines · Bastian Germann
Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.
"dpkg-buildpackage -b -j16" worked for me.
Quoted 29 lines
Derive make's -j parameter from the
DEB_BUILD_OPTIONSenvironment variable that ends up being set by Debian's build tools.Signed-off-by Bastian Germann <[email protected]>--- scripts/package/deb-build-option | 6 ++++++ 1 file changed, 6 insertions(+)scripts/package/deb-build-option +6
-1@@ -7,6 +7,12 @@ if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUI echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}- fi +for build_opt in $DEB_BUILD_OPTIONS; do + if [ "${build_opt#parallel=}" != "$build_opt" ]; then + echo -j${build_opt#parallel=} + fi +done + version=$(dpkg-parsechangelog -S Version) version_upstream="${version%-*}" debian_revision="${version#${version_upstream}}" --2.39.2
Signature · 2 lines
Best Regards Masahiro Yamada
Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
Quoted 9 lines · Bastian Germann
Quoted 4 lines
Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.
"dpkg-buildpackage -b -j16" worked for me.
This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:/usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all
So it is not used to run the actual build, just the top level make -f debian/rules invocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.
Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile or dpkg-buildpackage.
Quoted 10 lines · Bastian Germann
Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
Quoted 8 lines · Bastian Germann
Quoted 4 lines
Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.
Quoted 4 lines
Quoted 1 line
"dpkg-buildpackage -b -j16" worked for me.
This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
/usr/bin/make-f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all
"dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS but also MAKEFLAGS.
This is clearly explained in "man dpkb-buildpackage".
-j, --jobs[=jobs|auto]
...
Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...
Your statement sounds like
'MAKEFLAGS=-j<N> dpkg-buildpackage'
is the only way to build packages in parallel.
Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.
Quoted 4 lines
So it is not used to run the actual build, just the top level
make -f debian/rulesinvocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.
What is the "usual" way in this context?
Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?
If so, I agree. This patch caters to this case.
But, I think your code should go to debian/rules instead of scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.
Also, the commit description should explain the benefit of this patch.
Quoted 3 lines
Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux
Makefileor dpkg-buildpackage.
Try:
$ make deb-pkg -j16
and
$ dpkg-buildpackage -b -j16
Both run the package building in parallel.
You see a big difference in build speed w/wo the -j option. (and the build logs are interleaved, since they are emitted by multiple threads)
--
Best Regards
Masahiro Yamada
Am 15.03.23 um 15:18 schrieb Masahiro Yamada:
Quoted 4 lines
"dpkb-buildpackage -j<N>" sets not only
DEB_BUILD_OPTIONSbut also MAKEFLAGS.
Quoted 8 lines
This is clearly explained in "man dpkb-buildpackage".
-j, --jobs[=jobs|auto]
...Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...
I see. The testing/unstable dpkg-buildpackage version changed the -j behaviour. -j only sets DEB_BUILD_OPTIONS=parallel= now.
The man page now says:
--jobs-force[=jobs|auto]
This option (since dpkg 1.21.10) is equivalent to the --jobs option except that it will enable forced parallel mode, by adding the make -j option with the computed number of parallel jobs to the MAKEFLAGS environment variable.
Quoted 5 lines
Your statement sounds like
'MAKEFLAGS=-j<N> dpkg-buildpackage'
is the only way to build packages in parallel.
It is in v1.21.10 or later (or using --jobs-force which does the same thing).
Quoted 7 lines
Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
So it is not used to run the actual build, just the top level
make -f debian/rulesinvocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.What is the "usual" way in this context?
The usual way is dpkg-buildpackage -jN but that does not work with later versions.
Quoted 7 lines
Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?If so, I agree. This patch caters to this case.
Quoted 7 lines
But, I think your code should go to debian/rules instead of
scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.Also, the commit description should explain the
benefit of this patch.
I will clarify the dpkg-buildpackage behaviour change and will move the stuff to debian/rules in v2.
Quoted 61 lines · Bastian Germann
Am 15.03.23 um 15:18 schrieb Masahiro Yamada:
Quoted 4 lines
"dpkb-buildpackage -j<N>" sets not only
DEB_BUILD_OPTIONSbut also MAKEFLAGS.Quoted 8 lines
This is clearly explained in "man dpkb-buildpackage".
-j, --jobs[=jobs|auto]
...Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...
I see. The testing/unstable dpkg-buildpackage version changed the -j behaviour. -j only sets DEB_BUILD_OPTIONS=parallel= now.
The man page now says:
--jobs-force[=jobs|auto]
This option (since dpkg 1.21.10) is equivalent to the --jobs option except that it will enable forced parallel mode, by adding the make -j option with the computed number of parallel jobs to the MAKEFLAGS environment variable.Quoted 5 lines
Your statement sounds like
'MAKEFLAGS=-j<N> dpkg-buildpackage'
is the only way to build packages in parallel.
It is in v1.21.10 or later (or using --jobs-force which does the same thing).
Quoted 7 lines
Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
So it is not used to run the actual build, just the top level
make -f debian/rulesinvocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.What is the "usual" way in this context?
The usual way is dpkg-buildpackage -jN but that does not work with later versions.
Quoted 12 lines
Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?If so, I agree. This patch caters to this case.
But, I think your code should go to debian/rules instead of
scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.
Quoted 5 lines
Quoted 2 lines
Also, the commit description should explain the
benefit of this patch.I will clarify the dpkg-buildpackage behaviour change and will move the stuff to debian/rules in v2.
Please do so.
Now I understand the point of this patch.
Indeed, my dpkg-buildpackage is old.
(v1.21.1 on Ubuntu 22.04)
Thanks.
--
Best Regards
Masahiro Yamada
Quoted 82 lines · Masahiro Yamada
Quoted 10 lines · Bastian Germann
Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
Quoted 8 lines · Bastian Germann
Quoted 4 lines
Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.
Quoted 4 lines
Quoted 1 line
"dpkg-buildpackage -b -j16" worked for me.
This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
/usr/bin/make-f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all"dpkb-buildpackage -j<N>" sets not only
DEB_BUILD_OPTIONSbut also MAKEFLAGS.This is clearly explained in "man dpkb-buildpackage".
-j, --jobs[=jobs|auto]
...
Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...Your statement sounds like
'MAKEFLAGS=-j<N> dpkg-buildpackage'
is the only way to build packages in parallel.
Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.
Quoted 4 lines
So it is not used to run the actual build, just the top level
make -f debian/rulesinvocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.What is the "usual" way in this context?
Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?If so, I agree. This patch caters to this case.
But, I think your code should go to debian/rules instead of
scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.Also, the commit description should explain the benefit of this patch.
Quoted 3 lines
Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux
Makefileor dpkg-buildpackage.Try:
$ make deb-pkg -j16and
Quoted 10 lines
$ dpkg-buildpackage -b -j16Both run the package building in parallel.
You see a big difference in build speed w/wo the -j option. (and the build logs are interleaved, since they are emitted by multiple threads)
I have no strong opinion in having a DEB_BUILD_OPTIONS...
As you stated Masahiro I have seen it mostly in debian/rules.
See the example from [1]:
ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
MAKEFLAGS += -j$(NUMJOBS)endif
Normally, I use in my kernel build-script:
$ make deb-pkg -j${NUMJOBS}My €0,02.
-Sedat-
[1] https://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules-options
Quoted 97 lines · Sedat Dilek
Quoted 82 lines · Masahiro Yamada
Quoted 10 lines · Bastian Germann
Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
Quoted 8 lines · Bastian Germann
Quoted 4 lines
Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.
Quoted 4 lines
Quoted 1 line
"dpkg-buildpackage -b -j16" worked for me.
This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
/usr/bin/make-f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all"dpkb-buildpackage -j<N>" sets not only
DEB_BUILD_OPTIONSbut also MAKEFLAGS.This is clearly explained in "man dpkb-buildpackage".
-j, --jobs[=jobs|auto]
...
Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...Your statement sounds like
'MAKEFLAGS=-j<N> dpkg-buildpackage'
is the only way to build packages in parallel.
Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.
Quoted 4 lines
So it is not used to run the actual build, just the top level
make -f debian/rulesinvocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.What is the "usual" way in this context?
Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?If so, I agree. This patch caters to this case.
But, I think your code should go to debian/rules instead of
scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.Also, the commit description should explain the benefit of this patch.
Quoted 3 lines
Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux
Makefileor dpkg-buildpackage.Try:
$ make deb-pkg -j16and
Quoted 10 lines
$ dpkg-buildpackage -b -j16Both run the package building in parallel.
You see a big difference in build speed w/wo the -j option. (and the build logs are interleaved, since they are emitted by multiple threads)
I have no strong opinion in having a
DEB_BUILD_OPTIONS...
Quoted 9 lines
As you stated Masahiro I have seen it mostly in debian/rules.
See the example from [1]:
ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS))) MAKEFLAGS += -j$(NUMJOBS)endif
Yup, this code looks good to me.
Signature · 2 lines
Best Regards Masahiro Yamada