Skip to contentSkip to conversation

kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

PATCH In review Bastian Germann 2 participants 5 messages Mar 15, 2023 – Apr 10, 2023 Kbuild

lore
Share & tools
lore
#0 lore raw

v3 cover kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

Make use of DEB_BUILD_OPTIONS' parallel option because that is the only parallel build option that is set by default in current dpkg-buildpackage versions.

v2:
* Clarify that this is for current dpkg-buildpackage versions
* Evaluate DEB_BUILD_OPTIONS in debian/rules.
v3:
* Use the Policy snippet which sets MAKEFLAGS and uses patsubst over subst
* Mention the dpkg commit that changed the -j behaviour.

Bastian Germann (1):
kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

 scripts/package/mkdebian | 5 +++++
 1 file changed, 5 insertions(+)
Signature · 1 line
2.39.2
#1 lore raw

PATCH v3 1/1 kbuild: deb-pkg: Build parallely with current dpkg-buildpackage

With dpkg-buildpackage v1.21.10 or later, the only way to build the deb-pkg generated package parallely is adding -j<N> to the MAKEFLAGS environment variable or using the --jobs-force option, see dpkg commit 1d0ea9b2ba3f ("dpkg-buildpackage: Change -j, --jobs semantics to non-force mode"). The package ignores the usual parallel build option that is described in Debian Policy.

Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable that ends up being set by dpkg-buildpackage -j<N>. The snippet is copied from Debian Policy.

---
 scripts/package/mkdebian | 5 +++++
 1 file changed, 5 insertions(+)
scripts/package/mkdebian +5 -0
index f74380036bb5..ed5c2b65798b 100755
@@ -238,6 +238,11 @@ fi
 cat <<EOF > debian/rules
 #!$(command -v $MAKE) -f
 
+ifneq (,\$(filter parallel=%,\$(DEB_BUILD_OPTIONS)))
+	NUMJOBS = \$(patsubst parallel=%,%,\$(filter parallel=%,\$(DEB_BUILD_OPTIONS)))
+	MAKEFLAGS += -j\$(NUMJOBS)
+endif
+
 srctree ?= .
 
 build-indep:
Signature · 1 line
2.39.2
#2 lore raw
Quoted 8 lines · Bastian Germann

With dpkg-buildpackage v1.21.10 or later, the only way to build the deb-pkg generated package parallely is adding -j<N> to the MAKEFLAGS environment variable or using the --jobs-force option, see dpkg commit 1d0ea9b2ba3f ("dpkg-buildpackage: Change -j, --jobs semantics to non-force mode"). The package ignores the usual parallel build option that is described in Debian Policy.

Quoted 7 lines

Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable that ends up being set by dpkg-buildpackage -j<N>. The snippet is copied from Debian Policy.

---

I was about to pick this up, then I stopped it.

If I apply this patch,

"make -j2 bindeb-pkg" shows a new warning.

make[3]: warning: -j16 forced in makefile: resetting jobserver mode.

Please note nproc returns 16 on my machine.

"make bindeb-pkg" also runs in 16 threads because $(DEB_BUILD_OPTIONS) always contains "parallel=16" even if -j<N> option is given at all.

In other words, this patch will take away user's freedom to choose the number of threads.

Quoted 22 lines
 scripts/package/mkdebian | 5 +++++
 1 file changed, 5 insertions(+)
scripts/package/mkdebian +5 -1
index f74380036bb5..ed5c2b65798b 100755
@@ -238,6 +238,11 @@ fi
 cat <<EOF > debian/rules
 #!$(command -v $MAKE) -f
 
+ifneq (,\$(filter parallel=%,\$(DEB_BUILD_OPTIONS)))
+       NUMJOBS = \$(patsubst parallel=%,%,\$(filter parallel=%,\$(DEB_BUILD_OPTIONS)))
+       MAKEFLAGS += -j\$(NUMJOBS)
+endif
+
 srctree ?= .
 
 build-indep:
--

2.39.2

Signature · 2 lines
Best Regards
Masahiro Yamada
#3 lore raw

Am 21.03.23 um 11:18 schrieb Masahiro Yamada:

Quoted 6 lines

"make bindeb-pkg" also runs in 16 threads because $(DEB_BUILD_OPTIONS) always contains "parallel=16" even if -j<N> option is given at all.

In other words, this patch will take away user's freedom to choose the number of threads.

Okay. To solve this, I would forward make's -j option to the dpkg-buildpackage calls, which is now called with the defaults.

I think you should see the warning with older dpkg-buildpackage versions without the patch applied as well because their default is dpkg-buildpackage -jnproc and is in effect currently.

#4 lore raw
Quoted 11 lines · Bastian Germann

Am 21.03.23 um 11:18 schrieb Masahiro Yamada:

Quoted 6 lines

"make bindeb-pkg" also runs in 16 threads because $(DEB_BUILD_OPTIONS) always contains "parallel=16" even if -j<N> option is given at all.

In other words, this patch will take away user's freedom to choose the number of threads.

Okay. To solve this, I would forward make's -j option to the dpkg-buildpackage calls, which is now called with the defaults.

Yes. Agreed.

Quoted 3 lines

I think you should see the warning with older dpkg-buildpackage versions without the patch applied as well because their default is dpkg-buildpackage -jnproc and is in effect currently.

Signature · 2 lines
Best Regards
Masahiro Yamada