Skip to contentSkip to conversation

kbuild: deb-pkg: Allow parallel build

PATCH In review Bastian Germann 3 participants 9 messages Mar 13, 2023 – Mar 15, 2023 Kbuild

lore
Share & tools
lore
#0 lore raw

cover kbuild: deb-pkg: Allow parallel build

Make use of DEB_BUILD_OPTIONS' parallel option.

Bastian Germann (1):
kbuild: deb-pkg: Allow parallel build

 scripts/package/deb-build-option | 6 ++++++
 1 file changed, 6 insertions(+)
Signature · 1 line
2.39.2
#1 lore raw

PATCH 1/1 kbuild: deb-pkg: Allow parallel build

Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.

Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable that ends up being set by Debian's build tools.

---
 scripts/package/deb-build-option | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/package/deb-build-option +6 -0
index b079b0d121d4..dd170e2b3018 100755
@@ -7,6 +7,12 @@ if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUI
 	echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}-
 fi
 
+for build_opt in $DEB_BUILD_OPTIONS; do
+	if [ "${build_opt#parallel=}" != "$build_opt" ]; then
+		echo -j${build_opt#parallel=}
+	fi
+done
+
 version=$(dpkg-parsechangelog -S Version)
 version_upstream="${version%-*}"
 debian_revision="${version#${version_upstream}}"
Signature · 1 line
2.39.2
#2 lore raw
Quoted 4 lines · Bastian Germann

Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.

"dpkg-buildpackage -b -j16" worked for me.

Quoted 29 lines

Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable that ends up being set by Debian's build tools.

---
 scripts/package/deb-build-option | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/package/deb-build-option +6 -1
index b079b0d121d4..dd170e2b3018 100755
@@ -7,6 +7,12 @@ if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUI
        echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}-
 fi
 
+for build_opt in $DEB_BUILD_OPTIONS; do
+       if [ "${build_opt#parallel=}" != "$build_opt" ]; then
+               echo -j${build_opt#parallel=}
+       fi
+done
+
 version=$(dpkg-parsechangelog -S Version)
 version_upstream="${version%-*}"
 debian_revision="${version#${version_upstream}}"
--

2.39.2

Signature · 2 lines
Best Regards
Masahiro Yamada
#3 lore raw

Am 13.03.23 um 18:59 schrieb Masahiro Yamada:

Quoted 9 lines · Bastian Germann
Quoted 4 lines

Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.

"dpkg-buildpackage -b -j16" worked for me.

This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
/usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all

So it is not used to run the actual build, just the top level make -f debian/rules invocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.

Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile or dpkg-buildpackage.

#4 lore raw
Quoted 10 lines · Bastian Germann

Am 13.03.23 um 18:59 schrieb Masahiro Yamada:

Quoted 8 lines · Bastian Germann
Quoted 4 lines

Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.

Quoted 4 lines
Quoted 1 line

"dpkg-buildpackage -b -j16" worked for me.

This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
/usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all

"dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS but also MAKEFLAGS.

This is clearly explained in "man dpkb-buildpackage".

-j, --jobs[=jobs|auto]
...
Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...

Your statement sounds like

'MAKEFLAGS=-j<N> dpkg-buildpackage'

is the only way to build packages in parallel.

Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.

Quoted 4 lines

So it is not used to run the actual build, just the top level make -f debian/rules invocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.

What is the "usual" way in this context?

Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?

If so, I agree. This patch caters to this case.

But, I think your code should go to debian/rules instead of scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.

Also, the commit description should explain the benefit of this patch.

Quoted 3 lines

Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile or dpkg-buildpackage.

Try:

 $ make deb-pkg -j16

and

 $ dpkg-buildpackage -b -j16

Both run the package building in parallel.

You see a big difference in build speed w/wo the -j option. (and the build logs are interleaved, since they are emitted by multiple threads)

--
Best Regards
Masahiro Yamada

#5 lore raw

Am 15.03.23 um 15:18 schrieb Masahiro Yamada:

Quoted 4 lines

"dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS but also MAKEFLAGS.

Quoted 8 lines

This is clearly explained in "man dpkb-buildpackage".

-j, --jobs[=jobs|auto]

    ...

Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...

I see. The testing/unstable dpkg-buildpackage version changed the -j behaviour. -j only sets DEB_BUILD_OPTIONS=parallel= now.

The man page now says:

--jobs-force[=jobs|auto]
This option (since dpkg 1.21.10) is equivalent to the --jobs option except that it will enable forced parallel mode, by adding the make -j option with the computed number of parallel jobs to the MAKEFLAGS environment variable.

Quoted 5 lines

Your statement sounds like

'MAKEFLAGS=-j<N> dpkg-buildpackage'

is the only way to build packages in parallel.

It is in v1.21.10 or later (or using --jobs-force which does the same thing).

Quoted 7 lines

Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

So it is not used to run the actual build, just the top level make -f debian/rules invocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.

What is the "usual" way in this context?

The usual way is dpkg-buildpackage -jN but that does not work with later versions.

Quoted 7 lines

Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?

If so, I agree. This patch caters to this case.

Quoted 7 lines

But, I think your code should go to debian/rules instead of scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.

Also, the commit description should explain the
benefit of this patch.

I will clarify the dpkg-buildpackage behaviour change and will move the stuff to debian/rules in v2.

#6 lore raw
Quoted 61 lines · Bastian Germann

Am 15.03.23 um 15:18 schrieb Masahiro Yamada:

Quoted 4 lines

"dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS but also MAKEFLAGS.

Quoted 8 lines

This is clearly explained in "man dpkb-buildpackage".

-j, --jobs[=jobs|auto]

    ...

Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...

I see. The testing/unstable dpkg-buildpackage version changed the -j behaviour. -j only sets DEB_BUILD_OPTIONS=parallel= now.

The man page now says:

--jobs-force[=jobs|auto]
This option (since dpkg 1.21.10) is equivalent to the --jobs option except that it will enable forced parallel mode, by adding the make -j option with the computed number of parallel jobs to the MAKEFLAGS environment variable.

Quoted 5 lines

Your statement sounds like

'MAKEFLAGS=-j<N> dpkg-buildpackage'

is the only way to build packages in parallel.

It is in v1.21.10 or later (or using --jobs-force which does the same thing).

Quoted 7 lines

Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

So it is not used to run the actual build, just the top level make -f debian/rules invocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.

What is the "usual" way in this context?

The usual way is dpkg-buildpackage -jN but that does not work with later versions.

Quoted 12 lines

Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?

If so, I agree. This patch caters to this case.

But, I think your code should go to debian/rules instead of scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.

Quoted 5 lines
Quoted 2 lines

Also, the commit description should explain the
benefit of this patch.

I will clarify the dpkg-buildpackage behaviour change and will move the stuff to debian/rules in v2.

Please do so.

Now I understand the point of this patch.
Indeed, my dpkg-buildpackage is old.
(v1.21.1 on Ubuntu 22.04)

Thanks.

--
Best Regards
Masahiro Yamada

#7 lore raw
In reply to Masahiro Yamada
Quoted 82 lines · Masahiro Yamada
Quoted 10 lines · Bastian Germann

Am 13.03.23 um 18:59 schrieb Masahiro Yamada:

Quoted 8 lines · Bastian Germann
Quoted 4 lines

Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.

Quoted 4 lines
Quoted 1 line

"dpkg-buildpackage -b -j16" worked for me.

This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
/usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all

"dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS but also MAKEFLAGS.

This is clearly explained in "man dpkb-buildpackage".

-j, --jobs[=jobs|auto]
...
Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...

Your statement sounds like

'MAKEFLAGS=-j<N> dpkg-buildpackage'

is the only way to build packages in parallel.

Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.

Quoted 4 lines

So it is not used to run the actual build, just the top level make -f debian/rules invocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.

What is the "usual" way in this context?

Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?

If so, I agree. This patch caters to this case.

But, I think your code should go to debian/rules instead of scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.

Also, the commit description should explain the benefit of this patch.

Quoted 3 lines

Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile or dpkg-buildpackage.

Try:

 $ make deb-pkg -j16

and

Quoted 10 lines
 $ dpkg-buildpackage -b -j16

Both run the package building in parallel.

You see a big difference in build speed w/wo the -j option. (and the build logs are interleaved, since they are emitted by multiple threads)

I have no strong opinion in having a DEB_BUILD_OPTIONS...

As you stated Masahiro I have seen it mostly in debian/rules.

See the example from [1]:

ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))

    NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
    MAKEFLAGS += -j$(NUMJOBS)

endif

Normally, I use in my kernel build-script:

$ make deb-pkg -j${NUMJOBS}

My €0,02.

-Sedat-

[1] https://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules-options

#8 lore raw
Quoted 97 lines · Sedat Dilek
Quoted 82 lines · Masahiro Yamada
Quoted 10 lines · Bastian Germann

Am 13.03.23 um 18:59 schrieb Masahiro Yamada:

Quoted 8 lines · Bastian Germann
Quoted 4 lines

Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1.

Quoted 4 lines
Quoted 1 line

"dpkg-buildpackage -b -j16" worked for me.

This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
/usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all

"dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS but also MAKEFLAGS.

This is clearly explained in "man dpkb-buildpackage".

-j, --jobs[=jobs|auto]
...
Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ...

Your statement sounds like

'MAKEFLAGS=-j<N> dpkg-buildpackage'

is the only way to build packages in parallel.

Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel.

Quoted 4 lines

So it is not used to run the actual build, just the top level make -f debian/rules invocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing.

What is the "usual" way in this context?

Do you mean
'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building?

If so, I agree. This patch caters to this case.

But, I think your code should go to debian/rules instead of scripts/package/deb-build-option.
Kbuild's "make clean" works much faster with the parallel option.

Also, the commit description should explain the benefit of this patch.

Quoted 3 lines

Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile or dpkg-buildpackage.

Try:

 $ make deb-pkg -j16

and

Quoted 10 lines
 $ dpkg-buildpackage -b -j16

Both run the package building in parallel.

You see a big difference in build speed w/wo the -j option. (and the build logs are interleaved, since they are emitted by multiple threads)

I have no strong opinion in having a DEB_BUILD_OPTIONS...

Quoted 9 lines

As you stated Masahiro I have seen it mostly in debian/rules.

See the example from [1]:

ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))

    NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
    MAKEFLAGS += -j$(NUMJOBS)

endif

Yup, this code looks good to me.

Signature · 2 lines
Best Regards
Masahiro Yamada