Skip to contentSkip to conversation

test-manual: add initial reproducible builds documentation

PATCH Superseded Michael Opdenacker 3 participants 11 messages Jun 8, 2021 – Jun 9, 2021 1 message not in the archive

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

PATCH v2 1/1 test-manual: add initial reproducible builds documentation

From: Richard Purdie <[email protected]>

---
 documentation/test-manual/index.rst           |  1 +
 .../test-manual/reproducible-builds.rst       | 55 +++++++++++++++++++
 2 files changed, 56 insertions(+)
 create mode 100644 documentation/test-manual/reproducible-builds.rst
documentation/test-manual/index.rst +1 -0
index e2198c4c39..d122b5e270 100644
@@ -13,6 +13,7 @@ Yocto Project Test Environment Manual
    intro
    test-process
    understand-autobuilder
+   reproducible-builds
    history
 
 .. include:: /boilerplate.rst
documentation/test-manual/reproducible-builds.rst new file +55 -0

55 lines not on this page · raw message

Signature · 1 line
2.25.1
#2 lore raw

Hello,

Quoted 73 lines · Michael Opdenacker

From: Richard Purdie <[email protected]>

---
 documentation/test-manual/index.rst           |  1 +
 .../test-manual/reproducible-builds.rst       | 55 +++++++++++++++++++
 2 files changed, 56 insertions(+)
 create mode 100644 documentation/test-manual/reproducible-builds.rst
documentation/test-manual/index.rst +1 -0
index e2198c4c39..d122b5e270 100644
@@ -13,6 +13,7 @@ Yocto Project Test Environment Manual
    intro
    test-process
    understand-autobuilder
+   reproducible-builds
    history
 
 .. include:: /boilerplate.rst
documentation/test-manual/reproducible-builds.rst new file +45 -0
new file mode 100644
index 0000000000..c2e13d410f
@@ -0,0 +1,55 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+*******************
+Reproducible Builds
+*******************
+
+================
+How we define it
+================
+
+The Yocto Project defines reproducibility as where a given input build configuration will give the same binary output regardless of when it is built (now or in 5 years time), regardless of the path on the filesystem the build is run in, and regardless of the distro and tools on the underlying host system the build is running on.
+
+==============
+Why it matters
+==============
+
+The project aligns with the `Reproducible Builds project <https://reproducible-builds.org/>`_, which shares information about why reproducibility matters. The primary focus of the project is the ability to detect security issues being introduced. However, from a Yocto Project perspective, it is also hugely important that our builds are deterministic. When you build a given input set of metadata, we expect you to get consistent output. This has always been a key focus but is now true down to the binary level including timestamps.
+
+For example, at some point in the future life of a product, you find that you need to rebuild to add a security fix. If this happens, only the components that have been modified should change at the binary level. This would lead to much easier and clearer bounds on where validation is needed.
+
+This also gives an additional benefit to the project builds themselves, our hash equivalence for :ref:`Shared State <overview-manual/concepts:Shared State>` object reuse works much more effectively when the binary output remains the same.
+
+===================
+How we implement it
+===================
+
+There are many different aspects to build reproducibility, but some particular things we do within the build system to ensure reproducibility include:
+
+* Adding mappings to the compiler options to ensure debug filepaths are mapped to consistent target compatible paths. This is done through the ``DEBUG_PREFIX_MAP`` variable which sets the ``-fmacro-prefix-map`` and ``-fdebug-prefix-map`` compiler options correctly to map to target paths.
+* Being explicit about recipe dependencies and their configuration (no floating configure options or host dependencies creeping in). In particular this means making sure :term:`PACKAGECONFIG` coverage covers configure options which may otherwise try and auto-detect host dependencies.
+* Using recipe specific sysroots to isolate recipes so they only see their dependencies. These are visible as ``recipe-sysroot`` and ``recipe-sysroot-native`` directories within the :term:`WORKDIR` of a given recipe and are populated only with the dependencies a recipe has.
+* Build images from a reduced package set: only packages from recipes the image depends upon.
+* Filtering the tools available from the host's PATH to only a specific set of tools, set using the :term:`HOSTTOOLS` variable.
+
+=========================================
+Can we prove the project is reproducible?
+=========================================
+
+Yes, we can prove it and we now regularly test this on the Autobuilder. At the time of writing (Hardknott 3.3), OE-Core is 100% reproducible for all its recipes (i.e. world builds) apart from the Go language and Ruby documentation packages. Unfortunately, the current implementation of the Go language has fundamental reproducibility problems as it always depends upon the paths it is build in.
+
+To run our automated selftest, as we use in our CI on the Autobuilder, you can run::
+
+   oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
+
+This defaults to including a world build so, if other layers are added, it would also run the tests for recipes in the additional layers. The first build will be run using Shared State if available, the second build explicitly disables Shared State and builds on the specific host the build is running on. This means we can test reproducibility builds between different host distributions over time on the Autobuilder.

documentation/test-manual/reproducible-builds.rst line 55

Quoted 10 lines
documentation/test-manual/reproducible-builds.rst new file +10 -0
+
+If ``OEQA_DEBUGGING_SAVED_OUTPUT`` is set, any differing packages will be saved here. The test is also able to run the ``diffoscope`` command on the output to generate HTML files showing the differences between the packages, to aid debugging. On the Autobuilder, these appear under https://autobuilder.yocto.io/pub/repro-fail/ in the form ``oe-reproducible + <date> + <random ID>``, e.g. ``oe-reproducible-20200202-1lm8o1th``.
+
+The project's current reproducibility status can be seen at https://www.yoctoproject.org/reproducible-build-results/
+
+===============================
+Can I test my layer or recipes?
+===============================
+
+Yes, see the self test in :ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>` at `meta/lib/oeqa/selftest/cases/reproducible.py <https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_ which can be subclassed for specific use cases.

So, this was my pass (fixing typos and style, adding references, adding more punctuation, breaking long sentences), so that other people can offer their review now.

Thanks in advance,

Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw
Quoted 3 lines · Michael Opdenacker

So, this was my pass (fixing typos and style, adding references, adding more punctuation, breaking long sentences), so that other people can offer their review now.

Looks good, thanks!

Cheers,

Richard

#4 lore raw

Hi Richard,

Quoted 5 lines · Richard Purdie
Quoted 3 lines · Michael Opdenacker

So, this was my pass (fixing typos and style, adding references, adding more punctuation, breaking long sentences), so that other people can offer their review now.

Looks good, thanks!

Thanks for confirming!
For my own convenience (that will be easier to process your "YP compatible" changes which assume this one is accepted), I merged these changes into master-next.

If any of you have suggestions to improve or correct the text, you can send me a patch against master-next, or just comments to the V2 patch which I'll integrate manually.

Cheers,

Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw
In reply to Michael Opdenacker

Hi Michael, Richard,

Quoted 36 lines · Michael Opdenacker

From: Richard Purdie <[email protected]>

---
 documentation/test-manual/index.rst           |  1 +
 .../test-manual/reproducible-builds.rst       | 55 +++++++++++++++++++
 2 files changed, 56 insertions(+)
 create mode 100644 documentation/test-manual/reproducible-builds.rst
documentation/test-manual/index.rst +1 -0
index e2198c4c39..d122b5e270 100644
@@ -13,6 +13,7 @@ Yocto Project Test Environment Manual
    intro
    test-process
    understand-autobuilder
+   reproducible-builds
    history
 
 .. include:: /boilerplate.rst
documentation/test-manual/reproducible-builds.rst new file +8 -0
new file mode 100644
index 0000000000..c2e13d410f
@@ -0,0 +1,55 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+*******************
+Reproducible Builds
+*******************
+
+================
+How we define it

documentation/test-manual/reproducible-builds.rst line 18

Quoted 10 lines
documentation/test-manual/reproducible-builds.rst new file +10 -0
+================
+
+The Yocto Project defines reproducibility as where a given input build configuration will give the same binary output regardless of when it is built (now or in 5 years time), regardless of the path on the filesystem the build is run in, and regardless of the distro and tools on the underlying host system the build is running on.
+
+==============
+Why it matters
+==============
+
+The project aligns with the `Reproducible Builds project <https://reproducible-builds.org/>`_, which shares information about why reproducibility matters. The primary focus of the project is the ability to detect security issues being introduced. However, from a Yocto Project perspective, it is also hugely important that our builds are deterministic. When you build a given input set of metadata, we expect you to get consistent output. This has always been a key focus but is now true down to the binary level including timestamps.
+

"now" is too abstract? Can we say at least since when (e.g. which release)?

documentation/test-manual/reproducible-builds.rst line 29

Quoted 11 lines
documentation/test-manual/reproducible-builds.rst new file +11 -0
+For example, at some point in the future life of a product, you find that you need to rebuild to add a security fix. If this happens, only the components that have been modified should change at the binary level. This would lead to much easier and clearer bounds on where validation is needed.
+
+This also gives an additional benefit to the project builds themselves, our hash equivalence for :ref:`Shared State <overview-manual/concepts:Shared State>` object reuse works much more effectively when the binary output remains the same.
+
+===================
+How we implement it
+===================
+
+There are many different aspects to build reproducibility, but some particular things we do within the build system to ensure reproducibility include:
+
+* Adding mappings to the compiler options to ensure debug filepaths are mapped to consistent target compatible paths. This is done through the ``DEBUG_PREFIX_MAP`` variable which sets the ``-fmacro-prefix-map`` and ``-fdebug-prefix-map`` compiler options correctly to map to target paths.

Would documenting DEBUG_PREFIX_MAP make sense? Is it supposed to be
changed by users in some cases?
for reference, where it's being defined: https://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/bitbake.conf#n612

documentation/test-manual/reproducible-builds.rst line 31

Quoted 2 lines
documentation/test-manual/reproducible-builds.rst new file +2 -0
+* Being explicit about recipe dependencies and their configuration (no floating configure options or host dependencies creeping in). In particular this means making sure :term:`PACKAGECONFIG` coverage covers configure options which may otherwise try and auto-detect host dependencies.
+* Using recipe specific sysroots to isolate recipes so they only see their dependencies. These are visible as ``recipe-sysroot`` and ``recipe-sysroot-native`` directories within the :term:`WORKDIR` of a given recipe and are populated only with the dependencies a recipe has.

Can't we just reference RECIPE_SYSROOT and RECIPE_SYSROOT_NATIVE here? Aaaand I just discovered these aren't in the variable glossary but I find them very important?

Can be done later on since they do not exist yet.

documentation/test-manual/reproducible-builds.rst line 34

Quoted 3 lines
documentation/test-manual/reproducible-builds.rst new file +3 -0
+* Build images from a reduced package set: only packages from recipes the image depends upon.
+* Filtering the tools available from the host's PATH to only a specific set of tools, set using the :term:`HOSTTOOLS` variable.
+

``PATH``?

Also, I think we use dashes more than asterisks for lists?

I think there's also an additional point: we check the sources we download. Therefore, if a commit disappears (force-pushed/removed branch/tag), the tarball disappears from upstream, or its checksum differs from what we expect, we don't build.

Might be worth mentioning that we only download sources from the internet when we're missing them locally. So hosting a mirror for sources or making sure they are available locally is a need if you want to guarantee full reproducible builds.

And this is especially important to mention since some projects are removing their master branches for example in favor of main, breaking older builds. Or some projects change hosting services and do not redirect, or just disappear from the Internet.

documentation/test-manual/reproducible-builds.rst line 40

Quoted 6 lines
documentation/test-manual/reproducible-builds.rst new file +6 -0
+=========================================
+Can we prove the project is reproducible?
+=========================================
+
+Yes, we can prove it and we now regularly test this on the Autobuilder. At the time of writing (Hardknott 3.3), OE-Core is 100% reproducible for all its recipes (i.e. world builds) apart from the Go language and Ruby documentation packages. Unfortunately, the current implementation of the Go language has fundamental reproducibility problems as it always depends upon the paths it is build in.
+

s/now//

Please give a link to reproducibility test archives/categories on the Autobuilder if there's any available?

s/build/built/

I would add a reference to OE-Core term :term:OpenEmbedded-Core (OE-Core) which explains what exactly it is. It is an important reference since we really mean to say that openembedded-core layer recipes (which is meta layer in poky) is 100% reproducible, nothing else. Basically, OE-Core + Bitbake is guaranteed to be reproducible, but the moment you add another layer, no guarantee (e.g. because of additional configuration files, bbappends, overridden classes, etc...)

documentation/test-manual/reproducible-builds.rst line 46

Quoted 6 lines
documentation/test-manual/reproducible-builds.rst new file +6 -0
+To run our automated selftest, as we use in our CI on the Autobuilder, you can run::
+
+   oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
+
+This defaults to including a world build so, if other layers are added, it would also run the tests for recipes in the additional layers. The first build will be run using Shared State if available, the second build explicitly disables Shared State and builds on the specific host the build is running on. This means we can test reproducibility builds between different host distributions over time on the Autobuilder.
+

s/Shared State/:ref:Shared State <overview-manual/concepts:Shared State>/

since we already reference it earlier in this page.

documentation/test-manual/reproducible-builds.rst line 48

Quoted 2 lines
documentation/test-manual/reproducible-builds.rst new file +2 -0
+If ``OEQA_DEBUGGING_SAVED_OUTPUT`` is set, any differing packages will be saved here. The test is also able to run the ``diffoscope`` command on the output to generate HTML files showing the differences between the packages, to aid debugging. On the Autobuilder, these appear under https://autobuilder.yocto.io/pub/repro-fail/ in the form ``oe-reproducible + <date> + <random ID>``, e.g. ``oe-reproducible-20200202-1lm8o1th``.
+

Would it make sense to document OEQA_DEBUGGING_SAVED_OUTPUT too?

documentation/test-manual/reproducible-builds.rst line 55

Quoted 7 lines
documentation/test-manual/reproducible-builds.rst new file +7 -0
+The project's current reproducibility status can be seen at https://www.yoctoproject.org/reproducible-build-results/
+
+===============================
+Can I test my layer or recipes?
+===============================
+
+Yes, see the self test in :ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>` at `meta/lib/oeqa/selftest/cases/reproducible.py <https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_ which can be subclassed for specific use cases.

Please provide actual commands to run so the user does not have to guess :)

Cheers,
Quentin

#6 lore raw
Quoted 5 lines · Quentin Schulz

Hi Michael, Richard,

Quoted 2 lines · Michael Opdenacker

From: Richard Purdie <[email protected]>

Quoted 10 lines

documentation/test-manual/reproducible-builds.rst line 18

Quoted 8 lines
documentation/test-manual/reproducible-builds.rst new file +8 -0
+The Yocto Project defines reproducibility as where a given input build configuration will give the same binary output regardless of when it is built (now or in 5 years time), regardless of the path on the filesystem the build is run in, and regardless of the distro and tools on the underlying host system the build is running on.
+
+==============
+Why it matters
+==============
+
+The project aligns with the `Reproducible Builds project <https://reproducible-builds.org/>`_, which shares information about why reproducibility matters. The primary focus of the project is the ability to detect security issues being introduced. However, from a Yocto Project perspective, it is also hugely important that our builds are deterministic. When you build a given input set of metadata, we expect you to get consistent output. This has always been a key focus but is now true down to the binary level including timestamps.
+

"now" is too abstract? Can we say at least since when (e.g. which release)?

As of dunfell we enabled reprodubile builds in poky by default.

http://git.yoctoproject.org/cgit.cgi/poky/commit/meta-poky/conf/distro/poky.conf?id=25a17f0346ab05b3212bb280e4a8b622bcf99bce

Quoted 16 lines

documentation/test-manual/reproducible-builds.rst line 29

Quoted 12 lines
documentation/test-manual/reproducible-builds.rst new file +11 -0
+For example, at some point in the future life of a product, you find that you need to rebuild to add a security fix. If this happens, only the components that have been modified should change at the binary level. This would lead to much easier and clearer bounds on where validation is needed.
+
+This also gives an additional benefit to the project builds themselves, our hash equivalence for :ref:`Shared State <overview-manual/concepts:Shared State>` object reuse works much more effectively when the binary output remains the same.
+
+===================
+How we implement it
+===================
+
+There are many different aspects to build reproducibility, but some particular things we do within the build system to ensure reproducibility include:
+
+* Adding mappings to the compiler options to ensure debug filepaths are mapped to consistent target compatible paths. This is done through the ``DEBUG_PREFIX_MAP`` variable which sets the ``-fmacro-prefix-map`` and ``-fdebug-prefix-map`` compiler options correctly to map to target paths.

Would documenting DEBUG_PREFIX_MAP make sense? Is it supposed to be
changed by users in some cases?
for reference, where it's being defined: https://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/bitbake.conf#n612

Users shouldn't need to mess with it but it should be documented.

Quoted 7 lines

documentation/test-manual/reproducible-builds.rst line 31

Quoted 3 lines
documentation/test-manual/reproducible-builds.rst new file +2 -0
+* Being explicit about recipe dependencies and their configuration (no floating configure options or host dependencies creeping in). In particular this means making sure :term:`PACKAGECONFIG` coverage covers configure options which may otherwise try and auto-detect host dependencies.
+* Using recipe specific sysroots to isolate recipes so they only see their dependencies. These are visible as ``recipe-sysroot`` and ``recipe-sysroot-native`` directories within the :term:`WORKDIR` of a given recipe and are populated only with the dependencies a recipe has.

Can't we just reference RECIPE_SYSROOT and RECIPE_SYSROOT_NATIVE here? Aaaand I just discovered these aren't in the variable glossary but I find them very important?

I suspect unfortunately our recipe specific sysroots are basically not documented at all :(. Again, probably needs fixing ultimately.

Quoted 21 lines

Can be done later on since they do not exist yet.

documentation/test-manual/reproducible-builds.rst line 34

Quoted 3 lines
documentation/test-manual/reproducible-builds.rst new file +3 -0
+* Build images from a reduced package set: only packages from recipes the image depends upon.
+* Filtering the tools available from the host's PATH to only a specific set of tools, set using the :term:`HOSTTOOLS` variable.
+

``PATH``?

Also, I think we use dashes more than asterisks for lists?

I think there's also an additional point: we check the sources we download. Therefore, if a commit disappears (force-pushed/removed branch/tag), the tarball disappears from upstream, or its checksum differs from what we expect, we don't build.

Might be worth mentioning that we only download sources from the internet when we're missing them locally. So hosting a mirror for sources or making sure they are available locally is a need if you want to guarantee full reproducible builds.

Quoted 4 lines

And this is especially important to mention since some projects are removing their master branches for example in favor of main, breaking older builds. Or some projects change hosting services and do not redirect, or just disappear from the Internet.

Yes, we should put something in about input validation - all inputs to the build have some kind of validation, by checksum or source control hash/revision.

Quoted 10 lines

documentation/test-manual/reproducible-builds.rst line 40

Quoted 7 lines

+=========================================

documentation/test-manual/reproducible-builds.rst new file +5 -0
+Can we prove the project is reproducible?
+=========================================
+
+Yes, we can prove it and we now regularly test this on the Autobuilder. At the time of writing (Hardknott 3.3), OE-Core is 100% reproducible for all its recipes (i.e. world builds) apart from the Go language and Ruby documentation packages. Unfortunately, the current implementation of the Go language has fundamental reproducibility problems as it always depends upon the paths it is build in.
+

s/now//

Quoted 2 lines

Please give a link to reproducibility test archives/categories on the Autobuilder if there's any available?

Isn't that below with the current status? The autobuilder would be something like:

https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible

but also

https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-centos
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-debian
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-fedora
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-ubuntu

Quoted 27 lines

s/build/built/

I would add a reference to OE-Core term :term:OpenEmbedded-Core (OE-Core) which explains what exactly it is. It is an important reference since we really mean to say that openembedded-core layer recipes (which is meta layer in poky) is 100% reproducible, nothing else. Basically, OE-Core + Bitbake is guaranteed to be reproducible, but the moment you add another layer, no guarantee (e.g. because of additional configuration files, bbappends, overridden classes, etc...)

documentation/test-manual/reproducible-builds.rst line 46

Quoted 6 lines
documentation/test-manual/reproducible-builds.rst new file +6 -0
+To run our automated selftest, as we use in our CI on the Autobuilder, you can run::
+
+   oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
+
+This defaults to including a world build so, if other layers are added, it would also run the tests for recipes in the additional layers. The first build will be run using Shared State if available, the second build explicitly disables Shared State and builds on the specific host the build is running on. This means we can test reproducibility builds between different host distributions over time on the Autobuilder.
+

s/Shared State/:ref:Shared State <overview-manual/concepts:Shared State>/

since we already reference it earlier in this page.

Review

Quoted 2 lines
(quoted diff) +2 -0
+If ``OEQA_DEBUGGING_SAVED_OUTPUT`` is set, any differing packages will be saved here. The test is also able to run the ``diffoscope`` command on the output to generate HTML files showing the differences between the packages, to aid debugging. On the Autobuilder, these appear under https://autobuilder.yocto.io/pub/repro-fail/ in the form ``oe-reproducible + <date> + <random ID>``, e.g. ``oe-reproducible-20200202-1lm8o1th``.
+

Would it make sense to document OEQA_DEBUGGING_SAVED_OUTPUT too?

Quoted 9 lines

documentation/test-manual/reproducible-builds.rst line 55

Quoted 7 lines
documentation/test-manual/reproducible-builds.rst new file +7 -0
+The project's current reproducibility status can be seen at https://www.yoctoproject.org/reproducible-build-results/
+
+===============================
+Can I test my layer or recipes?
+===============================
+
+Yes, see the self test in :ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>` at `meta/lib/oeqa/selftest/cases/reproducible.py <https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_ which can be subclassed for specific use cases.

Please provide actual commands to run so the user does not have to guess :)

It is stated in the section above!

(oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds)

Cheers,

Richard

#7 lore raw
Quoted 97 lines · Richard Purdie
Quoted 5 lines · Quentin Schulz

Hi Michael, Richard,

Quoted 2 lines · Michael Opdenacker

From: Richard Purdie <[email protected]>

Quoted 10 lines

documentation/test-manual/reproducible-builds.rst line 18

Quoted 8 lines
documentation/test-manual/reproducible-builds.rst new file +8 -0
+The Yocto Project defines reproducibility as where a given input build configuration will give the same binary output regardless of when it is built (now or in 5 years time), regardless of the path on the filesystem the build is run in, and regardless of the distro and tools on the underlying host system the build is running on.
+
+==============
+Why it matters
+==============
+
+The project aligns with the `Reproducible Builds project <https://reproducible-builds.org/>`_, which shares information about why reproducibility matters. The primary focus of the project is the ability to detect security issues being introduced. However, from a Yocto Project perspective, it is also hugely important that our builds are deterministic. When you build a given input set of metadata, we expect you to get consistent output. This has always been a key focus but is now true down to the binary level including timestamps.
+

"now" is too abstract? Can we say at least since when (e.g. which release)?

As of dunfell we enabled reprodubile builds in poky by default.

http://git.yoctoproject.org/cgit.cgi/poky/commit/meta-poky/conf/distro/poky.conf?id=25a17f0346ab05b3212bb280e4a8b622bcf99bce

Quoted 16 lines

documentation/test-manual/reproducible-builds.rst line 29

Quoted 12 lines
documentation/test-manual/reproducible-builds.rst new file +11 -0
+For example, at some point in the future life of a product, you find that you need to rebuild to add a security fix. If this happens, only the components that have been modified should change at the binary level. This would lead to much easier and clearer bounds on where validation is needed.
+
+This also gives an additional benefit to the project builds themselves, our hash equivalence for :ref:`Shared State <overview-manual/concepts:Shared State>` object reuse works much more effectively when the binary output remains the same.
+
+===================
+How we implement it
+===================
+
+There are many different aspects to build reproducibility, but some particular things we do within the build system to ensure reproducibility include:
+
+* Adding mappings to the compiler options to ensure debug filepaths are mapped to consistent target compatible paths. This is done through the ``DEBUG_PREFIX_MAP`` variable which sets the ``-fmacro-prefix-map`` and ``-fdebug-prefix-map`` compiler options correctly to map to target paths.

Would documenting DEBUG_PREFIX_MAP make sense? Is it supposed to be
changed by users in some cases?
for reference, where it's being defined: https://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/bitbake.conf#n612

Users shouldn't need to mess with it but it should be documented.

Quoted 7 lines

documentation/test-manual/reproducible-builds.rst line 31

Quoted 3 lines
documentation/test-manual/reproducible-builds.rst new file +2 -0
+* Being explicit about recipe dependencies and their configuration (no floating configure options or host dependencies creeping in). In particular this means making sure :term:`PACKAGECONFIG` coverage covers configure options which may otherwise try and auto-detect host dependencies.
+* Using recipe specific sysroots to isolate recipes so they only see their dependencies. These are visible as ``recipe-sysroot`` and ``recipe-sysroot-native`` directories within the :term:`WORKDIR` of a given recipe and are populated only with the dependencies a recipe has.

Can't we just reference RECIPE_SYSROOT and RECIPE_SYSROOT_NATIVE here? Aaaand I just discovered these aren't in the variable glossary but I find them very important?

I suspect unfortunately our recipe specific sysroots are basically not documented at all :(. Again, probably needs fixing ultimately.

Quoted 21 lines

Can be done later on since they do not exist yet.

documentation/test-manual/reproducible-builds.rst line 34

Quoted 3 lines
documentation/test-manual/reproducible-builds.rst new file +3 -0
+* Build images from a reduced package set: only packages from recipes the image depends upon.
+* Filtering the tools available from the host's PATH to only a specific set of tools, set using the :term:`HOSTTOOLS` variable.
+

``PATH``?

Also, I think we use dashes more than asterisks for lists?

I think there's also an additional point: we check the sources we download. Therefore, if a commit disappears (force-pushed/removed branch/tag), the tarball disappears from upstream, or its checksum differs from what we expect, we don't build.

Might be worth mentioning that we only download sources from the internet when we're missing them locally. So hosting a mirror for sources or making sure they are available locally is a need if you want to guarantee full reproducible builds.

Quoted 4 lines

And this is especially important to mention since some projects are removing their master branches for example in favor of main, breaking older builds. Or some projects change hosting services and do not redirect, or just disappear from the Internet.

Yes, we should put something in about input validation - all inputs to the build have some kind of validation, by checksum or source control hash/revision.

Quoted 10 lines

documentation/test-manual/reproducible-builds.rst line 40

Quoted 7 lines

+=========================================

documentation/test-manual/reproducible-builds.rst new file +5 -0
+Can we prove the project is reproducible?
+=========================================
+
+Yes, we can prove it and we now regularly test this on the Autobuilder. At the time of writing (Hardknott 3.3), OE-Core is 100% reproducible for all its recipes (i.e. world builds) apart from the Go language and Ruby documentation packages. Unfortunately, the current implementation of the Go language has fundamental reproducibility problems as it always depends upon the paths it is build in.
+

s/now//

Quoted 2 lines

Please give a link to reproducibility test archives/categories on the Autobuilder if there's any available?

Isn't that below with the current status? The autobuilder would be something like:

Quoted 9 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible

but also

https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-centos
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-debian
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-fedora
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-ubuntu

Since we will have anchor links per section of this page, I feel like it's important to have links/instructions where we mention things even if it means we're repeating the same things in the same page.

Quoted 37 lines
Quoted 27 lines

s/build/built/

I would add a reference to OE-Core term :term:OpenEmbedded-Core (OE-Core) which explains what exactly it is. It is an important reference since we really mean to say that openembedded-core layer recipes (which is meta layer in poky) is 100% reproducible, nothing else. Basically, OE-Core + Bitbake is guaranteed to be reproducible, but the moment you add another layer, no guarantee (e.g. because of additional configuration files, bbappends, overridden classes, etc...)

documentation/test-manual/reproducible-builds.rst line 46

Quoted 6 lines
documentation/test-manual/reproducible-builds.rst new file +6 -0
+To run our automated selftest, as we use in our CI on the Autobuilder, you can run::
+
+   oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
+
+This defaults to including a world build so, if other layers are added, it would also run the tests for recipes in the additional layers. The first build will be run using Shared State if available, the second build explicitly disables Shared State and builds on the specific host the build is running on. This means we can test reproducibility builds between different host distributions over time on the Autobuilder.
+

s/Shared State/:ref:Shared State <overview-manual/concepts:Shared State>/

since we already reference it earlier in this page.

Review

Quoted 2 lines
(quoted diff) +2 -0
+If ``OEQA_DEBUGGING_SAVED_OUTPUT`` is set, any differing packages will be saved here. The test is also able to run the ``diffoscope`` command on the output to generate HTML files showing the differences between the packages, to aid debugging. On the Autobuilder, these appear under https://autobuilder.yocto.io/pub/repro-fail/ in the form ``oe-reproducible + <date> + <random ID>``, e.g. ``oe-reproducible-20200202-1lm8o1th``.
+

Would it make sense to document OEQA_DEBUGGING_SAVED_OUTPUT too?

Quoted 9 lines

documentation/test-manual/reproducible-builds.rst line 55

Quoted 7 lines
documentation/test-manual/reproducible-builds.rst new file +7 -0
+The project's current reproducibility status can be seen at https://www.yoctoproject.org/reproducible-build-results/
+
+===============================
+Can I test my layer or recipes?
+===============================
+
+Yes, see the self test in :ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>` at `meta/lib/oeqa/selftest/cases/reproducible.py <https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_ which can be subclassed for specific use cases.

Please provide actual commands to run so the user does not have to guess :)

Quoted 4 lines

It is stated in the section above!

(oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds)

So this means you need to test the world build in order to test your layer or recipes? Or is there a way to only test a specific recipe?

If not, it's anyway worth repeating ourselves :)

We should provide dirt-simple, easy-to-follow instructions so that it's easy to get people behind the idea that reproducible is a must and should be tested before it's too late :)

Cheers,
Quentin

#8 lore raw
Quoted 6 lines · Quentin Schulz
Quoted 3 lines

(oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds)

So this means you need to test the world build in order to test your layer or recipes? Or is there a way to only test a specific recipe?

You could test world which is what the test does.

You could subclass the test and change "targets" to a different target.

You can also change sstate_targets which would allow you to "pre-cache" some set of recipes before the test, meaning they were excluded from reproducibility testing. An in practice example would be setting sstate_targets to "core-image-sato", then targets to "core-image-sato-sdk" would test the delta between the two for reproducibility from what I remember.

Quoted 5 lines

If not, it's anyway worth repeating ourselves :)

We should provide dirt-simple, easy-to-follow instructions so that it's easy to get people behind the idea that reproducible is a must and should be tested before it's too late :)

I didn't mention much of the above as it becomes far from simple.

People will want "just my layer" and we have no way to know what that means in the general case. The tests are easy to form into your own needs though.

Cheers,

Richard

#9 lore raw

Hi Richard,

Quoted 10 lines · Richard Purdie

You could test world which is what the test does.

You could subclass the test and change "targets" to a different target.

You can also change sstate_targets which would allow you to "pre-cache" some set of recipes before the test, meaning they were excluded from reproducibility testing. An in practice example would be setting sstate_targets to "core-image-sato", then targets to "core-image-sato-sdk" would test the delta between the two for reproducibility from what I remember.

Thanks for this extra text. I tried to use it in my updates.

I'll post a V3 next in the thread to review the full text again, for people who would be interested.

Quoted 3 lines
Quoted 3 lines

If not, it's anyway worth repeating ourselves :)

Well, in this case this sounded a little too explicit, I stressed to use the command detailed above.

Quoted 4 lines
Quoted 4 lines

We should provide dirt-simple, easy-to-follow instructions so that it's easy to get people behind the idea that reproducible is a must and should be tested before it's too late :)

I like your "before it's too late" point very much!

I added a note to the "Why it matters" section to stress this.

Many thanks!

Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#10 lore raw
In reply to Richard Purdie

Quentin, Richard,

Thank you for your reviews and answers...

Quoted 15 lines · Richard Purdie
Quoted 5 lines · Quentin Schulz

Hi Michael, Richard,

Quoted 2 lines · Michael Opdenacker

From: Richard Purdie <[email protected]>

Quoted 9 lines

documentation/test-manual/reproducible-builds.rst line 18

Quoted 8 lines
documentation/test-manual/reproducible-builds.rst new file +8 -0
+The Yocto Project defines reproducibility as where a given input build configuration will give the same binary output regardless of when it is built (now or in 5 years time), regardless of the path on the filesystem the build is run in, and regardless of the distro and tools on the underlying host system the build is running on.
+
+==============
+Why it matters
+==============
+
+The project aligns with the `Reproducible Builds project <https://reproducible-builds.org/>`_, which shares information about why reproducibility matters. The primary focus of the project is the ability to detect security issues being introduced. However, from a Yocto Project perspective, it is also hugely important that our builds are deterministic. When you build a given input set of metadata, we expect you to get consistent output. This has always been a key focus but is now true down to the binary level including timestamps.
+

"now" is too abstract? Can we say at least since when (e.g. which release)?

Quoted 3 lines

As of dunfell we enabled reprodubile builds in poky by default.

http://git.yoctoproject.org/cgit.cgi/poky/commit/meta-poky/conf/distro/poky.conf?id=25a17f0346ab05b3212bb280e4a8b622bcf99bce

I agree wit Quentin, the documentation will live for years. I also replaced "at the time of writing" by a release name. I updated this part too.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
documentation/test-manual/reproducible-builds.rst new file +6 -0
+For example, at some point in the future life of a product, you find that you need to rebuild to add a security fix. If this happens, only the components that have been modified should change at the binary level. This would lead to much easier and clearer bounds on where validation is needed.
+
+This also gives an additional benefit to the project builds themselves, our hash equivalence for :ref:`Shared State <overview-manual/concepts:Shared State>` object reuse works much more effectively when the binary output remains the same.
+
+===================
+How we implement it
Quoted 8 lines

documentation/test-manual/reproducible-builds.rst line 29

Quoted 5 lines

+===================

documentation/test-manual/reproducible-builds.rst new file +4 -0
+
+There are many different aspects to build reproducibility, but some particular things we do within the build system to ensure reproducibility include:
+
+* Adding mappings to the compiler options to ensure debug filepaths are mapped to consistent target compatible paths. This is done through the ``DEBUG_PREFIX_MAP`` variable which sets the ``-fmacro-prefix-map`` and ``-fdebug-prefix-map`` compiler options correctly to map to target paths.

Would documenting DEBUG_PREFIX_MAP make sense? Is it supposed to be
changed by users in some cases?
for reference, where it's being defined: https://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/bitbake.conf#n612

Quoted 1 line

Users shouldn't need to mess with it but it should be documented.

Good. I'll keep track of this.

Quoted 8 lines
Quoted 6 lines

documentation/test-manual/reproducible-builds.rst line 31

Quoted 3 lines
documentation/test-manual/reproducible-builds.rst new file +2 -0
+* Being explicit about recipe dependencies and their configuration (no floating configure options or host dependencies creeping in). In particular this means making sure :term:`PACKAGECONFIG` coverage covers configure options which may otherwise try and auto-detect host dependencies.
+* Using recipe specific sysroots to isolate recipes so they only see their dependencies. These are visible as ``recipe-sysroot`` and ``recipe-sysroot-native`` directories within the :term:`WORKDIR` of a given recipe and are populated only with the dependencies a recipe has.

Can't we just reference RECIPE_SYSROOT and RECIPE_SYSROOT_NATIVE here? Aaaand I just discovered these aren't in the variable glossary but I find them very important?

I suspect unfortunately our recipe specific sysroots are basically not documented at all :(. Again, probably needs fixing ultimately.

Not completely true fortunately. At least you have the explanations on
https://docs.yoctoproject.org/overview-manual/concepts.html#configuration-compilation-and-staging

I'm keeping another note to keep track of this, and later we will create a bug when we have the new "Variable Glossary" category.

Quoted 7 lines
Quoted 7 lines

Can be done later on since they do not exist yet.

documentation/test-manual/reproducible-builds.rst line 34

Quoted 3 lines
documentation/test-manual/reproducible-builds.rst new file +3 -0
+* Build images from a reduced package set: only packages from recipes the image depends upon.
+* Filtering the tools available from the host's PATH to only a specific set of tools, set using the :term:`HOSTTOOLS` variable.
+

``PATH``?

Right, fixed.

Quoted 2 lines
Quoted 2 lines

Also, I think we use dashes more than asterisks for lists?

Oops, the result seems to be the same, but yes, that's our standard. Fixed. Do you know why dashes where chosen over lists?

I also forgot to pay attention to indenting. I've just fixed it too, to match the 80 columns format that we usually have.

Quoted 11 lines
Quoted 11 lines

I think there's also an additional point: we check the sources we download. Therefore, if a commit disappears (force-pushed/removed branch/tag), the tarball disappears from upstream, or its checksum differs from what we expect, we don't build.

Might be worth mentioning that we only download sources from the internet when we're missing them locally. So hosting a mirror for sources or making sure they are available locally is a need if you want to guarantee full reproducible builds.

Quoted 7 lines
Quoted 4 lines

And this is especially important to mention since some projects are removing their master branches for example in favor of main, breaking older builds. Or some projects change hosting services and do not redirect, or just disappear from the Internet.

Yes, we should put something in about input validation - all inputs to the build have some kind of validation, by checksum or source control hash/revision.

All right, I'll add something for you to review.

Quoted 8 lines
Quoted 8 lines

documentation/test-manual/reproducible-builds.rst line 40

Quoted 7 lines

+=========================================

documentation/test-manual/reproducible-builds.rst new file +5 -0
+Can we prove the project is reproducible?
+=========================================
+
+Yes, we can prove it and we now regularly test this on the Autobuilder. At the time of writing (Hardknott 3.3), OE-Core is 100% reproducible for all its recipes (i.e. world builds) apart from the Go language and Ruby documentation packages. Unfortunately, the current implementation of the Go language has fundamental reproducibility problems as it always depends upon the paths it is build in.
+

s/now//

Fixed

Quoted 6 lines
Quoted 3 lines

Please give a link to reproducibility test archives/categories on the Autobuilder if there's any available?

Isn't that below with the current status? The autobuilder would be something like:

Quoted 8 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible

but also

https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-centos
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-debian
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-fedora
https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-ubuntu

I didn't manage to find the links you referred to from
https://www.yoctoproject.org/reproducible-build-results/ .

I'm adding those too.k

Quoted 1 line
Quoted 1 line

s/build/built/

Fixed.

Quoted 8 lines
Quoted 8 lines

I would add a reference to OE-Core term :term:OpenEmbedded-Core (OE-Core) which explains what exactly it is. It is an important reference since we really mean to say that openembedded-core layer recipes (which is meta layer in poky) is 100% reproducible, nothing else. Basically, OE-Core + Bitbake is guaranteed to be reproducible, but the moment you add another layer, no guarantee (e.g. because of additional configuration files, bbappends, overridden classes, etc...)

Done (adding the reference). I also liked your comment so I added that as a note.

Quoted 10 lines
Quoted 10 lines

documentation/test-manual/reproducible-builds.rst line 46

Quoted 7 lines
documentation/test-manual/reproducible-builds.rst new file +6 -0
+To run our automated selftest, as we use in our CI on the Autobuilder, you can run::
+
+   oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
+
+This defaults to including a world build so, if other layers are added, it would also run the tests for recipes in the additional layers. The first build will be run using Shared State if available, the second build explicitly disables Shared State and builds on the specific host the build is running on. This means we can test reproducibility builds between different host distributions over time on the Autobuilder.
+

s/Shared State/:ref:Shared State <overview-manual/concepts:Shared State>/

since we already reference it earlier in this page.

Oops, good catch. Thanks.

Quoted 4 lines
Quoted 4 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+If ``OEQA_DEBUGGING_SAVED_OUTPUT`` is set, any differing packages will be saved here. The test is also able to run the ``diffoscope`` command on the output to generate HTML files showing the differences between the packages, to aid debugging. On the Autobuilder, these appear under https://autobuilder.yocto.io/pub/repro-fail/ in the form ``oe-reproducible + <date> + <random ID>``, e.g. ``oe-reproducible-20200202-1lm8o1th``.
+

Would it make sense to document OEQA_DEBUGGING_SAVED_OUTPUT too?

Richard, mailing list readers, any opinion on this?

Cheers,

Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#11 lore raw
Quoted 10 lines · Michael Opdenacker

Quentin, Richard,

Thank you for your reviews and answers...

Quoted 5 lines · Richard Purdie
Quoted 4 lines · Quentin Schulz

Hi Michael, Richard,

Quoted 1 line · Michael Opdenacker

From: Richard Purdie <[email protected]>

[...]

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

Also, I think we use dashes more than asterisks for lists?

Oops, the result seems to be the same, but yes, that's our standard. Fixed. Do you know why dashes where chosen over lists?

Probably the migration tool? But no, no idea. One more rule to put in the theoretical CONTRIBUTING file :p

Cheers,
Quentin