test-manual: add initial reproducible builds documentation
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v3 1/1 test-manual: add initial reproducible builds documentation
From: Richard Purdie <[email protected]>
--- documentation/test-manual/index.rst | 1 + .../test-manual/reproducible-builds.rst | 134 ++++++++++++++++++ 2 files changed, 135 insertions(+) create mode 100644 documentation/test-manual/reproducible-builds.rst
documentation/test-manual/index.rst +1 -0
@@ -13,6 +13,7 @@ Yocto Project Test Environment Manual intro test-process understand-autobuilder + reproducible-builds history .. include:: /boilerplate.rst
documentation/test-manual/reproducible-builds.rst new file +134 -0
134 lines not on this page · raw message
Signature · 1 line
2.25.1
Quoted 46 lines · Michael Opdenacker
From: Richard Purdie <[email protected]>
Signed-off-by Richard Purdie <[email protected]>Signed-off-by Michael Opdenacker <[email protected]>Reviewed-by Quentin Schulz <[email protected]>--- documentation/test-manual/index.rst | 1 + .../test-manual/reproducible-builds.rst | 134 ++++++++++++++++++ 2 files changed, 135 insertions(+) create mode 100644 documentation/test-manual/reproducible-builds.rstdocumentation/test-manual/index.rst +1
-0@@ -13,6 +13,7 @@ Yocto Project Test Environment Manual intro test-process understand-autobuilder + reproducible-builds history .. include:: /boilerplate.rstdocumentation/test-manual/reproducible-builds.rst new file +17
-0@@ -0,0 +1,134 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +******************* +Reproducible Builds +******************* + +================ +How we define it +================ + +The Yocto Project defines reproducibility as where a given input build +configuration will give the same binary output regardless of when it is built +(now or in 5 years time), regardless of the path on the filesystem the build is +run in, and regardless of the distro and tools on the underlying host system the +build is running on. + +==============
documentation/test-manual/reproducible-builds.rst line 27
Quoted 10 lines
documentation/test-manual/reproducible-builds.rst new file +10
-0+Why it matters +============== + +The project aligns with the `Reproducible Builds project +<https://reproducible-builds.org/>`_, which shares information about why +reproducibility matters. The primary focus of the project is the ability to +detect security issues being introduced. However, from a Yocto Project +perspective, it is also hugely important that our builds are deterministic. When +you build a given input set of metadata, we expect you to get consistent output. +This has always been a key focus but, since release 3.1 ("dunfell"), it is now true
Just picking nits here, but I could see a reference to:
https://docs.yoctoproject.org/ref-manual/migration-3.1.html#reproducible-builds-now-enabled-by-default
Quoted 98 lines
documentation/test-manual/reproducible-builds.rst new file +97
-0+down to the binary level including timestamps. + +For example, at some point in the future life of a product, you find that you +need to rebuild to add a security fix. If this happens, only the components that +have been modified should change at the binary level. This would lead to much +easier and clearer bounds on where validation is needed. + +This also gives an additional benefit to the project builds themselves, our hash +equivalence for :ref:`Shared State <overview-manual/concepts:Shared State>` +object reuse works much more effectively when the binary output remains the +same. + +.. note:: + + We strongly advise you to make sure your project builds reproducibly + before finalizing your production images. It would be too late if you + only address this issue when the first updates are required. + +=================== +How we implement it +=================== + +There are many different aspects to build reproducibility, but some particular +things we do within the build system to ensure reproducibility include: + +- Adding mappings to the compiler options to ensure debug filepaths are mapped + to consistent target compatible paths. This is done through the + ``DEBUG_PREFIX_MAP`` variable which sets the ``-fmacro-prefix-map`` and + ``-fdebug-prefix-map`` compiler options correctly to map to target paths. +- Being explicit about recipe dependencies and their configuration (no floating + configure options or host dependencies creeping in). In particular this means + making sure :term:`PACKAGECONFIG` coverage covers configure options which may + otherwise try and auto-detect host dependencies. +- Using recipe specific sysroots to isolate recipes so they only see their + dependencies. These are visible as ``recipe-sysroot`` and + ``recipe-sysroot-native`` directories within the :term:`WORKDIR` of a given + recipe and are populated only with the dependencies a recipe has. +- Build images from a reduced package set: only packages from recipes the image + depends upon. +- Filtering the tools available from the host's ``PATH`` to only a specific set + of tools, set using the :term:`HOSTTOOLS` variable. + +========================================= +Can we prove the project is reproducible? +========================================= + +Yes, we can prove it and we regularly test this on the Autobuilder. At the +time of writing (release 3.3, "hardknott"), :term:`OpenEmbedded-Core (OE-Core)` +is 100% reproducible for all its recipes (i.e. world builds) apart from the Go +language and Ruby documentation packages. Unfortunately, the current +implementation of the Go language has fundamental reproducibility problems as +it always depends upon the paths it is built in. + +.. note:: + + Only BitBake and :term:`OpenEmbedded-Core (OE-Core)`, which is the ``meta`` + layer in Poky, guarantee complete reproducibility. The moment you add + another layer, this warranty is voided, because of additional configuration + files, ``bbappend`` files, overridden classes, etc. + +To run our automated selftest, as we use in our CI on the Autobuilder, you can +run:: + + oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds + +This defaults to including a ``world`` build so, if other layers are added, it would +also run the tests for recipes in the additional layers. The first build will be +run using :ref:`Shared State <overview-manual/concepts:Shared State>` if +available, the second build explicitly disables +:ref:`Shared State <overview-manual/concepts:Shared State>` and builds on the +specific host the build is running on. This means we can test reproducibility +builds between different host distributions over time on the Autobuilder. + +If ``OEQA_DEBUGGING_SAVED_OUTPUT`` is set, any differing packages will be saved +here. The test is also able to run the ``diffoscope`` command on the output to +generate HTML files showing the differences between the packages, to aid +debugging. On the Autobuilder, these appear under +https://autobuilder.yocto.io/pub/repro-fail/ in the form ``oe-reproducible + +<date> + <random ID>``, e.g. ``oe-reproducible-20200202-1lm8o1th``. + +The project's current reproducibility status can be seen at +https://www.yoctoproject.org/reproducible-build-results/ + +You can also check the reproducibility status on supported host distributions: + +- CentOS: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-centos +- Debian: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-debian +- Fedora: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-fedora +- Ubuntu: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-ubuntu + +=============================== +Can I test my layer or recipes? +=============================== + +Once again, you can run a ``world`` test using the +:ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>` +command provided above. This functionality is implemented
documentation/test-manual/reproducible-builds.rst line 134
Quoted 10 lines
documentation/test-manual/reproducible-builds.rst new file +10
-0+in `meta/lib/oeqa/selftest/cases/reproducible.py +<https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_. + +You could subclass the test and change ``targets`` to a different target. + +You may also change ``sstate_targets`` which would allow you to "pre-cache" some +set of recipes before the test, meaning they are excluded from reproducibility +testing. As a practical example, you could set ``sstate_targets`` to +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.
I'm not sure this section has its place in our documentation since it seems bound to the current implementation and explains modifications of the code.
I'd vote it out.
Cheers,
Quentin
Quoted 16 lines · Quentin Schulz
documentation/test-manual/reproducible-builds.rst line 134
Quoted 9 lines · Michael Opdenacker
documentation/test-manual/reproducible-builds.rst new file +8
-0+ +You could subclass the test and change ``targets`` to a different target. + +You may also change ``sstate_targets`` which would allow you to "pre-cache" some +set of recipes before the test, meaning they are excluded from reproducibility +testing. As a practical example, you could set ``sstate_targets`` to +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.I'm not sure this section has its place in our documentation since it seems bound to the current implementation and explains modifications of the code.
Quoted 1 line
I'd vote it out.
Doesn't most of the manual document the current implementation?
When this wasn't here, you wanted an example! I do agree we should give people some kind of a hint about how to write their own tests. We're likely going to end up adding more example, not less?
Cheers,
Richard
Hi Richard,
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Quentin Schulz
documentation/test-manual/reproducible-builds.rst line 134
Quoted 9 lines · Michael Opdenacker
documentation/test-manual/reproducible-builds.rst new file +8
-0+ +You could subclass the test and change ``targets`` to a different target. + +You may also change ``sstate_targets`` which would allow you to "pre-cache" some +set of recipes before the test, meaning they are excluded from reproducibility +testing. As a practical example, you could set ``sstate_targets`` to +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.I'm not sure this section has its place in our documentation since it seems bound to the current implementation and explains modifications of the code.
Quoted 8 lines
Quoted 1 line
I'd vote it out.
Doesn't most of the manual document the current implementation?
When this wasn't here, you wanted an example! I do agree we should give people some kind of a hint about how to write their own tests. We're likely going to end up adding more example, not less?
Because I understood the original section as "it's very simple to do this, look into this file", so I assumed some arguments could be passed to make it easy to re-use. I wasn't expecting changes in a Python script.
As opposed to classes, variables and/or tasks, which to me seems to be kind of an ABI, so it is maintained and watched for changes and if there's any, usually listed in the migration manual, I would assume the content of Python scripts aren't expected to stay "stable" or their change be documented otherwise?
Here the suggested example is about modifying variables that are used only in a given Python class, which could change over time, including the name of the variables to be changed.
This also kind of encourages people to modify code in layers they don't have push access to, which is something that we usually don't want people to do IIRC?
I'm not sure I managed to convey my thoughts better. I just expected a simple command to be able to test my recipes, or via a configuration script.
Anyway, no big deal, I'm happy to see the reproducible test procedure is being documented :)
Cheers,
Quentin
Quoted 29 lines · Quentin Schulz
Hi Richard,
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Quentin Schulz
documentation/test-manual/reproducible-builds.rst line 134
Quoted 9 lines · Michael Opdenacker
documentation/test-manual/reproducible-builds.rst new file +8
-0+ +You could subclass the test and change ``targets`` to a different target. + +You may also change ``sstate_targets`` which would allow you to "pre-cache" some +set of recipes before the test, meaning they are excluded from reproducibility +testing. As a practical example, you could set ``sstate_targets`` to +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.I'm not sure this section has its place in our documentation since it seems bound to the current implementation and explains modifications of the code.
Quoted 8 lines
Quoted 1 line
I'd vote it out.
Doesn't most of the manual document the current implementation?
When this wasn't here, you wanted an example! I do agree we should give people some kind of a hint about how to write their own tests. We're likely going to end up adding more example, not less?
Quoted 4 lines
Because I understood the original section as "it's very simple to do this, look into this file", so I assumed some arguments could be passed to make it easy to re-use. I wasn't expecting changes in a Python script.
I think the confusion here is the definition of simple :).
At least in theory, you can create your own test case in your own layer and in that test, inherit and subclass the original test (standard python). In that subclassed test, you could change those two variables and you should then have your own test case.
Would that work? Should do, at least mostly but I will admit I've not tried it. Adding a new test in your layer and also subclassing an existing test is something we should document somewhere in the test manual ultimately. That is partly why I continued to expand on what I meant here.
Quoted 9 lines
As opposed to classes, variables and/or tasks, which to me seems to be kind of an ABI, so it is maintained and watched for changes and if there's any, usually listed in the migration manual, I would assume the content of Python scripts aren't expected to stay "stable" or their change be documented otherwise?
Here the suggested example is about modifying variables that are used only in a given Python class, which could change over time, including the name of the variables to be changed.
I agree it is a bit risky to have these in the manual, equally, the manual does have worked examples and I believe there is a good case for having them here as this is something we want to teach people how to do.
FWIW that test was designed to be sub-classed so there is at least some intent to allow people to do this. As such we'd probably try to maintain some kind of compatibility if we can.
Quoted 3 lines
This also kind of encourages people to modify code in layers they don't have push access to, which is something that we usually don't want people to do IIRC?
The key missing detail is you can put test code in your own layer. I can even point at an example:
(which is also how we test this stays working)
Quoted 6 lines
I'm not sure I managed to convey my thoughts better. I just expected a simple command to be able to test my recipes, or via a configuration script.
Anyway, no big deal, I'm happy to see the reproducible test procedure is being documented :)
Thanks for the review, it is helpful to have have a fresher perspective on this!
Cheers,
Richard
Hi Richard,
Quoted 42 lines · Richard Purdie
Quoted 29 lines · Quentin Schulz
Hi Richard,
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Quentin Schulz
documentation/test-manual/reproducible-builds.rst line 134
Quoted 9 lines · Michael Opdenacker
documentation/test-manual/reproducible-builds.rst new file +8
-0+ +You could subclass the test and change ``targets`` to a different target. + +You may also change ``sstate_targets`` which would allow you to "pre-cache" some +set of recipes before the test, meaning they are excluded from reproducibility +testing. As a practical example, you could set ``sstate_targets`` to +``core-image-sato``, then setting ``targets`` to ``core-image-sato-sdk`` would +run reproducibility tests only on the targets belonging only to ``core-image-sato-sdk``.I'm not sure this section has its place in our documentation since it seems bound to the current implementation and explains modifications of the code.
Quoted 8 lines
Quoted 1 line
I'd vote it out.
Doesn't most of the manual document the current implementation?
When this wasn't here, you wanted an example! I do agree we should give people some kind of a hint about how to write their own tests. We're likely going to end up adding more example, not less?
Quoted 4 lines
Because I understood the original section as "it's very simple to do this, look into this file", so I assumed some arguments could be passed to make it easy to re-use. I wasn't expecting changes in a Python script.
I think the confusion here is the definition of simple :).
At least in theory, you can create your own test case in your own layer and in that test, inherit and subclass the original test (standard python). In that subclassed test, you could change those two variables and you should then have your own test case.
Quoted 4 lines
Would that work? Should do, at least mostly but I will admit I've not tried it. Adding a new test in your layer and also subclassing an existing test is something we should document somewhere in the test manual ultimately. That is partly why I continued to expand on what I meant here.
I re-read the previous mail and I think my brain just ignored the following sentence:
You could subclass the test and change ``targets`` to a different target.
Which should have put me on the right track. I was missing this bit in my understanding. And now, I happily concede that my comments didn't make much sense considering what you suggested :)
Quoted 11 lines
Quoted 10 lines
As opposed to classes, variables and/or tasks, which to me seems to be kind of an ABI, so it is maintained and watched for changes and if there's any, usually listed in the migration manual, I would assume the content of Python scripts aren't expected to stay "stable" or their change be documented otherwise?
Here the suggested example is about modifying variables that are used only in a given Python class, which could change over time, including the name of the variables to be changed.
Quoted 8 lines
I agree it is a bit risky to have these in the manual, equally, the manual does have worked examples and I believe there is a good case for having them here as this is something we want to teach people how to do.
FWIW that test was designed to be sub-classed so there is at least some intent to allow people to do this. As such we'd probably try to maintain some kind of compatibility if we can.
Agreed. I would try to make sure subclassing works since you seem to insinuate you're not 100% sure it works right now.
I would very much see a "code" example, with the correct path in the layer (if that matters in some ways to bitbake) somewhere in the docs then. If it does not make sense in this section, at least refer to where it will lie.
Quoted 11 lines
Quoted 3 lines
This also kind of encourages people to modify code in layers they don't have push access to, which is something that we usually don't want people to do IIRC?
The key missing detail is you can put test code in your own layer. I can even point at an example:
(which is also how we test this stays working)
Indeed, thanks for clarifying this, it makes sense to me now.
Cheers,
Quentin
Hello Quentin,
Quoted 9 lines · Quentin Schulz
documentation/test-manual/reproducible-builds.rst line 27
Quoted 7 lines
documentation/test-manual/reproducible-builds.rst new file +7
-0+The project aligns with the `Reproducible Builds project +<https://reproducible-builds.org/>`_, which shares information about why +reproducibility matters. The primary focus of the project is the ability to +detect security issues being introduced. However, from a Yocto Project +perspective, it is also hugely important that our builds are deterministic. When +you build a given input set of metadata, we expect you to get consistent output. +This has always been a key focus but, since release 3.1 ("dunfell"), it is now trueJust picking nits here, but I could see a reference to:
https://docs.yoctoproject.org/ref-manual/migration-3.1.html#reproducible-builds-now-enabled-by-default
Good idea. I implemented it.
Thanks!
Michael.
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Hello,
Commenting my own patch, before others do :-}
documentation/test-manual/reproducible-builds.rst line 109
Quoted 2 lines · Michael Opdenacker
documentation/test-manual/reproducible-builds.rst new file +2
-0+The project's current reproducibility status can be seen at +https://www.yoctoproject.org/reproducible-build-results/
Oops, my bad, I could have used the :yocto_home: macro here. Will do.
documentation/test-manual/reproducible-builds.rst line 116
Quoted 7 lines
documentation/test-manual/reproducible-builds.rst new file +7
-0+ +You can also check the reproducibility status on supported host distributions: + +- CentOS: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-centos +- Debian: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-debian +- Fedora: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-fedora +- Ubuntu: https://autobuilder.yoctoproject.org/typhoon/#/builders/reproducible-ubuntu
Same here, will use :yocto_ab:
documentation/test-manual/reproducible-builds.rst line 126
Quoted 10 lines
documentation/test-manual/reproducible-builds.rst new file +10
-0+ +=============================== +Can I test my layer or recipes? +=============================== + +Once again, you can run a ``world`` test using the +:ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>` +command provided above. This functionality is implemented +in `meta/lib/oeqa/selftest/cases/reproducible.py +<https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py>`_.
Same here, will use :yocto_git:
Pfooh, that was just in time before Quentin and Nico catch those ;-)
Michael.
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com