Skip to contentSkip to conversation

Reproducible build

OE-core Frédéric Martinsons 3 participants 8 messages Apr 8, 2023 – Apr 11, 2023 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Forget my previous question,

I simply edit
https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py#n137
to replace by zavariant, and with OEQA_DEBUGGING_SAVED_OUTPUT set , I managed to see the html diff

Nevertheless, It took one hour, but it seems that it cannot be avoided since there are two builds and one
without any shared state.

Quoted 26 lines · Frederic Martinsons via lists.openembedded.org

Hello,

I plan to work on https://bugzilla.yoctoproject.org/show_bug.cgi?id=15090
in the coming days.
To show how things are tested I ran "oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds" but even on a strong machine, it took several hours (I waited for 6h and it is not finished yet).

Does anyone know if it is possible to test reproducibility on a single package (here, zvariant) ?
Should I tweak the python test class or is there a simple way ?

Thanks

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179826):
https://lists.openembedded.org/g/openembedded-core/message/179826
Mute This Topic: https://lists.openembedded.org/mt/98126404/6213388
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw
Quoted 11 lines · Frederic Martinsons

Forget my previous question,

I simply
edit https://git.openembedded.org/openembedded-core/tree/meta/lib/oeq
a/selftest/cases/reproducible.py#n137
to replace by zavariant, and with OEQA_DEBUGGING_SAVED_OUTPUT set , I managed to see the html diff

Nevertheless, It took one hour, but it seems that it cannot be avoided since there are two builds and one
without any shared state.

Unfortunately that is the nature of the test, one side has to be built from scratch. Tweaking the test as you'd done to target a specific component is what I'd have done too to speed things up for a specific recipe.

To get a further speedup, you could probably keep the sstate for each of the two builds separate but retain both, then in theory on the pieces you change would rebuild. Again, it would mean tweaking the test but for debugging, I think that is fine. Rust can be a pain if you just rebuild those as it still takes a while. Thanks for taking a look!

Cheers,

Richard

#3 lore raw
Quoted 24 lines · Richard Purdie
Quoted 11 lines · Frederic Martinsons

Forget my previous question,

I simply
edit https://git.openembedded.org/openembedded-core/tree/meta/lib/oeq
a/selftest/cases/reproducible.py#n137
to replace by zavariant, and with OEQA_DEBUGGING_SAVED_OUTPUT set , I managed to see the html diff

Nevertheless, It took one hour, but it seems that it cannot be avoided since there are two builds and one
without any shared state.

Unfortunately that is the nature of the test, one side has to be built from scratch. Tweaking the test as you'd done to target a specific component is what I'd have done too to speed things up for a specific recipe.

To get a further speedup, you could probably keep the sstate for each of the two builds separate but retain both, then in theory on the pieces you change would rebuild. Again, it would mean tweaking the test but for debugging, I think that is fine. Rust can be a pain if you just rebuild those as it still takes a while. Thanks for taking a look!

Quoted 5 lines

Cheers,

Richard

Thank you for the tip, I'll try that, everything that help to speed up is good to take of course.

#4 lore raw

Hi Frederic,

Frederic Martinsons <[email protected]> escreveu no dia domingo, 9/04/2023 à(s) 16:05:

Quoted 29 lines
Quoted 24 lines · Richard Purdie
Quoted 11 lines · Frederic Martinsons

Forget my previous question,

I simply
edit https://git.openembedded.org/openembedded-core/tree/meta/lib/oeq
a/selftest/cases/reproducible.py#n137
to replace by zavariant, and with OEQA_DEBUGGING_SAVED_OUTPUT set , I managed to see the html diff

Nevertheless, It took one hour, but it seems that it cannot be avoided since there are two builds and one
without any shared state.

Unfortunately that is the nature of the test, one side has to be built from scratch. Tweaking the test as you'd done to target a specific component is what I'd have done too to speed things up for a specific recipe.

To get a further speedup, you could probably keep the sstate for each of the two builds separate but retain both, then in theory on the pieces you change would rebuild. Again, it would mean tweaking the test but for debugging, I think that is fine. Rust can be a pain if you just rebuild those as it still takes a while. Thanks for taking a look!

Quoted 8 lines
Quoted 5 lines

Cheers,

Richard

Thank you for the tip, I'll try that, everything that help to speed up is good to take of course.

To speedup the build you can take the rust-native from the state cache [1] and this will improve a bit bit the time needed:

sstate_target = [ "rust-native" ]

[1] https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py#n139

Jose

Quoted 12 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179841):
https://lists.openembedded.org/g/openembedded-core/message/179841
Mute This Topic: https://lists.openembedded.org/mt/98126404/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Best regards,

José Quaresma
#5 lore raw
Quoted 67 lines · Jose Quaresma

Hi Frederic,

Frederic Martinsons <[email protected]> escreveu no dia domingo, 9/04/2023 à(s) 16:05:

Quoted 29 lines
Quoted 24 lines · Richard Purdie
Quoted 11 lines · Frederic Martinsons

Forget my previous question,

I simply
edit https://git.openembedded.org/openembedded-core/tree/meta/lib/oeq
a/selftest/cases/reproducible.py#n137
to replace by zavariant, and with OEQA_DEBUGGING_SAVED_OUTPUT set , I managed to see the html diff

Nevertheless, It took one hour, but it seems that it cannot be avoided since there are two builds and one
without any shared state.

Unfortunately that is the nature of the test, one side has to be built from scratch. Tweaking the test as you'd done to target a specific component is what I'd have done too to speed things up for a specific recipe.

To get a further speedup, you could probably keep the sstate for each of the two builds separate but retain both, then in theory on the pieces you change would rebuild. Again, it would mean tweaking the test but for debugging, I think that is fine. Rust can be a pain if you just rebuild those as it still takes a while. Thanks for taking a look!

Quoted 8 lines
Quoted 5 lines

Cheers,

Richard

Thank you for the tip, I'll try that, everything that help to speed up is good to take of course.

To speedup the build you can take the rust-native from the state cache [1] and this will improve a bit bit the time needed:

sstate_target = [ "rust-native" ]

[1] https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py#n139

Jose

Quoted 12 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179841):
https://lists.openembedded.org/g/openembedded-core/message/179841
Mute This Topic: https://lists.openembedded.org/mt/98126404/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Quoted 5 lines

--
Best regards,

José Quaresma

Hey, thank you José for the advice, I'll apply that for sure.

By the way, like I commented in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15090 , I'm currently try to reproduce (ahah) the reproducibility issue of zvariant with the use of reprotest (https://salsa.debian.org/reproducible-builds/reprotest)
and outside yocto environment. For the moment, I was not able to pinpoint the issue (I must have
put aside the faketime usage since it caused me segfault of rustc).

I plan to dig further in reproducible.py class and how the package is built under yocto to try to find out
what is going on.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw

Hello with the patch below:

meta/lib/oeqa/selftest/cases/reproducible.py +0 -1
@@ -134,9 +134,9 @@ class ReproducibleTests(OESelftestTestCase):
     max_report_size = 250 * 1024 * 1024
 
     # targets are the things we want to test the reproducibility of
-    targets = ['core-image-minimal', 'core-image-sato',

'core-image-full-cmdline', 'core-image-weston', 'world']
+ targets = ['zvariant']

     # sstate targets are things to pull from sstate to potentially cut
build/debugging time
-    sstate_targets = []
+    sstate_targets = ['rust-native', 'cargo-native']
     save_results = False
     if 'OEQA_DEBUGGING_SAVED_OUTPUT' in os.environ:
         save_results = os.environ['OEQA_DEBUGGING_SAVED_OUTPUT']

I managed to reduce the time to test to 30mn (which is reasonable). Today, I made 5 run of this
reproducibility test and if library is indeed different between build A and build B every time, this
is the same exact difference, that is to day that the 5 libraries in build A are identical and so are
the 5 libraries in build B.

I admit I'm surprised , I would have waited for some random diff. I'll continue my investigation but if this rang a bell to someone here, feel free to tell (this does
seem strange to me).

Quoted 100 lines · Frederic Martinsons via lists.openembedded.org
Quoted 59 lines · Jose Quaresma

Hi Frederic,

Frederic Martinsons <[email protected]> escreveu no dia domingo, 9/04/2023 à(s) 16:05:

Quoted 29 lines
Quoted 24 lines · Richard Purdie
Quoted 11 lines · Frederic Martinsons

Forget my previous question,

I simply
edit https://git.openembedded.org/openembedded-core/tree/meta/lib/oeq
a/selftest/cases/reproducible.py#n137
to replace by zavariant, and with OEQA_DEBUGGING_SAVED_OUTPUT set , I managed to see the html diff

Nevertheless, It took one hour, but it seems that it cannot be avoided since there are two builds and one
without any shared state.

Unfortunately that is the nature of the test, one side has to be built from scratch. Tweaking the test as you'd done to target a specific component is what I'd have done too to speed things up for a specific recipe.

To get a further speedup, you could probably keep the sstate for each of the two builds separate but retain both, then in theory on the pieces you change would rebuild. Again, it would mean tweaking the test but for debugging, I think that is fine. Rust can be a pain if you just rebuild those as it still takes a while. Thanks for taking a look!

Quoted 8 lines
Quoted 5 lines

Cheers,

Richard

Thank you for the tip, I'll try that, everything that help to speed up is good to take of course.

To speedup the build you can take the rust-native from the state cache
[1] and this will improve a bit bit the time needed:

sstate_target = [ "rust-native" ]

[1] https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/reproducible.py#n139

Jose

Quoted 4 lines
Quoted 5 lines

--
Best regards,

José Quaresma

Hey, thank you José for the advice, I'll apply that for sure.

By the way, like I commented in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15090 , I'm currently try to reproduce (ahah) the reproducibility issue of zvariant with the use of reprotest (https://salsa.debian.org/reproducible-builds/reprotest)
and outside yocto environment. For the moment, I was not able to pinpoint the issue (I must have
put aside the faketime usage since it caused me segfault of rustc).

I plan to dig further in reproducible.py class and how the package is built under yocto to try to find out
what is going on.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179892):
https://lists.openembedded.org/g/openembedded-core/message/179892
Mute This Topic: https://lists.openembedded.org/mt/98126404/6213388
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#8 lore raw
Quoted 27 lines · Frédéric Martinsons

Hello with the patch below:

meta/lib/oeqa/selftest/cases/reproducible.py +0 -0
@@ -134,9 +134,9 @@ class ReproducibleTests(OESelftestTestCase):

     max_report_size = 250 * 1024 * 1024

     # targets are the things we want to test the reproducibility of
-    targets = ['core-image-minimal', 'core-image-sato', 'core-image-
full-cmdline', 'core-image-weston', 'world']
+    targets = ['zvariant']
     # sstate targets are things to pull from sstate to potentially
cut build/debugging time
-    sstate_targets = []
+    sstate_targets = ['rust-native', 'cargo-native']
     save_results = False
     if 'OEQA_DEBUGGING_SAVED_OUTPUT' in os.environ:
         save_results = os.environ['OEQA_DEBUGGING_SAVED_OUTPUT']

I managed to reduce the time to test to 30mn (which is reasonable). Today, I made 5 run of this
reproducibility test and if library is indeed different between build A and build B every time, this
is the same exact difference, that is to day that the 5 libraries in build A are identical and so are
the 5 libraries in build B.

Quoted 4 lines

I admit I'm surprised , I would have waited for some random diff. I'll continue my investigation but if this rang a bell to someone here, feel free to tell (this does
seem strange to me).

It likely means it is the path the build is made in that is being encoded into the build somehow. The same binary is resulting for a given build path?

Cheers,

Richard

#9 lore raw
Quoted 37 lines · Richard Purdie
Quoted 27 lines · Frédéric Martinsons

Hello with the patch below:

meta/lib/oeqa/selftest/cases/reproducible.py +0 -1
@@ -134,9 +134,9 @@ class ReproducibleTests(OESelftestTestCase):
     max_report_size = 250 * 1024 * 1024
 
     # targets are the things we want to test the reproducibility of
-    targets = ['core-image-minimal', 'core-image-sato', 'core-image-

full-cmdline', 'core-image-weston', 'world']
+ targets = ['zvariant']

     # sstate targets are things to pull from sstate to potentially
cut build/debugging time
meta/lib/oeqa/selftest/cases/reproducible.py +1 -1
-    sstate_targets = []
+    sstate_targets = ['rust-native', 'cargo-native']
     save_results = False
     if 'OEQA_DEBUGGING_SAVED_OUTPUT' in os.environ:
         save_results = os.environ['OEQA_DEBUGGING_SAVED_OUTPUT']

I managed to reduce the time to test to 30mn (which is reasonable). Today, I made 5 run of this
reproducibility test and if library is indeed different between build A and build B every time, this
is the same exact difference, that is to day that the 5 libraries in build A are identical and so are
the 5 libraries in build B.

Quoted 4 lines

I admit I'm surprised , I would have waited for some random diff. I'll continue my investigation but if this rang a bell to someone here, feel free to tell (this does
seem strange to me).

It likely means it is the path the build is made in that is being encoded into the build somehow. The same binary is resulting for a given build path?

Quoted 4 lines

Cheers,

Richard

I issued "strings" and "hexdump -C" on the libraries and didn't see any build path (the remap-path-prefix in rust-common.bbclass do it's job and I only saw /usr/src/debug with the rust source file of zvariant).

But maybe the build path is leaking anyway in some kind of binary format?

Quoted 1 line