Skip to contentSkip to conversation

Overriding SDE_FILE

DISCUSSION Jacob Kroon 3 participants 14 messages Feb 24, 2020 – Feb 28, 2020

lore
Share & tools
lore
#0 lore raw

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Review

Quoted 5 lines
(quoted diff) +4 -1
-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

do_create_source_date_epoch_stamp() {
    mkdir -p ${SDE_DIR}
    date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

Cheers,
Jacob

#1 lore raw
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
    mkdir -p ${SDE_DIR}
    date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

#2 lore raw
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.
* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

#3 lore raw
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

*

* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

#4 lore raw
Quoted 63 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 4 lines

*

Quoted 6 lines
Quoted 4 lines

* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

#5 lore raw
Quoted 62 lines · Joshua Watt
Quoted 61 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 4 lines

*

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

#6 lore raw
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

*

* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.

Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd

That's really weird, and I'm not sure I like it, but worth a thought.

#7 lore raw
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

*

* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

Quoted 8 lines

Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.

Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd

That's really weird, and I'm not sure I like it, but worth a thought.

If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe.

How about ...

   BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not
   source_date_epoch_var(d) else ''}"
#8 lore raw
Quoted 90 lines · Douglas Royds
Quoted 87 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

*

* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.

Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd

Quoted 9 lines
Quoted 2 lines

That's really weird, and I'm not sure I like it, but worth a thought.

If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe.

How about ...

    BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not
    source_date_epoch_var(d) else ''}"

Ya, that works. The only reason I chose "is None" was in case a user really wanted to do

  SOURCE_DATE_EPOCH = "0"

#9 lore raw
Quoted 96 lines · Joshua Watt
Quoted 94 lines · Douglas Royds
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE. +#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

*

* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

Quoted 8 lines

Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.

Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd

That's really weird, and I'm not sure I like it, but worth a thought.

If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe.

Quoted 9 lines
Quoted 5 lines

How about ...

    BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not
    source_date_epoch_var(d) else ''}"

Ya, that works. The only reason I chose "is None" was in case a user really wanted to do

  SOURCE_DATE_EPOCH = "0"

True, best to allow for that.

#10 lore raw
Quoted 100 lines · Douglas Royds via Openembedded-core
Quoted 98 lines · Joshua Watt
Quoted 96 lines · Douglas Royds
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE. +#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

*

* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

Quoted 8 lines

Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.

Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd

That's really weird, and I'm not sure I like it, but worth a thought.

If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe.

How about ...

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
    BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not
    source_date_epoch_var(d) else ''}"

Ya, that works. The only reason I chose "is None" was in case a user really wanted to do

  SOURCE_DATE_EPOCH = "0"

True, best to allow for that.

Joshua, I just tried the two top patches in your reproducible branch; I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason..

#11 lore raw
Quoted 104 lines · Jacob Kroon
Quoted 103 lines · Douglas Royds via Openembedded-core
Quoted 101 lines · Joshua Watt
Quoted 99 lines · Douglas Royds
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE. +#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

  * Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

 *

  * As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

Quoted 8 lines

Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.

Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd

That's really weird, and I'm not sure I like it, but worth a thought.

If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe.

How about ...

    BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not source_date_epoch_var(d) else ''}"

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Ya, that works. The only reason I chose "is None" was in case a user really wanted to do

  SOURCE_DATE_EPOCH = "0"

True, best to allow for that.

Joshua, I just tried the two top patches in your reproducible branch; I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason..

In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} in my do_install() task.

#12 lore raw
Quoted 107 lines · Jacob Kroon
Quoted 106 lines · Jacob Kroon
Quoted 105 lines · Douglas Royds via Openembedded-core
Quoted 103 lines · Joshua Watt
Quoted 90 lines · Douglas Royds
Quoted 87 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

Quoted 9 lines

-# Once the value of SOURCE_DATE_EPOCH is determined, it
is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it
is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the
do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the
appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

Quoted 7 lines

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

  * Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

 *

  * As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.

Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd

Quoted 9 lines
Quoted 2 lines

That's really weird, and I'm not sure I like it, but worth a thought.

If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe.

How about ...

    BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not source_date_epoch_var(d) else ''}"

Ya, that works. The only reason I chose "is None" was in case a user really wanted to do

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

  SOURCE_DATE_EPOCH = "0"

True, best to allow for that.

Joshua, I just tried the two top patches in your reproducible branch; I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason..

In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} in my do_install() task.

I don't think it matters. My understanding is that an export variable (like SOURCE_DATE_EPOCH) is included in the taskhash of all tasks unless it is specifically whitelisted. It's possible there is some bug I haven't throughly tested it yet.... or it just can't work that way at all for some reason. You might be able to dump the task signature for your do_install task to see if SOURCE_DATE_EPOCH is being included.

#13 lore raw
Quoted 114 lines · Joshua Watt
Quoted 112 lines · Jacob Kroon
Quoted 111 lines · Jacob Kroon
Quoted 109 lines · Douglas Royds via Openembedded-core
Quoted 96 lines · Joshua Watt
Quoted 94 lines · Douglas Royds
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 23 lines · Jacob Kroon

Hi Douglas,

You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:

-# Once the value of SOURCE_DATE_EPOCH is determined, it
is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it
is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the
do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the
appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#

But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:

do_create_source_date_epoch_stamp() {
     mkdir -p ${SDE_DIR}
     date -d "1981-03-03" "+%s" > ${SDE_FILE}
}

What is the intended way to achieve the thing I'm trying to do here ?

FYI, JPEW has a proposed patch here

Quoted 8 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce

for allowing a recipe to just set SOURCE_DATE_EPOCH directly.

But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?

/Jacob

Quoted 9 lines

On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.

Joshua's proposed patch looks promising:

  * Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.

Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:

 SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"

e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.

Quoted 8 lines

 *

  * As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.

After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.

True.

What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?

Quoted 8 lines

Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.

Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd

That's really weird, and I'm not sure I like it, but worth a thought.

If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe.

Quoted 9 lines
Quoted 5 lines

How about ...

    BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not source_date_epoch_var(d) else ''}"

Ya, that works. The only reason I chose "is None" was in case a user really wanted to do

  SOURCE_DATE_EPOCH = "0"

True, best to allow for that.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Joshua, I just tried the two top patches in your reproducible branch; I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason..

In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} in my do_install() task.

I don't think it matters. My understanding is that an export variable (like SOURCE_DATE_EPOCH) is included in the taskhash of all tasks unless it is specifically whitelisted. It's possible there is some bug I haven't throughly tested it yet.... or it just can't work that way at all for some reason. You might be able to dump the task signature for your do_install task to see if SOURCE_DATE_EPOCH is being included.

Using bitbake-dumpsig I can see that SOURCE_DATE_EPOCH is still in "basewhitelist:"