Overriding SDE_FILE
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:
Review
Quoted 5 lines
(quoted diff) +4
-1-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}. +# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE. +# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task +# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE. +#
But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:
do_create_source_date_epoch_stamp() {
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
Cheers,
Jacob
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:
Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
for allowing a recipe to just set SOURCE_DATE_EPOCH directly.
But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ?
/Jacob
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest.
Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority.
* As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.
Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
Quoted 63 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 4 lines
*
Quoted 6 lines
Quoted 4 lines
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.
Quoted 62 lines · Joshua Watt
Quoted 61 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 4 lines
*
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.
True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED?
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg.
SOURCE_DATE_EPOCH_FIXED?
Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.
Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd
That's really weird, and I'm not sure I like it, but worth a thought.
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg.
SOURCE_DATE_EPOCH_FIXED?
Quoted 8 lines
Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.
Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd
That's really weird, and I'm not sure I like it, but worth a thought.
If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe.
How about ...
BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not
source_date_epoch_var(d) else ''}"Quoted 90 lines · Douglas Royds
Quoted 87 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg.
SOURCE_DATE_EPOCH_FIXED?Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.
Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd
Quoted 9 lines
Quoted 2 lines
That's really weird, and I'm not sure I like it, but worth a thought.
If it works, it's good. I like the fact that the user can just set
SOURCE_DATE_EPOCHdirectly in their recipe.How about ...
BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not source_date_epoch_var(d) else ''}"
Ya, that works. The only reason I chose "is None" was in case a user really wanted to do
SOURCE_DATE_EPOCH = "0"
Quoted 96 lines · Joshua Watt
Quoted 94 lines · Douglas Royds
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE. +#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg.
SOURCE_DATE_EPOCH_FIXED?Quoted 8 lines
Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.
Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd
That's really weird, and I'm not sure I like it, but worth a thought.
If it works, it's good. I like the fact that the user can just set
SOURCE_DATE_EPOCHdirectly in their recipe.
Quoted 9 lines
Quoted 5 lines
How about ...
BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not source_date_epoch_var(d) else ''}"Ya, that works. The only reason I chose "is None" was in case a user really wanted to do
SOURCE_DATE_EPOCH= "0"
True, best to allow for that.
Quoted 100 lines · Douglas Royds via Openembedded-core
Quoted 98 lines · Joshua Watt
Quoted 96 lines · Douglas Royds
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE. +#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg.
SOURCE_DATE_EPOCH_FIXED?Quoted 8 lines
Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.
Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd
That's really weird, and I'm not sure I like it, but worth a thought.
If it works, it's good. I like the fact that the user can just set
SOURCE_DATE_EPOCHdirectly in their recipe.How about ...
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not source_date_epoch_var(d) else ''}"Ya, that works. The only reason I chose "is None" was in case a user really wanted to do
SOURCE_DATE_EPOCH= "0"True, best to allow for that.
Joshua, I just tried the two top patches in your reproducible branch; I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason..
Quoted 104 lines · Jacob Kroon
Quoted 103 lines · Douglas Royds via Openembedded-core
Quoted 101 lines · Joshua Watt
Quoted 99 lines · Douglas Royds
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of
SOURCE_DATE_EPOCHis determined, it is stored in the recipe's ${SDE_FILE}.
+# Once the value ofSOURCE_DATE_EPOCHis determined, it is stored in the recipe'sSDE_FILE.
+# If none of these mechanisms are suitable, replace thedo_deploy_source_date_epochtask
+# with recipe-specific functionality to write the appropriateSOURCE_DATE_EPOCHinto theSDE_FILE. +#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg.
SOURCE_DATE_EPOCH_FIXED?Quoted 8 lines
Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.
Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd
That's really weird, and I'm not sure I like it, but worth a thought.
If it works, it's good. I like the fact that the user can just set
SOURCE_DATE_EPOCHdirectly in their recipe.How about ...
BB_HASHBASE_WHITELIST+= "${@'SOURCE_DATE_EPOCH' if notsource_date_epoch_var(d)else ''}"
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Ya, that works. The only reason I chose "is None" was in case a user really wanted to do
SOURCE_DATE_EPOCH= "0"True, best to allow for that.
Joshua, I just tried the two top patches in your reproducible branch; I can see that
SOURCE_DATE_EPOCHis not inBB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason..
In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} in my do_install() task.
Quoted 107 lines · Jacob Kroon
Quoted 106 lines · Jacob Kroon
Quoted 105 lines · Douglas Royds via Openembedded-core
Quoted 103 lines · Joshua Watt
Quoted 90 lines · Douglas Royds
Quoted 87 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 16 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:Quoted 9 lines
-# Once the value of SOURCE_DATE_EPOCH is determined, it
is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it
is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the
do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the
appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "
do_create_source_date_epoch_stamp()" in my recipe:Quoted 7 lines
do_create_source_date_epoch_stamp(){
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg.
SOURCE_DATE_EPOCH_FIXED?Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.
Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd
Quoted 9 lines
Quoted 2 lines
That's really weird, and I'm not sure I like it, but worth a thought.
If it works, it's good. I like the fact that the user can just set
SOURCE_DATE_EPOCHdirectly in their recipe.How about ...
BB_HASHBASE_WHITELIST+= "${@'SOURCE_DATE_EPOCH' if notsource_date_epoch_var(d)else ''}"Ya, that works. The only reason I chose "is None" was in case a user really wanted to do
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
SOURCE_DATE_EPOCH= "0"True, best to allow for that.
Joshua, I just tried the two top patches in your reproducible branch; I can see that
SOURCE_DATE_EPOCHis not inBB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason..In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} in my
do_install()task.
I don't think it matters. My understanding is that an export variable (like SOURCE_DATE_EPOCH) is included in the taskhash of all tasks unless it is specifically whitelisted. It's possible there is some bug I haven't throughly tested it yet.... or it just can't work that way at all for some reason. You might be able to dump the task signature for your do_install task to see if SOURCE_DATE_EPOCH is being included.
Quoted 114 lines · Joshua Watt
Quoted 112 lines · Jacob Kroon
Quoted 111 lines · Jacob Kroon
Quoted 109 lines · Douglas Royds via Openembedded-core
Quoted 96 lines · Joshua Watt
Quoted 94 lines · Douglas Royds
Quoted 81 lines · Joshua Watt
Quoted 69 lines · Douglas Royds
Quoted 66 lines · Joshua Watt
Quoted 65 lines · Joshua Watt
Quoted 40 lines · Douglas Royds
Quoted 27 lines · Jacob Kroon
Quoted 23 lines · Jacob Kroon
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit e7b891b76954c784f5a93bd0a1c91315673ce40d:
-# Once the value of SOURCE_DATE_EPOCH is determined, it
is stored in the recipe's ${SDE_FILE}.
+# Once the value of SOURCE_DATE_EPOCH is determined, it
is stored in the recipe's SDE_FILE.
+# If none of these mechanisms are suitable, replace the
do_deploy_source_date_epoch task
+# with recipe-specific functionality to write the
appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
+#But I can't really get this to work. What did work for me was to replace "do_create_source_date_epoch_stamp()" in my recipe:
do_create_source_date_epoch_stamp() {
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}What is the intended way to achieve the thing I'm trying to do here ?
FYI, JPEW has a proposed patch here
Quoted 8 lines
for allowing a recipe to just set
SOURCE_DATE_EPOCHdirectly.But maybe that currently is at odds with
SOURCE_DATE_EPOCHbeing inBB_HASHBASE_WHITELIST?/Jacob
Quoted 9 lines
On the surface of it, my comment appears to be just wrong: It does make sense to replace
do_create_source_date_epoch_stamp()as you suggest.Joshua's proposed patch looks promising:
* Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set
SOURCE_DATE_EPOCH, then they want it to take priority.Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like:
SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}"
e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare.
Quoted 8 lines
*
* As you observe,
SOURCE_DATE_EPOCHwould need to be removed fromBB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place.Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember.
After a discussion with Richard, we figured out why
SOURCE_DATE_EPOCHhas to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors.True.
What to do? Would it work to use a different non-whitelisted variable in the recipe, eg.
SOURCE_DATE_EPOCH_FIXED?Quoted 8 lines
Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible.
Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd
That's really weird, and I'm not sure I like it, but worth a thought.
If it works, it's good. I like the fact that the user can just set
SOURCE_DATE_EPOCHdirectly in their recipe.Quoted 9 lines
Quoted 5 lines
How about ...
BB_HASHBASE_WHITELIST+= "${@'SOURCE_DATE_EPOCH' if notsource_date_epoch_var(d)else ''}"Ya, that works. The only reason I chose "is None" was in case a user really wanted to do
SOURCE_DATE_EPOCH= "0"True, best to allow for that.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Joshua, I just tried the two top patches in your reproducible branch; I can see that
SOURCE_DATE_EPOCHis not inBB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason..In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} in my
do_install()task.I don't think it matters. My understanding is that an export variable (like
SOURCE_DATE_EPOCH) is included in the taskhash of all tasks unless it is specifically whitelisted. It's possible there is some bug I haven't throughly tested it yet.... or it just can't work that way at all for some reason. You might be able to dump the task signature for yourdo_installtask to see ifSOURCE_DATE_EPOCHis being included.
Using bitbake-dumpsig I can see that SOURCE_DATE_EPOCH is still in "basewhitelist:"