classes/reproducible_build: Move SDE deploy to another directory
PATCH 1/1 classes/reproducible_build: Move SDE deploy to another directory
The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the do_deploy_source_date_epoch task. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.
The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.
[YOCTO #13501]
--- meta/classes/reproducible_build.bbclass | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
meta/classes/reproducible_build.bbclass +6 -2
@@ -39,19 +39,23 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr SDE_DIR ="${WORKDIR}/source-date-epoch" SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt" +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch" SSTATETASKS += "do_deploy_source_date_epoch" do_deploy_source_date_epoch () { echo "Deploying SDE to ${SDE_DIR}." + mkdir -p ${SDE_DEPLOYDIR} + cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt } python do_deploy_source_date_epoch_setscene () { sstate_setscene(d) + os.rename(os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt'), d.getVar('SDE_FILE')) } -do_deploy_source_date_epoch[dirs] = "${SDE_DIR}" -do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}" +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}" addtask do_deploy_source_date_epoch_setscene addtask do_deploy_source_date_epoch before do_configure after do_patch
Signature · 1 line
2.21.0
PATCH v2 1/1 classes/reproducible_build: Move SDE deploy to another directory
The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the do_deploy_source_date_epoch task. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.
The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.
[YOCTO #13501]
--- meta/classes/reproducible_build.bbclass | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
meta/classes/reproducible_build.bbclass +10 -2
@@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr SDE_DIR ="${WORKDIR}/source-date-epoch" SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt" +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch" SSTATETASKS += "do_deploy_source_date_epoch" do_deploy_source_date_epoch () { echo "Deploying SDE to ${SDE_DIR}." + mkdir -p ${SDE_DEPLOYDIR} + if [ -e ${SDE_FILE} ]; then + cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt + fi } python do_deploy_source_date_epoch_setscene () { sstate_setscene(d) + sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt') + if os.path.exists(sde_file): + os.rename(sde_file, d.getVar('SDE_FILE')) } -do_deploy_source_date_epoch[dirs] = "${SDE_DIR}" -do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}" +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}" addtask do_deploy_source_date_epoch_setscene addtask do_deploy_source_date_epoch before do_configure after do_patch
Signature · 1 line
2.21.0
Quoted 46 lines · Joshua Watt
The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the
do_deploy_source_date_epochtask. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.
[YOCTO #13501]
Signed-off-by Joshua Watt <[email protected]>--- meta/classes/reproducible_build.bbclass | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)meta/classes/reproducible_build.bbclass +7
-0@@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr SDE_DIR ="${WORKDIR}/source-date-epoch" SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt" +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch" SSTATETASKS += "do_deploy_source_date_epoch" do_deploy_source_date_epoch () { echo "Deploying SDE to ${SDE_DIR}." + mkdir -p ${SDE_DEPLOYDIR} + if [ -e ${SDE_FILE} ]; then + cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt + fi } python do_deploy_source_date_epoch_setscene () { sstate_setscene(d) + sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt') + if os.path.exists(sde_file):
meta/classes/reproducible_build.bbclass inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr · line 65
Quoted 10 lines
meta/classes/reproducible_build.bbclass +3
-2+ os.rename(sde_file, d.getVar('SDE_FILE')) } -do_deploy_source_date_epoch[dirs] = "${SDE_DIR}" -do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}" +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}" addtask do_deploy_source_date_epoch_setscene addtask do_deploy_source_date_epoch before do_configure after do_patch
This seems to have broken builds with reproducible builds enabled and sstate downloaded from a mirror - I just get a blizzard of:
ERROR: Logfile of failure stored in:/home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: autoconf-native-2.69-r11 do_deploy_source_date_epoch_setscene:
Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_deploy_source_date_epoch_setscene(d)
0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
0054:python do_deploy_source_date_epoch_setscene () {
0055: sstate_setscene(d)
0056: sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
0057: if os.path.exists(sde_file):
*** 0058: os.rename(sde_file, d.getVar('SDE_FILE'))
0059:}
0060:
0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'
ERROR: Logfile of failure stored in:/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/temp/log.do_deploy_source_date_epoch_setscene.113869
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/autoconf/autoconf_2.69.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: m4-native-1.4.18-r0 do_deploy_source_date_epoch_setscene: Error
executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_deploy_source_date_epoch_setscene(d)
0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
0054:python do_deploy_source_date_epoch_setscene () {
0055: sstate_setscene(d)
0056: sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
0057: if os.path.exists(sde_file):
*** 0058: os.rename(sde_file, d.getVar('SDE_FILE'))
0059:}
0060:
0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/source-date-epoch/__source_date_epoch.txt'
ERROR: Logfile of failure stored in:/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/temp/log.do_deploy_source_date_epoch_setscene.113897
Signature · 1 line
Alex Kiernan
Quoted 84 lines · Alex Kiernan
Quoted 45 lines · Joshua Watt
The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the
do_deploy_source_date_epochtask. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.
[YOCTO #13501]
Signed-off-by Joshua Watt <[email protected]>--- meta/classes/reproducible_build.bbclass | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)meta/classes/reproducible_build.bbclass +7
-0@@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr SDE_DIR ="${WORKDIR}/source-date-epoch" SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt" +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch" SSTATETASKS += "do_deploy_source_date_epoch" do_deploy_source_date_epoch () { echo "Deploying SDE to ${SDE_DIR}." + mkdir -p ${SDE_DEPLOYDIR} + if [ -e ${SDE_FILE} ]; then + cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt + fi } python do_deploy_source_date_epoch_setscene () { sstate_setscene(d) + sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt') + if os.path.exists(sde_file):meta/classes/reproducible_build.bbclass inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr · line 65
Quoted 10 lines
meta/classes/reproducible_build.bbclass +3
-2+ os.rename(sde_file, d.getVar('SDE_FILE')) } -do_deploy_source_date_epoch[dirs] = "${SDE_DIR}" -do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}" +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}" addtask do_deploy_source_date_epoch_setscene addtask do_deploy_source_date_epoch before do_configure after do_patchThis seems to have broken builds with reproducible builds enabled and sstate downloaded from a mirror - I just get a blizzard of:
ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: autoconf-native-2.69-r11do_deploy_source_date_epoch_setscene:
Error executing a python function inexec_python_func()autogenerated:The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func()autogenerated', lineno: 2, function: <module>0001: *** 0002:do_deploy_source_date_epoch_setscene(d) 0003: File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass', lineno: 58, function: do_deploy_source_date_epoch_setscene 0054:python do_deploy_source_date_epoch_setscene () { 0055: sstate_setscene(d) 0056: sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt') 0057: if os.path.exists(sde_file): *** 0058: os.rename(sde_file, d.getVar('SDE_FILE')) 0059:} 0060: 0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" 0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
Quoted 3 lines
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'
Sorry about that. It looks like destination directory isn't getting created. You can try giving http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/hash-equivalence&id=7177bb4c4c174f0852b984b9b2d2d23de92e70cd a try to see if it resolves the issue.
Quoted 34 lines
ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/temp/log.do_deploy_source_date_epoch_setscene.113869
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/autoconf/autoconf_2.69.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: m4-native-1.4.18-r0do_deploy_source_date_epoch_setscene: Error
executing a python function inexec_python_func()autogenerated:The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func()autogenerated', lineno: 2, function: <module>0001: *** 0002:do_deploy_source_date_epoch_setscene(d) 0003: File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass', lineno: 58, function: do_deploy_source_date_epoch_setscene 0054:python do_deploy_source_date_epoch_setscene () { 0055: sstate_setscene(d) 0056: sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt') 0057: if os.path.exists(sde_file): *** 0058: os.rename(sde_file, d.getVar('SDE_FILE')) 0059:} 0060: 0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" 0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/source-date-epoch/__source_date_epoch.txt'ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/temp/log.do_deploy_source_date_epoch_setscene.113897
Quoted 91 lines · Joshua Watt
Quoted 84 lines · Alex Kiernan
Quoted 45 lines · Joshua Watt
The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the
do_deploy_source_date_epochtask. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.
[YOCTO #13501]
Signed-off-by Joshua Watt <[email protected]>--- meta/classes/reproducible_build.bbclass | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)meta/classes/reproducible_build.bbclass +7
-0@@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr SDE_DIR ="${WORKDIR}/source-date-epoch" SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt" +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch" SSTATETASKS += "do_deploy_source_date_epoch" do_deploy_source_date_epoch () { echo "Deploying SDE to ${SDE_DIR}." + mkdir -p ${SDE_DEPLOYDIR} + if [ -e ${SDE_FILE} ]; then + cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt + fi } python do_deploy_source_date_epoch_setscene () { sstate_setscene(d) + sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt') + if os.path.exists(sde_file):meta/classes/reproducible_build.bbclass inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr · line 65
Quoted 10 lines
meta/classes/reproducible_build.bbclass +3
-2+ os.rename(sde_file, d.getVar('SDE_FILE')) } -do_deploy_source_date_epoch[dirs] = "${SDE_DIR}" -do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}" +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}" addtask do_deploy_source_date_epoch_setscene addtask do_deploy_source_date_epoch before do_configure after do_patchThis seems to have broken builds with reproducible builds enabled and sstate downloaded from a mirror - I just get a blizzard of:
ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: autoconf-native-2.69-r11do_deploy_source_date_epoch_setscene:
Error executing a python function inexec_python_func()autogenerated:The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func()autogenerated', lineno: 2, function: <module>0001: *** 0002:do_deploy_source_date_epoch_setscene(d) 0003: File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass', lineno: 58, function: do_deploy_source_date_epoch_setscene 0054:python do_deploy_source_date_epoch_setscene () { 0055: sstate_setscene(d) 0056: sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt') 0057: if os.path.exists(sde_file): *** 0058: os.rename(sde_file, d.getVar('SDE_FILE')) 0059:} 0060: 0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" 0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"Quoted 3 lines
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'
Quoted 5 lines
Sorry about that. It looks like destination directory isn't getting created. You can try giving
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/hash-equivalence&id=7177bb4c4c174f0852b984b9b2d2d23de92e70cd
a try to see if it resolves the issue.
LGTM.
Signature · 1 line
Alex Kiernan