Skip to contentSkip to conversation

classes/reproducible_build: Move SDE deploy to another directory

PATCH Superseded Joshua Watt 2 participants 5 messages Sep 16, 2019 – Sep 27, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 classes/reproducible_build: Move SDE deploy to another directory

The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the do_deploy_source_date_epoch task. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.

The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.

[YOCTO #13501]

---
 meta/classes/reproducible_build.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
meta/classes/reproducible_build.bbclass +6 -2
index 8788ad7145c..4753d458b58 100644
@@ -39,19 +39,23 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr
 
 SDE_DIR ="${WORKDIR}/source-date-epoch"
 SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
+SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
 
 SSTATETASKS += "do_deploy_source_date_epoch"
 
 do_deploy_source_date_epoch () {
     echo "Deploying SDE to ${SDE_DIR}."
+    mkdir -p ${SDE_DEPLOYDIR}
+    cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt
 }
 
 python do_deploy_source_date_epoch_setscene () {
     sstate_setscene(d)
+    os.rename(os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt'), d.getVar('SDE_FILE'))
 }
 
-do_deploy_source_date_epoch[dirs] = "${SDE_DIR}"
-do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
+do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
+do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
 addtask do_deploy_source_date_epoch_setscene
 addtask do_deploy_source_date_epoch before do_configure after do_patch
 
Signature · 1 line
2.21.0
#1 lore raw

PATCH v2 1/1 classes/reproducible_build: Move SDE deploy to another directory

The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the do_deploy_source_date_epoch task. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.

The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.

[YOCTO #13501]

---
 meta/classes/reproducible_build.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
meta/classes/reproducible_build.bbclass +10 -2
index 8788ad7145c..99b749a9ee2 100644
@@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr
 
 SDE_DIR ="${WORKDIR}/source-date-epoch"
 SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
+SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
 
 SSTATETASKS += "do_deploy_source_date_epoch"
 
 do_deploy_source_date_epoch () {
     echo "Deploying SDE to ${SDE_DIR}."
+    mkdir -p ${SDE_DEPLOYDIR}
+    if [ -e ${SDE_FILE} ]; then
+        cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt
+    fi
 }
 
 python do_deploy_source_date_epoch_setscene () {
     sstate_setscene(d)
+    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
+    if os.path.exists(sde_file):
+        os.rename(sde_file, d.getVar('SDE_FILE'))
 }
 
-do_deploy_source_date_epoch[dirs] = "${SDE_DIR}"
-do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
+do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
+do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
 addtask do_deploy_source_date_epoch_setscene
 addtask do_deploy_source_date_epoch before do_configure after do_patch
 
Signature · 1 line
2.21.0
#2 lore raw
Quoted 46 lines · Joshua Watt

The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the do_deploy_source_date_epoch task. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.

The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.

[YOCTO #13501]

---
 meta/classes/reproducible_build.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
meta/classes/reproducible_build.bbclass +7 -0
index 8788ad7145c..99b749a9ee2 100644
@@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr
 
 SDE_DIR ="${WORKDIR}/source-date-epoch"
 SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
+SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
 
 SSTATETASKS += "do_deploy_source_date_epoch"
 
 do_deploy_source_date_epoch () {
     echo "Deploying SDE to ${SDE_DIR}."
+    mkdir -p ${SDE_DEPLOYDIR}
+    if [ -e ${SDE_FILE} ]; then
+        cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt
+    fi
 }
 
 python do_deploy_source_date_epoch_setscene () {
     sstate_setscene(d)
+    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
+    if os.path.exists(sde_file):

meta/classes/reproducible_build.bbclass inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr · line 65

Quoted 10 lines
meta/classes/reproducible_build.bbclass +3 -2
+        os.rename(sde_file, d.getVar('SDE_FILE'))
 }
 
-do_deploy_source_date_epoch[dirs] = "${SDE_DIR}"
-do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
+do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
+do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
 addtask do_deploy_source_date_epoch_setscene
 addtask do_deploy_source_date_epoch before do_configure after do_patch
 

This seems to have broken builds with reproducible builds enabled and sstate downloaded from a mirror - I just get a blizzard of:

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: autoconf-native-2.69-r11 do_deploy_source_date_epoch_setscene:
Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_deploy_source_date_epoch_setscene(d)
     0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
     0054:python do_deploy_source_date_epoch_setscene () {
     0055:    sstate_setscene(d)
     0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
     0057:    if os.path.exists(sde_file):
 *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
     0059:}
     0060:
     0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
     0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"

Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/temp/log.do_deploy_source_date_epoch_setscene.113869
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/autoconf/autoconf_2.69.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: m4-native-1.4.18-r0 do_deploy_source_date_epoch_setscene: Error
executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_deploy_source_date_epoch_setscene(d)
     0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
     0054:python do_deploy_source_date_epoch_setscene () {
     0055:    sstate_setscene(d)
     0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
     0057:    if os.path.exists(sde_file):
 *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
     0059:}
     0060:
     0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
     0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"

Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/source-date-epoch/__source_date_epoch.txt'

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/temp/log.do_deploy_source_date_epoch_setscene.113897

Signature · 1 line
Alex Kiernan
#3 lore raw
Quoted 84 lines · Alex Kiernan
Quoted 45 lines · Joshua Watt

The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the do_deploy_source_date_epoch task. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.

The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.

[YOCTO #13501]

---
  meta/classes/reproducible_build.bbclass | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
meta/classes/reproducible_build.bbclass +7 -0
index 8788ad7145c..99b749a9ee2 100644
@@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr
 
  SDE_DIR ="${WORKDIR}/source-date-epoch"
  SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
+SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
 
  SSTATETASKS += "do_deploy_source_date_epoch"
 
  do_deploy_source_date_epoch () {
      echo "Deploying SDE to ${SDE_DIR}."
+    mkdir -p ${SDE_DEPLOYDIR}
+    if [ -e ${SDE_FILE} ]; then
+        cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt
+    fi
  }
 
  python do_deploy_source_date_epoch_setscene () {
      sstate_setscene(d)
+    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
+    if os.path.exists(sde_file):

meta/classes/reproducible_build.bbclass inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr · line 65

Quoted 10 lines
meta/classes/reproducible_build.bbclass +3 -2
+        os.rename(sde_file, d.getVar('SDE_FILE'))
  }
 
-do_deploy_source_date_epoch[dirs] = "${SDE_DIR}"
-do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
+do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
+do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
  addtask do_deploy_source_date_epoch_setscene
  addtask do_deploy_source_date_epoch before do_configure after do_patch
 

This seems to have broken builds with reproducible builds enabled and sstate downloaded from a mirror - I just get a blizzard of:

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: autoconf-native-2.69-r11 do_deploy_source_date_epoch_setscene:
Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

      0001:
  *** 0002:do_deploy_source_date_epoch_setscene(d)
      0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
      0054:python do_deploy_source_date_epoch_setscene () {
      0055:    sstate_setscene(d)
      0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
      0057:    if os.path.exists(sde_file):
  *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
      0059:}
      0060:
      0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
      0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
Quoted 3 lines

Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'

Sorry about that. It looks like destination directory isn't getting created. You can try giving http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/hash-equivalence&id=7177bb4c4c174f0852b984b9b2d2d23de92e70cd a try to see if it resolves the issue.

Quoted 34 lines

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/temp/log.do_deploy_source_date_epoch_setscene.113869
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/autoconf/autoconf_2.69.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: m4-native-1.4.18-r0 do_deploy_source_date_epoch_setscene: Error
executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

      0001:
  *** 0002:do_deploy_source_date_epoch_setscene(d)
      0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
      0054:python do_deploy_source_date_epoch_setscene () {
      0055:    sstate_setscene(d)
      0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
      0057:    if os.path.exists(sde_file):
  *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
      0059:}
      0060:
      0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
      0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"

Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/source-date-epoch/__source_date_epoch.txt'

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/temp/log.do_deploy_source_date_epoch_setscene.113897

#4 lore raw
Quoted 91 lines · Joshua Watt
Quoted 84 lines · Alex Kiernan
Quoted 45 lines · Joshua Watt

The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the do_deploy_source_date_epoch task. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch.

The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted.

[YOCTO #13501]

---
  meta/classes/reproducible_build.bbclass | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
meta/classes/reproducible_build.bbclass +7 -0
index 8788ad7145c..99b749a9ee2 100644
@@ -39,19 +39,27 @@ inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr
 
  SDE_DIR ="${WORKDIR}/source-date-epoch"
  SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
+SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
 
  SSTATETASKS += "do_deploy_source_date_epoch"
 
  do_deploy_source_date_epoch () {
      echo "Deploying SDE to ${SDE_DIR}."
+    mkdir -p ${SDE_DEPLOYDIR}
+    if [ -e ${SDE_FILE} ]; then
+        cp -p ${SDE_FILE} ${SDE_DEPLOYDIR}/__source_date_epoch.txt
+    fi
  }
 
  python do_deploy_source_date_epoch_setscene () {
      sstate_setscene(d)
+    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
+    if os.path.exists(sde_file):

meta/classes/reproducible_build.bbclass inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'repr · line 65

Quoted 10 lines
meta/classes/reproducible_build.bbclass +3 -2
+        os.rename(sde_file, d.getVar('SDE_FILE'))
  }
 
-do_deploy_source_date_epoch[dirs] = "${SDE_DIR}"
-do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
+do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
+do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
  addtask do_deploy_source_date_epoch_setscene
  addtask do_deploy_source_date_epoch before do_configure after do_patch
 

This seems to have broken builds with reproducible builds enabled and sstate downloaded from a mirror - I just get a blizzard of:

ERROR: Logfile of failure stored in:
/home/akiernan/nanohub/build/tmp/work/x86_64-linux/automake-native/1.16.1-r0/temp/log.do_deploy_source_date_epoch_setscene.113823
WARNING: Setscene task (virtual:native:/home/akiernan/nanohub/build/../poky/meta/recipes-devtools/automake/automake_1.16.1.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
ERROR: autoconf-native-2.69-r11 do_deploy_source_date_epoch_setscene:
Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

      0001:
  *** 0002:do_deploy_source_date_epoch_setscene(d)
      0003:
File: '/home/akiernan/nanohub/build/../poky/meta/classes/reproducible_build.bbclass',
lineno: 58, function: do_deploy_source_date_epoch_setscene
      0054:python do_deploy_source_date_epoch_setscene () {
      0055:    sstate_setscene(d)
      0056:    sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'),
'__source_date_epoch.txt')
      0057:    if os.path.exists(sde_file):
  *** 0058:        os.rename(sde_file, d.getVar('SDE_FILE'))
      0059:}
      0060:
      0061:do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
      0062:do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
Quoted 3 lines

Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/deploy-source-date-epoch/__source_date_epoch.txt' -> '/home/akiernan/nanohub/build/tmp/work/x86_64-linux/autoconf-native/2.69-r11/source-date-epoch/__source_date_epoch.txt'

Quoted 5 lines

Sorry about that. It looks like destination directory isn't getting created. You can try giving
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/hash-equivalence&id=7177bb4c4c174f0852b984b9b2d2d23de92e70cd
a try to see if it resolves the issue.

LGTM.

Signature · 1 line
Alex Kiernan