Skip to contentSkip to conversation

Provide list of deployment artifacts

PATCH Superseded Ed Bartosh 3 participants 26 messages Aug 30, 2016

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

cover Provide list of deployment artifacts

Hi,

This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced

The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.

To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.

To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate if SSTATE_SKIP_CREATION is set.

The following changes since commit 087c580b286816265f487e02746bfa6e26081554:

init-install: Fixes the install script failing when not finding any mmcblk devices (2016-08-30 07:57:50 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/oe-core/artifacts-9869
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/artifacts-9869

Ed Bartosh (18):
image-live.bbclass: deploy images to DEPLOYDIR image-vm.bbclass: deploy images to DEPLOYDIR image.bbclass: deploy images to DEPLOYDIR image.bbclass: deploy images to DEPLOYDIR image_types_uboot.bbclass: deploy images to DEPLOYDIR syslinux.bbclass: deploy bootloader to DEPLOYDIR
build-appliance-image: process images in DEPLOYDIR populate_sdk_base.bbclass: deploy sdk artifacts to DEPLOYDIR rootfs-postcommands.bbclass: generate manifest in DEPLOYDIR
selftest: renamed variable
rootfs.py: use DEPLOYDIR instead of DEPLOY_DIR_IMAGE image.bbclass: put image_complete under sstate control image.bbclass: cleanup DEPLOYDIR
populate_sdk_base: put populate_sdk under sstate control sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set
image: populate_sdk_base: skip sstate creation
image: populate_sdk_base: set stamp-extra-info flag
toaster: fire TaskArtifacts event

 meta/classes/image-live.bbclass                    | 12 +++---
 meta/classes/image-vm.bbclass                      | 22 +++++------
 meta/classes/image.bbclass                         | 14 +++++--
 meta/classes/image_types.bbclass                   | 44 +++++++++++-----------
 meta/classes/image_types_uboot.bbclass             |  2 +-
 meta/classes/populate_sdk_base.bbclass             | 28 +++++++++-----
 meta/classes/rootfs-postcommands.bbclass           |  4 +-
 meta/classes/sstate.bbclass                        |  2 +
 meta/classes/syslinux.bbclass                      |  2 +-
 meta/classes/toaster.bbclass                       | 17 +++++++++
 meta/lib/oe/rootfs.py                              |  6 +--
 meta/lib/oeqa/selftest/imagefeatures.py            |  4 +-
 .../images/build-appliance-image_15.0.0.bb         |  8 ++--
 13 files changed, 100 insertions(+), 65 deletions(-)

--
Regards,
Ed

author
#1 lore raw

PATCH 1/18 image-live.bbclass: deploy images to DEPLOYDIR

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.

---
 meta/classes/image-live.bbclass | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
meta/classes/image-live.bbclass +6 -6
index f0e6647..1294b11 100644
@@ -43,7 +43,7 @@ ROOT_LIVE ?= "root=/dev/ram0"
 INITRD_IMAGE_LIVE ?= "core-image-minimal-initramfs"
 INITRD_LIVE ?= "${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_LIVE}-${MACHINE}.cpio.gz"
 
-ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.ext4"
+ROOTFS ?= "${DEPLOYDIR}/${IMAGE_LINK_NAME}.ext4"
 
 IMAGE_TYPEDEP_live = "ext4"
 IMAGE_TYPEDEP_iso = "ext4"
@@ -144,14 +144,14 @@ build_iso() {
 	if [ "${PCBIOS}" = "1" ] && [ "${EFI}" != "1" ] ; then
 		# PCBIOS only media
 		mkisofs -V ${BOOTIMG_VOLUME_ID} \
-		        -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso \
+		        -o ${DEPLOYDIR}/${IMAGE_NAME}.iso \
 			-b ${ISO_BOOTIMG} -c ${ISO_BOOTCAT} \
 			$mkisofs_compress_opts \
 			${MKISOFS_OPTIONS} $mkisofs_iso_level ${ISODIR}
 	else
 		# EFI only OR EFI+PCBIOS
 		mkisofs -A ${BOOTIMG_VOLUME_ID} -V ${BOOTIMG_VOLUME_ID} \
-		        -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso \
+		        -o ${DEPLOYDIR}/${IMAGE_NAME}.iso \
 			-b ${ISO_BOOTIMG} -c ${ISO_BOOTCAT} \
 			$mkisofs_compress_opts ${MKISOFS_OPTIONS} $mkisofs_iso_level \
 			-eltorito-alt-boot -eltorito-platform efi \
@@ -160,7 +160,7 @@ build_iso() {
 		isohybrid_args="-u"
 	fi
 
-	isohybrid $isohybrid_args ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso
+	isohybrid $isohybrid_args ${DEPLOYDIR}/${IMAGE_NAME}.iso
 }
 
 build_fat_img() {
@@ -252,13 +252,13 @@ build_hddimg() {
 			fi
 		fi
 
-		build_fat_img ${HDDDIR} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
+		build_fat_img ${HDDDIR} ${DEPLOYDIR}/${IMAGE_NAME}.hddimg
 
 		if [ "${PCBIOS}" = "1" ]; then
 			syslinux_hddimg_install
 		fi
 
-		chmod 644 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
+		chmod 644 ${DEPLOYDIR}/${IMAGE_NAME}.hddimg
 	fi
 }
 
Signature · 1 line
2.1.4
author
#2 lore raw

PATCH 2/18 image-vm.bbclass: deploy images to DEPLOYDIR

In reply to Ed Bartosh

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.

---
 meta/classes/image-vm.bbclass | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)
meta/classes/image-vm.bbclass +11 -11
index bf57e2c..2a86b40 100644
@@ -33,14 +33,14 @@ IMAGE_TYPEDEP_hdddirect = "${VM_ROOTFS_TYPE}"
 IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect"
 
 VM_ROOTFS_TYPE ?= "ext4"
-ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}"
+ROOTFS ?= "${DEPLOYDIR}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}"
 
 # Used by bootloader
 LABELS_VM ?= "boot"
 ROOT_VM ?= "root=/dev/sda2"
 # Using an initramfs is optional. Enable it by setting INITRD_IMAGE_VM.
 INITRD_IMAGE_VM ?= ""
-INITRD_VM ?= "${@'${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}"
+INITRD_VM ?= "${@'${DEPLOYDIR}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}"
 do_bootdirectdisk[depends] += "${@'${INITRD_IMAGE_VM}:do_image_complete' if '${INITRD_IMAGE_VM}' else ''}"
 
 BOOTDD_VOLUME_ID   ?= "boot"
@@ -52,7 +52,7 @@ DISK_SIGNATURE[vardepsexclude] = "DISK_SIGNATURE_GENERATED"
 build_boot_dd() {
 	HDDDIR="${S}/hdd/boot"
 	HDDIMG="${S}/hdd.image"
-	IMAGE=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect
+	IMAGE=${DEPLOYDIR}/${IMAGE_NAME}.hdddirect
 
 	populate_kernel $HDDDIR
 
@@ -104,13 +104,13 @@ build_boot_dd() {
 	dd if=$HDDIMG of=$IMAGE conv=notrunc seek=1 bs=512
 	dd if=${ROOTFS} of=$IMAGE conv=notrunc seek=$OFFSET bs=512
 
-	cd ${DEPLOY_DIR_IMAGE}
+	cd ${DEPLOYDIR}
 
-	if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ]; then
-		rm -f $(readlink -f ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect)
+	if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ]; then
+		rm -f $(readlink -f ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect)
 	fi
 
-	ln -sf ${IMAGE_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect
+	ln -sf ${IMAGE_NAME}.hdddirect ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect
 } 
 
 python do_bootdirectdisk() {
@@ -145,13 +145,13 @@ DISK_SIGNATURE_GENERATED := "${@generate_disk_signature()}"
 
 run_qemu_img (){
     type="$1"
-    qemu-img convert -O $type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.$type
+    qemu-img convert -O $type ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOYDIR}/${IMAGE_NAME}.$type
 
-    if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type ]; then
-        rm -f $(readlink -f ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type)
+    if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type ]; then
+        rm -f $(readlink -f ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type)
     fi
 
-    ln -sf ${IMAGE_NAME}.$type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type
+    ln -sf ${IMAGE_NAME}.$type ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type
 }
 create_vmdk_image () {
     run_qemu_img vmdk
Signature · 1 line
2.1.4
author
#3 lore raw

PATCH 3/18 image.bbclass: deploy images to DEPLOYDIR

In reply to Ed Bartosh

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.

---
 meta/classes/image.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/image.bbclass +2 -2
index c06dee2..82a07d5 100644
@@ -440,7 +440,7 @@ python () {
             cmds.append("\t" + image_cmd)
         else:
             bb.fatal("No IMAGE_CMD defined for IMAGE_FSTYPES entry '%s' - possibly invalid type name or missing support class" % t)
-        cmds.append(localdata.expand("\tcd ${DEPLOY_DIR_IMAGE}"))
+        cmds.append(localdata.expand("\tcd ${DEPLOYDIR}"))
 
         # Since a copy of IMAGE_CMD_xxx will be inlined within do_image_xxx,
         # prevent a redundant copy of IMAGE_CMD_xxx being emitted as a function.
@@ -558,7 +558,7 @@ python set_image_size () {
 #
 python create_symlinks() {
 
-    deploy_dir = d.getVar('DEPLOY_DIR_IMAGE', True)
+    deploy_dir = d.getVar('DEPLOYDIR', True)
     img_name = d.getVar('IMAGE_NAME', True)
     link_name = d.getVar('IMAGE_LINK_NAME', True)
     manifest_name = d.getVar('IMAGE_MANIFEST', True)
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH 4/18 image.bbclass: deploy images to DEPLOYDIR

In reply to Ed Bartosh

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.

---
 meta/classes/image.bbclass       |  2 +-
 meta/classes/image_types.bbclass | 44 ++++++++++++++++++++--------------------
 2 files changed, 23 insertions(+), 23 deletions(-)
meta/classes/image.bbclass +1 -1
index 82a07d5..ea3792a7 100644
@@ -118,7 +118,7 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','RM_OLD_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'DEPLOYDIR']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
meta/classes/image_types.bbclass +22 -22

119 lines not on this page · raw message

Signature · 1 line
2.1.4
author
#5 lore raw

PATCH 5/18 image_types_uboot.bbclass: deploy images to DEPLOYDIR

In reply to Ed Bartosh

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.

---
 meta/classes/image_types_uboot.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types_uboot.bbclass +1 -1
index f72d9b2..b8a281e 100644
@@ -2,7 +2,7 @@ inherit image_types kernel-arch
 
 oe_mkimage () {
     mkimage -A ${UBOOT_ARCH} -O linux -T ramdisk -C $2 -n ${IMAGE_NAME} \
-        -d ${DEPLOY_DIR_IMAGE}/$1 ${DEPLOY_DIR_IMAGE}/$1.u-boot
+        -d ${DEPLOYDIR}/$1 ${DEPLOYDIR}/$1.u-boot
     if [ x$3 = x"clean" ]; then
         rm $1
     fi
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH 6/18 syslinux.bbclass: deploy bootloader to DEPLOYDIR

In reply to Ed Bartosh

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.

---
 meta/classes/syslinux.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/syslinux.bbclass +1 -1
index 35d9eed..fd4c32e 100644
@@ -72,7 +72,7 @@ syslinux_hddimg_populate() {
 }
 
 syslinux_hddimg_install() {
-	syslinux ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
+	syslinux ${DEPLOYDIR}/${IMAGE_NAME}.hddimg
 }
 
 syslinux_hdddirect_install() {
Signature · 1 line
2.1.4
author
#7 lore raw

PATCH 7/18 build-appliance-image: process images in DEPLOYDIR

In reply to Ed Bartosh

Updated path to images as deployment directory is changed from DEPLOY_DIR_IMAGE to DEPLOYDIR.

---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
meta/recipes-core/images/build-appliance-image_15.0.0.bb +4 -4
index dc621d6..f7d57a8 100644
@@ -32,7 +32,7 @@ BA_INCLUDE_SOURCES ??= "0"
 
 IMAGE_CMD_ext4_append () {
 	# We don't need to reserve much space for root, 0.5% is more than enough
-	tune2fs -m 0.5 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext4
+	tune2fs -m 0.5 ${DEPLOYDIR}/${IMAGE_NAME}.rootfs.ext4
 }
 
 fakeroot do_populate_poky_src () {
@@ -101,9 +101,9 @@ create_bundle_files () {
 	cd ${WORKDIR}
 	mkdir -p Yocto_Build_Appliance
 	cp *.vmx* Yocto_Build_Appliance
-	ln -sf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
-	zip -r ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance
-	ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance.zip 
+	ln -sf ${DEPLOYDIR}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
+	zip -r ${DEPLOYDIR}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance
+	ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOYDIR}/Yocto_Build_Appliance.zip
 }
 create_bundle_files[vardepsexclude] = "DATETIME"
 
Signature · 1 line
2.1.4
author
#8 lore raw

PATCH 8/18 populate_sdk_base.bbclass: deploy sdk artifacts to DEPLOYDIR

In reply to Ed Bartosh

Changed deployment directory from SDK_DEPLOY to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.

---
 meta/classes/populate_sdk_base.bbclass | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
meta/classes/populate_sdk_base.bbclass +12 -10
index 645a7f4..4d2bc9a 100644
@@ -26,6 +26,8 @@ SDK_DIR = "${WORKDIR}/sdk"
 SDK_OUTPUT = "${SDK_DIR}/image"
 SDK_DEPLOY = "${DEPLOY_DIR}/sdk"
 
+DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
+
 B_task-populate-sdk = "${SDK_DIR}"
 
 SDKTARGETSYSROOT = "${SDKPATH}/sysroots/${REAL_MULTIMACH_TARGET_SYS}"
@@ -58,8 +60,8 @@ SDK_RELOCATE_AFTER_INSTALL ?= "1"
 SDKEXTPATH ?= "~/${@d.getVar('DISTRO', True)}_sdk"
 SDK_TITLE ?= "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} SDK"
 
-SDK_TARGET_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.target.manifest"
-SDK_HOST_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.host.manifest"
+SDK_TARGET_MANIFEST = "${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.target.manifest"
+SDK_HOST_MANIFEST = "${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.host.manifest"
 python write_target_sdk_manifest () {
     from oe.sdk import sdk_list_installed_packages
     from oe.utils import format_pkg_list
@@ -180,14 +182,14 @@ SDKTAROPTS = "--owner=root --group=root"
 
 fakeroot tar_sdk() {
 	# Package it up
-	mkdir -p ${SDK_DEPLOY}
+	mkdir -p ${DEPLOYDIR}
 	cd ${SDK_OUTPUT}/${SDKPATH}
-	tar ${SDKTAROPTS} -cf - . | pixz > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
+	tar ${SDKTAROPTS} -cf - . | pixz > ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
 }
 
 fakeroot create_shar() {
 	# copy in the template shar extractor script
-	cp ${COREBASE}/meta/files/toolchain-shar-extract.sh ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+	cp ${COREBASE}/meta/files/toolchain-shar-extract.sh ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	rm -f ${T}/pre_install_command ${T}/post_install_command
 
@@ -203,7 +205,7 @@ ${SDK_POST_INSTALL_COMMAND}
 EOF
 	sed -i -e '/@SDK_PRE_INSTALL_COMMAND@/r ${T}/pre_install_command' \
 		-e '/@SDK_POST_INSTALL_COMMAND@/r ${T}/post_install_command' \
-		${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+		${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# substitute variables
 	sed -i -e 's#@SDK_ARCH@#${SDK_ARCH}#g' \
@@ -215,16 +217,16 @@ EOF
 		-e 's#@SDK_VERSION@#${SDK_VERSION}#g' \
 		-e '/@SDK_PRE_INSTALL_COMMAND@/d' \
 		-e '/@SDK_POST_INSTALL_COMMAND@/d' \
-		${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+		${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# add execution permission
-	chmod +x ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+	chmod +x ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# append the SDK tarball
-	cat ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+	cat ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# delete the old tarball, we don't need it anymore
-	rm ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
+	rm ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
 }
 
 populate_sdk_log_check() {
Signature · 1 line
2.1.4
author
#9 lore raw

PATCH 9/18 rootfs-postcommands.bbclass: generate manifest in DEPLOYDIR

In reply to Ed Bartosh

Updated manifest path as deployment directory is changed from DEPLOY_DIR_IMAGE to DEPLOYDIR.

---
 meta/classes/rootfs-postcommands.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/rootfs-postcommands.bbclass +2 -2
index b38b6a5..f8a63b0 100644
@@ -15,7 +15,7 @@ ROOTFS_POSTPROCESS_COMMAND += "rootfs_update_timestamp ; "
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", "read_only_rootfs_hook; ", "",d)}'
 
 # Write manifest
-IMAGE_MANIFEST = "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.manifest"
+IMAGE_MANIFEST = "${DEPLOYDIR}/${IMAGE_NAME}.rootfs.manifest"
 ROOTFS_POSTUNINSTALL_COMMAND =+ "write_image_manifest ; "
 # Set default postinst log file
 POSTINST_LOGFILE ?= "${localstatedir}/log/postinstall.log"
@@ -217,7 +217,7 @@ python write_image_manifest () {
     from oe.rootfs import image_list_installed_packages
     from oe.utils import format_pkg_list
 
-    deploy_dir = d.getVar('DEPLOY_DIR_IMAGE', True)
+    deploy_dir = d.getVar('DEPLOYDIR', True)
     link_name = d.getVar('IMAGE_LINK_NAME', True)
     manifest_name = d.getVar('IMAGE_MANIFEST', True)
 
Signature · 1 line
2.1.4
author
#10 lore raw

PATCH 10/18 selftest: renamed variable

In reply to Ed Bartosh

Renamed variable deploy_dir to deploy_dir_image to avoid confusion with DEPLOYDIR variable.

---
 meta/lib/oeqa/selftest/imagefeatures.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oeqa/selftest/imagefeatures.py +2 -2
index 08e382f..d015c49 100644
@@ -113,9 +113,9 @@ class ImageFeatures(oeSelfTest):
         image_name = 'core-image-minimal'
         bitbake(image_name)
 
-        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE')
         link_name = get_bb_var('IMAGE_LINK_NAME', image_name)
-        image_path = os.path.join(deploy_dir, "%s.ext4" % link_name)
+        image_path = os.path.join(deploy_dir_image, "%s.ext4" % link_name)
         bmap_path = "%s.bmap" % image_path
 
         # check if result image and bmap file are in deploy directory
Signature · 1 line
2.1.4
author
#11 lore raw

PATCH 11/18 rootfs.py: use DEPLOYDIR instead of DEPLOY_DIR_IMAGE

In reply to Ed Bartosh

Updated the code of Rootfs class to use DEPLOYDIR variable as it's now used as a new deployment destination.

---
 meta/lib/oe/rootfs.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/lib/oe/rootfs.py +3 -3
index 7c620e9..076bc33 100644
@@ -19,7 +19,7 @@ class Rootfs(object, metaclass=ABCMeta):
         self.d = d
         self.pm = None
         self.image_rootfs = self.d.getVar('IMAGE_ROOTFS', True)
-        self.deploy_dir_image = self.d.getVar('DEPLOY_DIR_IMAGE', True)
+        self.deploydir = self.d.getVar('DEPLOYDIR', True)
         self.progress_reporter = progress_reporter
 
         self.install_order = Manifest.INSTALL_ORDER
@@ -182,12 +182,12 @@ class Rootfs(object, metaclass=ABCMeta):
 
         bb.utils.mkdirhier(self.image_rootfs)
 
-        bb.utils.mkdirhier(self.deploy_dir_image)
+        bb.utils.mkdirhier(self.deploydir)
 
         shutil.copytree(postinst_intercepts_dir, intercepts_dir)
 
         shutil.copy(self.d.expand("${COREBASE}/meta/files/deploydir_readme.txt"),
-                    self.deploy_dir_image +
+                    self.deploydir +
                     "/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt")
 
         execute_pre_post_process(self.d, pre_process_cmds)
Signature · 1 line
2.1.4
author
#12 lore raw

PATCH 12/18 image.bbclass: put image_complete under sstate control

In reply to Ed Bartosh

Adding image_complete task should make sstate machinery to generate manifest for deployed images and do final deployment to DEPLOY_DIR_IMAGE.

---
 meta/classes/image.bbclass | 5 +++++
 1 file changed, 5 insertions(+)
meta/classes/image.bbclass +5 -0
index ea3792a7..3bfc328 100644
@@ -264,6 +264,9 @@ do_image[dirs] = "${TOPDIR}"
 do_image[umask] = "022"
 addtask do_image after do_rootfs before do_build
 
+DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
+SSTATETASKS += "do_image_complete"
+
 fakeroot python do_image_complete () {
     from oe.utils import execute_pre_post_process
 
@@ -273,6 +276,8 @@ fakeroot python do_image_complete () {
 }
 do_image_complete[dirs] = "${TOPDIR}"
 do_image_complete[umask] = "022"
+do_image_complete[sstate-inputdirs] = "${DEPLOYDIR}"
+do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 addtask do_image_complete after do_image before do_build
 
 # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
Signature · 1 line
2.1.4
author
#13 lore raw

PATCH 13/18 image.bbclass: cleanup DEPLOYDIR

In reply to Ed Bartosh

Make sure DEPLOYDIR doesn't contain images from past deployments to prevent them to be included into sstate manifests.

---
 meta/classes/image.bbclass | 1 +
 1 file changed, 1 insertion(+)
meta/classes/image.bbclass +1 -0
index 3bfc328..bbdfd8a 100644
@@ -262,6 +262,7 @@ fakeroot python do_image () {
 }
 do_image[dirs] = "${TOPDIR}"
 do_image[umask] = "022"
+do_image[cleandirs] = "${DEPLOYDIR}"
 addtask do_image after do_rootfs before do_build
 
 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
Signature · 1 line
2.1.4
author
#14 lore raw

PATCH 14/18 populate_sdk_base: put populate_sdk under sstate control

In reply to Ed Bartosh

Adding populate_sdk task to SSTATE_TASKS should make sstate machinery to generate manifest for deployed sdk artifacts and do final deployment to SDK_DEPLOY.

---
 meta/classes/populate_sdk_base.bbclass | 4 ++++
 1 file changed, 4 insertions(+)
meta/classes/populate_sdk_base.bbclass +4 -0
index 4d2bc9a..c1dac39 100644
@@ -27,6 +27,7 @@ SDK_OUTPUT = "${SDK_DIR}/image"
 SDK_DEPLOY = "${DEPLOY_DIR}/sdk"
 
 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
+SSTATETASKS += "do_populate_sdk"
 
 B_task-populate-sdk = "${SDK_DIR}"
 
@@ -118,6 +119,9 @@ fakeroot python do_populate_sdk() {
     populate_sdk(d)
 }
 
+do_populate_sdk[sstate-inputdirs] = "${DEPLOYDIR}"
+do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}"
+
 fakeroot create_sdk_files() {
 	cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
 
Signature · 1 line
2.1.4
author
#15 lore raw

PATCH 15/18 sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set

In reply to Ed Bartosh

SSTATE_SKIP_CREATION variable will be used to skip creation of sstate .tgz files. It makes sense for image creation tasks as tarring images and keeping them in sstate would consume a lot of disk space.

---
 meta/classes/sstate.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/sstate.bbclass +2 -0
index 2496928..0f0baeb 100644
@@ -566,6 +566,8 @@ def sstate_package(ss, d):
     for state in ss['dirs']:
         if not os.path.exists(state[1]):
             continue
+        if d.getVar('SSTATE_SKIP_CREATION', True) == '1':
+            continue
         srcbase = state[0].rstrip("/").rsplit('/', 1)[0]
         for walkroot, dirs, files in os.walk(state[1]):
             for file in files:
Signature · 1 line
2.1.4
author
#16 lore raw

PATCH 16/18 image: populate_sdk_base: skip sstate creation

In reply to Ed Bartosh

Set SSTATE_SKIP_CREATION variable to '1' for image_complete and populate_sdk tasks as we don't want to put images and sdk installer into sstate due to their big size. We added these tasks under sstate control only for two reasons:
- to generate sstate manifest
- to make final deployments

---
 meta/classes/image.bbclass             | 1 +
 meta/classes/populate_sdk_base.bbclass | 1 +
 2 files changed, 2 insertions(+)
meta/classes/image.bbclass +1 -0
index bbdfd8a..8e84b1b 100644
@@ -267,6 +267,7 @@ addtask do_image after do_rootfs before do_build
 
 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
 SSTATETASKS += "do_image_complete"
+SSTATE_SKIP_CREATION_task-image-complete = '1'
 
 fakeroot python do_image_complete () {
     from oe.utils import execute_pre_post_process
meta/classes/populate_sdk_base.bbclass +1 -0
index c1dac39..1914a3b 100644
@@ -28,6 +28,7 @@ SDK_DEPLOY = "${DEPLOY_DIR}/sdk"
 
 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
 SSTATETASKS += "do_populate_sdk"
+SSTATE_SKIP_CREATION_task-populate-sdk = '1'
 
 B_task-populate-sdk = "${SDK_DIR}"
 
Signature · 1 line
2.1.4
author
#17 lore raw

PATCH 17/18 image: populate_sdk_base: set stamp-extra-info flag

In reply to Ed Bartosh

This flag is used in the name of sstate manifest.

Setting it to predetermined value for image_complete and populate_sdk tasks should help to get correct manifest filenames when processing runQueueTask events.

---
 meta/classes/image.bbclass             | 1 +
 meta/classes/populate_sdk_base.bbclass | 1 +
 2 files changed, 2 insertions(+)
meta/classes/image.bbclass +1 -0
index 8e84b1b..49fe500 100644
@@ -280,6 +280,7 @@ do_image_complete[dirs] = "${TOPDIR}"
 do_image_complete[umask] = "022"
 do_image_complete[sstate-inputdirs] = "${DEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_image_complete[stamp-extra-info] = "${MACHINE}"
 addtask do_image_complete after do_image before do_build
 
 # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
meta/classes/populate_sdk_base.bbclass +1 -0
index 1914a3b..ede3a7b 100644
@@ -122,6 +122,7 @@ fakeroot python do_populate_sdk() {
 
 do_populate_sdk[sstate-inputdirs] = "${DEPLOYDIR}"
 do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}"
+do_populate_sdk[stamp-extra-info] = "${MACHINE}"
 
 fakeroot create_sdk_files() {
 	cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
Signature · 1 line
2.1.4
author
#18 lore raw

PATCH 18/18 toaster: fire TaskArtifacts event

In reply to Ed Bartosh

Fire TaskArtifact MetaData event for deployment tasks when task either completed or skipped. Event contains full task id (recipe+task) and list of deployment artifacts from sstate manifest.

This should allow Toaster to always get notified about deployment artifacts produced by the build.

[YOCTO #9869]

---
 meta/classes/toaster.bbclass | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
meta/classes/toaster.bbclass +17 -0
index 972efb9..4bddf34 100644
@@ -324,6 +324,20 @@ python toaster_buildhistory_dump() {
 
 }
 
+# get list of artifacts from sstate manifest
+python toaster_artifacts() {
+    if e.taskname in ["do_deploy", "do_image_complete", "do_populate_sdk", "do_populate_sdk_ext"]:
+        d2 = d.createCopy()
+        d2.setVar('FILE', e.taskfile)
+        d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
+        manifest = oe.sstatesig.sstate_get_manifest_filename(e.taskname[3:], d2)[0]
+        if os.access(manifest, os.R_OK):
+            with open(manifest) as fmanifest:
+                artifacts = [fname.strip() for fname in fmanifest]
+                data = {"task": e.taskid, "artifacts": artifacts}
+                bb.event.fire(bb.event.MetadataEvent("TaskArtifacts", data), d2)
+}
+
 # set event handlers
 addhandler toaster_layerinfo_dumpdata
 toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -334,6 +348,9 @@ toaster_collect_task_stats[eventmask] = "bb.event.BuildCompleted bb.build.TaskSu
 addhandler toaster_buildhistory_dump
 toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
 
+addhandler toaster_artifacts
+toaster_artifacts[eventmask] = "bb.runqueue.runQueueTaskSkipped bb.runqueue.runQueueTaskCompleted"
+
 do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata "
 do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
 
Signature · 1 line
2.1.4
#19 lore raw
In reply to Ed Bartosh

Hi Ed,

Quoted 10 lines · Ed Bartosh

This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced

The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is
obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to
handle artifacts in simple way and remove a lot of current Toaster code doing guess work.

Quoted 8 lines

To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.

To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been
set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files
to sstate if SSTATE_SKIP_CREATION is set.

I'd like to see this series reordered to:

a) Set DEPLOYDIR in image.bbclass/populate_sdk_base.bbclass to point at DEPLOY_DIR_IMAGE
b) Add SSTATE_SKIP_CREATION support
c) Make all the DEPLOY_DIR_IMAGE -> DEPLOYDIR changes in one patch
d) Enable sstate for images (change DEPLOYDIR definition, add the flags including cleandirs, stamp-extra, set SSTATE_SKIP_CREATION)
e) As per d but for populate_sdk_base

That way it applies incrementally and like changes are all grouped together. It will mean some commit messages are longer and squashed together but I think that is fine.

Cheers,

Richard

#20 lore raw
In reply to Ed Bartosh
Quoted 8 lines · Ed Bartosh

Hi,

This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced

The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.

Quoted 5 lines

To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.

To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate if SSTATE_SKIP_CREATION is set.

SKIP creation of what? Variable name seems a little vague for me. Even needing the manifest, does it makes sense to store the image and SDK on sstate?

Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
author
#21 lore raw
Quoted 9 lines · Otavio Salvador
Quoted 8 lines · Ed Bartosh

Hi,

This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced

The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.

Quoted 7 lines
Quoted 5 lines

To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.

To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate if SSTATE_SKIP_CREATION is set.

SKIP creation of what?

SKIP creation of SSTATE entity.

Quoted 3 lines

Variable name seems a little vague for me. Even
needing the manifest, does it makes sense to store the image and SDK on sstate?

It doesn't and skipping this is a purpose of introducing SSTATE_SKIP_CREATION variable.

--
Regards,
Ed

#22 lore raw
Quoted 10 lines · Ed Bartosh
Quoted 9 lines · Otavio Salvador
Quoted 8 lines · Ed Bartosh

Hi,

This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced

The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.

To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate if SSTATE_SKIP_CREATION is set.

SKIP creation of what?

SKIP creation of SSTATE entity.

I know but of WHAT?

SSTATE_SKIP_IMAGE_CREATION would be a clearer name for me.

Quoted 4 lines
Quoted 3 lines

Variable name seems a little vague for me. Even
needing the manifest, does it makes sense to store the image and SDK on sstate?

It doesn't and skipping this is a purpose of introducing SSTATE_SKIP_CREATION variable.

So it is always going to be skipped?

Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
author
#23 lore raw
Quoted 13 lines · Otavio Salvador
Quoted 12 lines · Ed Bartosh
Quoted 11 lines · Otavio Salvador
Quoted 10 lines · Ed Bartosh

Hi,

This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced

The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.

To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate if SSTATE_SKIP_CREATION is set.

SKIP creation of what?

SKIP creation of SSTATE entity.

I know but of WHAT?

Of anything we don't want to put to sstate. In this patchset it's set for do_image_complete and do_populate_sdk, so it's for images and sdk installer.

Quoted 1 line

SSTATE_SKIP_IMAGE_CREATION would be a clearer name for me.

I'm not sure about it. I'd need one more variable
SSTATE_SKIP_SDK_INSTALLER if I go this way.

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

Variable name seems a little vague for me. Even
needing the manifest, does it makes sense to store the image and SDK on sstate?

It doesn't and skipping this is a purpose of introducing SSTATE_SKIP_CREATION variable.

So it is always going to be skipped?

Yes it is for do_image_complete and do_populate_sdk tasks for now.

--
Regards,
Ed

#24 lore raw
Quoted 2 lines · Ed Bartosh
Quoted 1 line

So it is always going to be skipped?

Yes it is for do_image_complete and do_populate_sdk tasks for now.

Oh ok.

Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
#25 lore raw
Quoted 6 lines · Otavio Salvador
Quoted 2 lines · Ed Bartosh
Quoted 1 line

So it is always going to be skipped?

Yes it is for do_image_complete and do_populate_sdk tasks for now.

Oh ok.

Its a performance issue. I can see the attraction of putting these into sstate however it would add significant time to the builds which many people would find unacceptable.

Having manifests of deployed files is a huge win for a whole variety of reasons and it standardises our output tasks. This compromise gives us that without the performance hit of compressing everything into an sstate artefact.

Cheers,

Richard