Provide list of deployment artifacts
cover Provide list of deployment artifacts
Hi,
This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.
To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate if SSTATE_SKIP_CREATION is set.
The following changes since commit 087c580b286816265f487e02746bfa6e26081554:
init-install: Fixes the install script failing when not finding any mmcblk devices (2016-08-30 07:57:50 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/oe-core/artifacts-9869
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/artifacts-9869
Ed Bartosh (18):
image-live.bbclass: deploy images to DEPLOYDIR image-vm.bbclass: deploy images to DEPLOYDIR image.bbclass: deploy images to DEPLOYDIR image.bbclass: deploy images to DEPLOYDIR image_types_uboot.bbclass: deploy images to DEPLOYDIR syslinux.bbclass: deploy bootloader to DEPLOYDIR
build-appliance-image: process images in DEPLOYDIR populate_sdk_base.bbclass: deploy sdk artifacts to DEPLOYDIR rootfs-postcommands.bbclass: generate manifest in DEPLOYDIR
selftest: renamed variable
rootfs.py: use DEPLOYDIR instead of DEPLOY_DIR_IMAGE image.bbclass: put image_complete under sstate control image.bbclass: cleanup DEPLOYDIR
populate_sdk_base: put populate_sdk under sstate control sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set
image: populate_sdk_base: skip sstate creation
image: populate_sdk_base: set stamp-extra-info flag
toaster: fire TaskArtifacts event
meta/classes/image-live.bbclass | 12 +++--- meta/classes/image-vm.bbclass | 22 +++++------ meta/classes/image.bbclass | 14 +++++-- meta/classes/image_types.bbclass | 44 +++++++++++----------- meta/classes/image_types_uboot.bbclass | 2 +- meta/classes/populate_sdk_base.bbclass | 28 +++++++++----- meta/classes/rootfs-postcommands.bbclass | 4 +- meta/classes/sstate.bbclass | 2 + meta/classes/syslinux.bbclass | 2 +- meta/classes/toaster.bbclass | 17 +++++++++ meta/lib/oe/rootfs.py | 6 +-- meta/lib/oeqa/selftest/imagefeatures.py | 4 +- .../images/build-appliance-image_15.0.0.bb | 8 ++-- 13 files changed, 100 insertions(+), 65 deletions(-)
--
Regards,
Ed
PATCH 1/18 image-live.bbclass: deploy images to DEPLOYDIR
Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.
--- meta/classes/image-live.bbclass | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
meta/classes/image-live.bbclass +6 -6
@@ -43,7 +43,7 @@ ROOT_LIVE ?= "root=/dev/ram0" INITRD_IMAGE_LIVE ?= "core-image-minimal-initramfs" INITRD_LIVE ?= "${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_LIVE}-${MACHINE}.cpio.gz" -ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.ext4" +ROOTFS ?= "${DEPLOYDIR}/${IMAGE_LINK_NAME}.ext4" IMAGE_TYPEDEP_live = "ext4" IMAGE_TYPEDEP_iso = "ext4"
@@ -144,14 +144,14 @@ build_iso() { if [ "${PCBIOS}" = "1" ] && [ "${EFI}" != "1" ] ; then # PCBIOS only media mkisofs -V ${BOOTIMG_VOLUME_ID} \ - -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso \ + -o ${DEPLOYDIR}/${IMAGE_NAME}.iso \ -b ${ISO_BOOTIMG} -c ${ISO_BOOTCAT} \ $mkisofs_compress_opts \ ${MKISOFS_OPTIONS} $mkisofs_iso_level ${ISODIR} else # EFI only OR EFI+PCBIOS mkisofs -A ${BOOTIMG_VOLUME_ID} -V ${BOOTIMG_VOLUME_ID} \ - -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso \ + -o ${DEPLOYDIR}/${IMAGE_NAME}.iso \ -b ${ISO_BOOTIMG} -c ${ISO_BOOTCAT} \ $mkisofs_compress_opts ${MKISOFS_OPTIONS} $mkisofs_iso_level \ -eltorito-alt-boot -eltorito-platform efi \
@@ -160,7 +160,7 @@ build_iso() { isohybrid_args="-u" fi - isohybrid $isohybrid_args ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso + isohybrid $isohybrid_args ${DEPLOYDIR}/${IMAGE_NAME}.iso } build_fat_img() {
@@ -252,13 +252,13 @@ build_hddimg() { fi fi - build_fat_img ${HDDDIR} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg + build_fat_img ${HDDDIR} ${DEPLOYDIR}/${IMAGE_NAME}.hddimg if [ "${PCBIOS}" = "1" ]; then syslinux_hddimg_install fi - chmod 644 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg + chmod 644 ${DEPLOYDIR}/${IMAGE_NAME}.hddimg fi }
Signature · 1 line
2.1.4
PATCH 2/18 image-vm.bbclass: deploy images to DEPLOYDIR
In reply to Ed BartoshChanged deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.
--- meta/classes/image-vm.bbclass | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
meta/classes/image-vm.bbclass +11 -11
@@ -33,14 +33,14 @@ IMAGE_TYPEDEP_hdddirect = "${VM_ROOTFS_TYPE}" IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect" VM_ROOTFS_TYPE ?= "ext4" -ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}" +ROOTFS ?= "${DEPLOYDIR}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}" # Used by bootloader LABELS_VM ?= "boot" ROOT_VM ?= "root=/dev/sda2" # Using an initramfs is optional. Enable it by setting INITRD_IMAGE_VM. INITRD_IMAGE_VM ?= "" -INITRD_VM ?= "${@'${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}" +INITRD_VM ?= "${@'${DEPLOYDIR}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}" do_bootdirectdisk[depends] += "${@'${INITRD_IMAGE_VM}:do_image_complete' if '${INITRD_IMAGE_VM}' else ''}" BOOTDD_VOLUME_ID ?= "boot"
@@ -52,7 +52,7 @@ DISK_SIGNATURE[vardepsexclude] = "DISK_SIGNATURE_GENERATED" build_boot_dd() { HDDDIR="${S}/hdd/boot" HDDIMG="${S}/hdd.image" - IMAGE=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect + IMAGE=${DEPLOYDIR}/${IMAGE_NAME}.hdddirect populate_kernel $HDDDIR
@@ -104,13 +104,13 @@ build_boot_dd() { dd if=$HDDIMG of=$IMAGE conv=notrunc seek=1 bs=512 dd if=${ROOTFS} of=$IMAGE conv=notrunc seek=$OFFSET bs=512 - cd ${DEPLOY_DIR_IMAGE} + cd ${DEPLOYDIR} - if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ]; then - rm -f $(readlink -f ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect) + if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ]; then + rm -f $(readlink -f ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect) fi - ln -sf ${IMAGE_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect + ln -sf ${IMAGE_NAME}.hdddirect ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect } python do_bootdirectdisk() {
@@ -145,13 +145,13 @@ DISK_SIGNATURE_GENERATED := "${@generate_disk_signature()}" run_qemu_img (){ type="$1" - qemu-img convert -O $type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.$type + qemu-img convert -O $type ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOYDIR}/${IMAGE_NAME}.$type - if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type ]; then - rm -f $(readlink -f ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type) + if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type ]; then + rm -f $(readlink -f ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type) fi - ln -sf ${IMAGE_NAME}.$type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type + ln -sf ${IMAGE_NAME}.$type ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type } create_vmdk_image () { run_qemu_img vmdk
Signature · 1 line
2.1.4
PATCH 3/18 image.bbclass: deploy images to DEPLOYDIR
In reply to Ed BartoshChanged deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.
--- meta/classes/image.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/image.bbclass +2 -2
@@ -440,7 +440,7 @@ python () { cmds.append("\t" + image_cmd) else: bb.fatal("No IMAGE_CMD defined for IMAGE_FSTYPES entry '%s' - possibly invalid type name or missing support class" % t) - cmds.append(localdata.expand("\tcd ${DEPLOY_DIR_IMAGE}")) + cmds.append(localdata.expand("\tcd ${DEPLOYDIR}")) # Since a copy of IMAGE_CMD_xxx will be inlined within do_image_xxx, # prevent a redundant copy of IMAGE_CMD_xxx being emitted as a function.
@@ -558,7 +558,7 @@ python set_image_size () { # python create_symlinks() { - deploy_dir = d.getVar('DEPLOY_DIR_IMAGE', True) + deploy_dir = d.getVar('DEPLOYDIR', True) img_name = d.getVar('IMAGE_NAME', True) link_name = d.getVar('IMAGE_LINK_NAME', True) manifest_name = d.getVar('IMAGE_MANIFEST', True)
Signature · 1 line
2.1.4
PATCH 4/18 image.bbclass: deploy images to DEPLOYDIR
In reply to Ed BartoshChanged deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.
--- meta/classes/image.bbclass | 2 +- meta/classes/image_types.bbclass | 44 ++++++++++++++++++++-------------------- 2 files changed, 23 insertions(+), 23 deletions(-)
meta/classes/image.bbclass +1 -1
@@ -118,7 +118,7 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','RM_OLD_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'DEPLOYDIR'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables)
meta/classes/image_types.bbclass +22 -22
119 lines not on this page · raw message
Signature · 1 line
2.1.4
PATCH 5/18 image_types_uboot.bbclass: deploy images to DEPLOYDIR
In reply to Ed BartoshChanged deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.
--- meta/classes/image_types_uboot.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types_uboot.bbclass +1 -1
@@ -2,7 +2,7 @@ inherit image_types kernel-arch oe_mkimage () { mkimage -A ${UBOOT_ARCH} -O linux -T ramdisk -C $2 -n ${IMAGE_NAME} \ - -d ${DEPLOY_DIR_IMAGE}/$1 ${DEPLOY_DIR_IMAGE}/$1.u-boot + -d ${DEPLOYDIR}/$1 ${DEPLOYDIR}/$1.u-boot if [ x$3 = x"clean" ]; then rm $1 fi
Signature · 1 line
2.1.4
PATCH 6/18 syslinux.bbclass: deploy bootloader to DEPLOYDIR
In reply to Ed BartoshChanged deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.
--- meta/classes/syslinux.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/syslinux.bbclass +1 -1
@@ -72,7 +72,7 @@ syslinux_hddimg_populate() { } syslinux_hddimg_install() { - syslinux ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg + syslinux ${DEPLOYDIR}/${IMAGE_NAME}.hddimg } syslinux_hdddirect_install() {
Signature · 1 line
2.1.4
PATCH 7/18 build-appliance-image: process images in DEPLOYDIR
In reply to Ed BartoshUpdated path to images as deployment directory is changed from DEPLOY_DIR_IMAGE to DEPLOYDIR.
--- meta/recipes-core/images/build-appliance-image_15.0.0.bb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
meta/recipes-core/images/build-appliance-image_15.0.0.bb +4 -4
@@ -32,7 +32,7 @@ BA_INCLUDE_SOURCES ??= "0" IMAGE_CMD_ext4_append () { # We don't need to reserve much space for root, 0.5% is more than enough - tune2fs -m 0.5 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext4 + tune2fs -m 0.5 ${DEPLOYDIR}/${IMAGE_NAME}.rootfs.ext4 } fakeroot do_populate_poky_src () {
@@ -101,9 +101,9 @@ create_bundle_files () { cd ${WORKDIR} mkdir -p Yocto_Build_Appliance cp *.vmx* Yocto_Build_Appliance - ln -sf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk - zip -r ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance - ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance.zip + ln -sf ${DEPLOYDIR}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk + zip -r ${DEPLOYDIR}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance + ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOYDIR}/Yocto_Build_Appliance.zip } create_bundle_files[vardepsexclude] = "DATETIME"
Signature · 1 line
2.1.4
PATCH 8/18 populate_sdk_base.bbclass: deploy sdk artifacts to DEPLOYDIR
In reply to Ed BartoshChanged deployment directory from SDK_DEPLOY to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.
--- meta/classes/populate_sdk_base.bbclass | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
meta/classes/populate_sdk_base.bbclass +12 -10
@@ -26,6 +26,8 @@ SDK_DIR = "${WORKDIR}/sdk" SDK_OUTPUT = "${SDK_DIR}/image" SDK_DEPLOY = "${DEPLOY_DIR}/sdk" +DEPLOYDIR = "${WORKDIR}/deploy-${PN}" + B_task-populate-sdk = "${SDK_DIR}" SDKTARGETSYSROOT = "${SDKPATH}/sysroots/${REAL_MULTIMACH_TARGET_SYS}"
@@ -58,8 +60,8 @@ SDK_RELOCATE_AFTER_INSTALL ?= "1" SDKEXTPATH ?= "~/${@d.getVar('DISTRO', True)}_sdk" SDK_TITLE ?= "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} SDK" -SDK_TARGET_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.target.manifest" -SDK_HOST_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.host.manifest" +SDK_TARGET_MANIFEST = "${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.target.manifest" +SDK_HOST_MANIFEST = "${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.host.manifest" python write_target_sdk_manifest () { from oe.sdk import sdk_list_installed_packages from oe.utils import format_pkg_list
@@ -180,14 +182,14 @@ SDKTAROPTS = "--owner=root --group=root" fakeroot tar_sdk() { # Package it up - mkdir -p ${SDK_DEPLOY} + mkdir -p ${DEPLOYDIR} cd ${SDK_OUTPUT}/${SDKPATH} - tar ${SDKTAROPTS} -cf - . | pixz > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz + tar ${SDKTAROPTS} -cf - . | pixz > ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz } fakeroot create_shar() { # copy in the template shar extractor script - cp ${COREBASE}/meta/files/toolchain-shar-extract.sh ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + cp ${COREBASE}/meta/files/toolchain-shar-extract.sh ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh rm -f ${T}/pre_install_command ${T}/post_install_command
@@ -203,7 +205,7 @@ ${SDK_POST_INSTALL_COMMAND} EOF sed -i -e '/@SDK_PRE_INSTALL_COMMAND@/r ${T}/pre_install_command' \ -e '/@SDK_POST_INSTALL_COMMAND@/r ${T}/post_install_command' \ - ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh # substitute variables sed -i -e 's#@SDK_ARCH@#${SDK_ARCH}#g' \
@@ -215,16 +217,16 @@ EOF -e 's#@SDK_VERSION@#${SDK_VERSION}#g' \ -e '/@SDK_PRE_INSTALL_COMMAND@/d' \ -e '/@SDK_POST_INSTALL_COMMAND@/d' \ - ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh # add execution permission - chmod +x ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + chmod +x ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh # append the SDK tarball - cat ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + cat ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh # delete the old tarball, we don't need it anymore - rm ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz + rm ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz } populate_sdk_log_check() {
Signature · 1 line
2.1.4
PATCH 9/18 rootfs-postcommands.bbclass: generate manifest in DEPLOYDIR
In reply to Ed BartoshUpdated manifest path as deployment directory is changed from DEPLOY_DIR_IMAGE to DEPLOYDIR.
--- meta/classes/rootfs-postcommands.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/rootfs-postcommands.bbclass +2 -2
@@ -15,7 +15,7 @@ ROOTFS_POSTPROCESS_COMMAND += "rootfs_update_timestamp ; " ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", "read_only_rootfs_hook; ", "",d)}' # Write manifest -IMAGE_MANIFEST = "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.manifest" +IMAGE_MANIFEST = "${DEPLOYDIR}/${IMAGE_NAME}.rootfs.manifest" ROOTFS_POSTUNINSTALL_COMMAND =+ "write_image_manifest ; " # Set default postinst log file POSTINST_LOGFILE ?= "${localstatedir}/log/postinstall.log"
@@ -217,7 +217,7 @@ python write_image_manifest () { from oe.rootfs import image_list_installed_packages from oe.utils import format_pkg_list - deploy_dir = d.getVar('DEPLOY_DIR_IMAGE', True) + deploy_dir = d.getVar('DEPLOYDIR', True) link_name = d.getVar('IMAGE_LINK_NAME', True) manifest_name = d.getVar('IMAGE_MANIFEST', True)
Signature · 1 line
2.1.4
PATCH 10/18 selftest: renamed variable
In reply to Ed BartoshRenamed variable deploy_dir to deploy_dir_image to avoid confusion with DEPLOYDIR variable.
--- meta/lib/oeqa/selftest/imagefeatures.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oeqa/selftest/imagefeatures.py +2 -2
@@ -113,9 +113,9 @@ class ImageFeatures(oeSelfTest): image_name = 'core-image-minimal' bitbake(image_name) - deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE') + deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE') link_name = get_bb_var('IMAGE_LINK_NAME', image_name) - image_path = os.path.join(deploy_dir, "%s.ext4" % link_name) + image_path = os.path.join(deploy_dir_image, "%s.ext4" % link_name) bmap_path = "%s.bmap" % image_path # check if result image and bmap file are in deploy directory
Signature · 1 line
2.1.4
PATCH 11/18 rootfs.py: use DEPLOYDIR instead of DEPLOY_DIR_IMAGE
In reply to Ed BartoshUpdated the code of Rootfs class to use DEPLOYDIR variable as it's now used as a new deployment destination.
--- meta/lib/oe/rootfs.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/lib/oe/rootfs.py +3 -3
@@ -19,7 +19,7 @@ class Rootfs(object, metaclass=ABCMeta): self.d = d self.pm = None self.image_rootfs = self.d.getVar('IMAGE_ROOTFS', True) - self.deploy_dir_image = self.d.getVar('DEPLOY_DIR_IMAGE', True) + self.deploydir = self.d.getVar('DEPLOYDIR', True) self.progress_reporter = progress_reporter self.install_order = Manifest.INSTALL_ORDER
@@ -182,12 +182,12 @@ class Rootfs(object, metaclass=ABCMeta): bb.utils.mkdirhier(self.image_rootfs) - bb.utils.mkdirhier(self.deploy_dir_image) + bb.utils.mkdirhier(self.deploydir) shutil.copytree(postinst_intercepts_dir, intercepts_dir) shutil.copy(self.d.expand("${COREBASE}/meta/files/deploydir_readme.txt"), - self.deploy_dir_image + + self.deploydir + "/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt") execute_pre_post_process(self.d, pre_process_cmds)
Signature · 1 line
2.1.4
PATCH 12/18 image.bbclass: put image_complete under sstate control
In reply to Ed BartoshAdding image_complete task should make sstate machinery to generate manifest for deployed images and do final deployment to DEPLOY_DIR_IMAGE.
--- meta/classes/image.bbclass | 5 +++++ 1 file changed, 5 insertions(+)
meta/classes/image.bbclass +5 -0
@@ -264,6 +264,9 @@ do_image[dirs] = "${TOPDIR}" do_image[umask] = "022" addtask do_image after do_rootfs before do_build +DEPLOYDIR = "${WORKDIR}/deploy-${PN}" +SSTATETASKS += "do_image_complete" + fakeroot python do_image_complete () { from oe.utils import execute_pre_post_process
@@ -273,6 +276,8 @@ fakeroot python do_image_complete () { } do_image_complete[dirs] = "${TOPDIR}" do_image_complete[umask] = "022" +do_image_complete[sstate-inputdirs] = "${DEPLOYDIR}" +do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" addtask do_image_complete after do_image before do_build # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
Signature · 1 line
2.1.4
PATCH 13/18 image.bbclass: cleanup DEPLOYDIR
In reply to Ed BartoshMake sure DEPLOYDIR doesn't contain images from past deployments to prevent them to be included into sstate manifests.
--- meta/classes/image.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes/image.bbclass +1 -0
@@ -262,6 +262,7 @@ fakeroot python do_image () { } do_image[dirs] = "${TOPDIR}" do_image[umask] = "022" +do_image[cleandirs] = "${DEPLOYDIR}" addtask do_image after do_rootfs before do_build DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
Signature · 1 line
2.1.4
PATCH 14/18 populate_sdk_base: put populate_sdk under sstate control
In reply to Ed BartoshAdding populate_sdk task to SSTATE_TASKS should make sstate machinery to generate manifest for deployed sdk artifacts and do final deployment to SDK_DEPLOY.
--- meta/classes/populate_sdk_base.bbclass | 4 ++++ 1 file changed, 4 insertions(+)
meta/classes/populate_sdk_base.bbclass +4 -0
@@ -27,6 +27,7 @@ SDK_OUTPUT = "${SDK_DIR}/image" SDK_DEPLOY = "${DEPLOY_DIR}/sdk" DEPLOYDIR = "${WORKDIR}/deploy-${PN}" +SSTATETASKS += "do_populate_sdk" B_task-populate-sdk = "${SDK_DIR}"
@@ -118,6 +119,9 @@ fakeroot python do_populate_sdk() { populate_sdk(d) } +do_populate_sdk[sstate-inputdirs] = "${DEPLOYDIR}" +do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}" + fakeroot create_sdk_files() { cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
Signature · 1 line
2.1.4
PATCH 15/18 sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set
In reply to Ed BartoshSSTATE_SKIP_CREATION variable will be used to skip creation of sstate .tgz files. It makes sense for image creation tasks as tarring images and keeping them in sstate would consume a lot of disk space.
--- meta/classes/sstate.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/sstate.bbclass +2 -0
@@ -566,6 +566,8 @@ def sstate_package(ss, d): for state in ss['dirs']: if not os.path.exists(state[1]): continue + if d.getVar('SSTATE_SKIP_CREATION', True) == '1': + continue srcbase = state[0].rstrip("/").rsplit('/', 1)[0] for walkroot, dirs, files in os.walk(state[1]): for file in files:
Signature · 1 line
2.1.4
PATCH 16/18 image: populate_sdk_base: skip sstate creation
In reply to Ed BartoshSet SSTATE_SKIP_CREATION variable to '1' for image_complete and populate_sdk tasks as we don't want to put images and sdk installer into sstate due to their big size. We added these tasks under sstate control only for two reasons:
- to generate sstate manifest
- to make final deployments
--- meta/classes/image.bbclass | 1 + meta/classes/populate_sdk_base.bbclass | 1 + 2 files changed, 2 insertions(+)
meta/classes/image.bbclass +1 -0
@@ -267,6 +267,7 @@ addtask do_image after do_rootfs before do_build DEPLOYDIR = "${WORKDIR}/deploy-${PN}" SSTATETASKS += "do_image_complete" +SSTATE_SKIP_CREATION_task-image-complete = '1' fakeroot python do_image_complete () { from oe.utils import execute_pre_post_process
meta/classes/populate_sdk_base.bbclass +1 -0
@@ -28,6 +28,7 @@ SDK_DEPLOY = "${DEPLOY_DIR}/sdk" DEPLOYDIR = "${WORKDIR}/deploy-${PN}" SSTATETASKS += "do_populate_sdk" +SSTATE_SKIP_CREATION_task-populate-sdk = '1' B_task-populate-sdk = "${SDK_DIR}"
Signature · 1 line
2.1.4
PATCH 17/18 image: populate_sdk_base: set stamp-extra-info flag
In reply to Ed BartoshThis flag is used in the name of sstate manifest.
Setting it to predetermined value for image_complete and populate_sdk tasks should help to get correct manifest filenames when processing runQueueTask events.
--- meta/classes/image.bbclass | 1 + meta/classes/populate_sdk_base.bbclass | 1 + 2 files changed, 2 insertions(+)
meta/classes/image.bbclass +1 -0
@@ -280,6 +280,7 @@ do_image_complete[dirs] = "${TOPDIR}" do_image_complete[umask] = "022" do_image_complete[sstate-inputdirs] = "${DEPLOYDIR}" do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_image_complete[stamp-extra-info] = "${MACHINE}" addtask do_image_complete after do_image before do_build # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
meta/classes/populate_sdk_base.bbclass +1 -0
@@ -122,6 +122,7 @@ fakeroot python do_populate_sdk() { do_populate_sdk[sstate-inputdirs] = "${DEPLOYDIR}" do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}" +do_populate_sdk[stamp-extra-info] = "${MACHINE}" fakeroot create_sdk_files() { cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
Signature · 1 line
2.1.4
PATCH 18/18 toaster: fire TaskArtifacts event
In reply to Ed BartoshFire TaskArtifact MetaData event for deployment tasks when task either completed or skipped. Event contains full task id (recipe+task) and list of deployment artifacts from sstate manifest.
This should allow Toaster to always get notified about deployment artifacts produced by the build.
[YOCTO #9869]
--- meta/classes/toaster.bbclass | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
meta/classes/toaster.bbclass +17 -0
@@ -324,6 +324,20 @@ python toaster_buildhistory_dump() { } +# get list of artifacts from sstate manifest +python toaster_artifacts() { + if e.taskname in ["do_deploy", "do_image_complete", "do_populate_sdk", "do_populate_sdk_ext"]: + d2 = d.createCopy() + d2.setVar('FILE', e.taskfile) + d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}")) + manifest = oe.sstatesig.sstate_get_manifest_filename(e.taskname[3:], d2)[0] + if os.access(manifest, os.R_OK): + with open(manifest) as fmanifest: + artifacts = [fname.strip() for fname in fmanifest] + data = {"task": e.taskid, "artifacts": artifacts} + bb.event.fire(bb.event.MetadataEvent("TaskArtifacts", data), d2) +} + # set event handlers addhandler toaster_layerinfo_dumpdata toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -334,6 +348,9 @@ toaster_collect_task_stats[eventmask] = "bb.event.BuildCompleted bb.build.TaskSu addhandler toaster_buildhistory_dump toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted" +addhandler toaster_artifacts +toaster_artifacts[eventmask] = "bb.runqueue.runQueueTaskSkipped bb.runqueue.runQueueTaskCompleted" + do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata " do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
Signature · 1 line
2.1.4
Hi Ed,
Quoted 10 lines · Ed Bartosh
This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
The list of artifacts produced by deployment tasks (
do_deploy,do_image_completeand do_populate_sdk[_ext] is
obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to
handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
Quoted 8 lines
To generate manifests for
do_image_completeanddo_populate_sdkthey have been put under sstate control.To avoid storing big files(images and sdk installer) in sstate new variable
SSTATE_SKIP_CREATIONhas been
set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files
to sstate ifSSTATE_SKIP_CREATIONis set.
I'd like to see this series reordered to:
a) Set DEPLOYDIR in image.bbclass/populate_sdk_base.bbclass to point at DEPLOY_DIR_IMAGE
b) Add SSTATE_SKIP_CREATION support
c) Make all the DEPLOY_DIR_IMAGE -> DEPLOYDIR changes in one patch
d) Enable sstate for images (change DEPLOYDIR definition, add the flags including cleandirs, stamp-extra, set SSTATE_SKIP_CREATION)
e) As per d but for populate_sdk_base
That way it applies incrementally and like changes are all grouped together. It will mean some commit messages are longer and squashed together but I think that is fine.
Cheers,
Richard
Quoted 8 lines · Ed Bartosh
Hi,
This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
The list of artifacts produced by deployment tasks (
do_deploy,do_image_completeand do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
Quoted 5 lines
To generate manifests for
do_image_completeanddo_populate_sdkthey have been put under sstate control.To avoid storing big files(images and sdk installer) in sstate new variable
SSTATE_SKIP_CREATIONhas been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate ifSSTATE_SKIP_CREATIONis set.
SKIP creation of what? Variable name seems a little vague for me. Even needing the manifest, does it makes sense to store the image and SDK on sstate?
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Quoted 9 lines · Otavio Salvador
Quoted 8 lines · Ed Bartosh
Hi,
This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
The list of artifacts produced by deployment tasks (
do_deploy,do_image_completeand do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
Quoted 7 lines
Quoted 5 lines
To generate manifests for
do_image_completeanddo_populate_sdkthey have been put under sstate control.To avoid storing big files(images and sdk installer) in sstate new variable
SSTATE_SKIP_CREATIONhas been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate ifSSTATE_SKIP_CREATIONis set.SKIP creation of what?
SKIP creation of SSTATE entity.
Quoted 3 lines
Variable name seems a little vague for me. Even
needing the manifest, does it makes sense to store the image and SDK on sstate?
It doesn't and skipping this is a purpose of introducing SSTATE_SKIP_CREATION variable.
--
Regards,
Ed
Quoted 10 lines · Ed Bartosh
Quoted 9 lines · Otavio Salvador
Quoted 8 lines · Ed Bartosh
Hi,
This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
The list of artifacts produced by deployment tasks (
do_deploy,do_image_completeand do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
To generate manifests for
do_image_completeanddo_populate_sdkthey have been put under sstate control.To avoid storing big files(images and sdk installer) in sstate new variable
SSTATE_SKIP_CREATIONhas been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate ifSSTATE_SKIP_CREATIONis set.SKIP creation of what?
SKIP creation of SSTATE entity.
I know but of WHAT?
SSTATE_SKIP_IMAGE_CREATION would be a clearer name for me.
Quoted 4 lines
Quoted 3 lines
Variable name seems a little vague for me. Even
needing the manifest, does it makes sense to store the image and SDK on sstate?It doesn't and skipping this is a purpose of introducing
SSTATE_SKIP_CREATIONvariable.
So it is always going to be skipped?
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Quoted 13 lines · Otavio Salvador
Quoted 12 lines · Ed Bartosh
Quoted 11 lines · Otavio Salvador
Quoted 10 lines · Ed Bartosh
Hi,
This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
The list of artifacts produced by deployment tasks (
do_deploy,do_image_completeand do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.To generate manifests for
do_image_completeanddo_populate_sdkthey have been put under sstate control.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
To avoid storing big files(images and sdk installer) in sstate new variable
SSTATE_SKIP_CREATIONhas been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate ifSSTATE_SKIP_CREATIONis set.SKIP creation of what?
SKIP creation of SSTATE entity.
I know but of WHAT?
Of anything we don't want to put to sstate. In this patchset it's set for do_image_complete and do_populate_sdk, so it's for images and sdk installer.
Quoted 1 line
SSTATE_SKIP_IMAGE_CREATIONwould be a clearer name for me.
I'm not sure about it. I'd need one more variableSSTATE_SKIP_SDK_INSTALLER if I go this way.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Variable name seems a little vague for me. Even
needing the manifest, does it makes sense to store the image and SDK on sstate?It doesn't and skipping this is a purpose of introducing
SSTATE_SKIP_CREATIONvariable.So it is always going to be skipped?
Yes it is for do_image_complete and do_populate_sdk tasks for now.
--
Regards,
Ed
Quoted 2 lines · Ed Bartosh
Quoted 1 line
So it is always going to be skipped?
Yes it is for
do_image_completeanddo_populate_sdktasks for now.
Oh ok.
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Quoted 6 lines · Otavio Salvador
Quoted 2 lines · Ed Bartosh
Quoted 1 line
So it is always going to be skipped?
Yes it is for
do_image_completeanddo_populate_sdktasks for now.Oh ok.
Its a performance issue. I can see the attraction of putting these into sstate however it would add significant time to the builds which many people would find unacceptable.
Having manifests of deployed files is a huge win for a whole variety of reasons and it standardises our output tasks. This compromise gives us that without the performance hit of compressing everything into an sstate artefact.
Cheers,
Richard