Provide list of deployment artifacts
v3 cover Provide list of deployment artifacts
Hi,
This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.
To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files to sstate if SSTATE_SKIP_CREATION is set.
Changes in v2: Reorganized patchset to make it bisectable (Thanks Richard)
Used task in the name of DEPLOYDIR to avoid using the same directory for different tasks of the same recipe
Changes in v3: Fixed bug that caused deletion of image manifest
The following changes since commit 087c580b286816265f487e02746bfa6e26081554:
init-install: Fixes the install script failing when not finding any mmcblk devices (2016-08-30 07:57:50 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/oe-core/artifacts-9869
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/artifacts-9869
Ed Bartosh (6):
image: populate_sdk_base: add DEPLOYDIR variable sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set
image: populate_sdk: deploy images to DEPLOYDIR image.bbclass: put image_complete under sstate control
populate_sdk_base: put populate_sdk under sstate control
toaster: fire TaskArtifacts event
meta/classes/image-live.bbclass | 12 +++--- meta/classes/image-vm.bbclass | 22 +++++------ meta/classes/image.bbclass | 15 ++++++-- meta/classes/image_types.bbclass | 44 +++++++++++----------- meta/classes/image_types_uboot.bbclass | 2 +- meta/classes/populate_sdk_base.bbclass | 27 ++++++++----- meta/classes/rootfs-postcommands.bbclass | 4 +- meta/classes/sstate.bbclass | 2 + meta/classes/syslinux.bbclass | 2 +- meta/classes/toaster.bbclass | 17 +++++++++ meta/lib/oe/rootfs.py | 6 +-- meta/lib/oeqa/selftest/imagefeatures.py | 4 +- .../images/build-appliance-image_15.0.0.bb | 8 ++-- 13 files changed, 99 insertions(+), 66 deletions(-)
--
Ed
PATCH v3 1/6 image: populate_sdk_base: add DEPLOYDIR variable
This is a preparation for changing deployment directory for image and populate_sdk targets.
Introduced new variable DEPLOYDIR. Set it to current image/sdk deployment locations.
--- meta/classes/image.bbclass | 2 ++ meta/classes/populate_sdk_base.bbclass | 2 ++ 2 files changed, 4 insertions(+)
meta/classes/image.bbclass +2 -0
@@ -74,6 +74,8 @@ IMAGE_INSTALL[type] = "list" export PACKAGE_INSTALL ?= "${IMAGE_INSTALL} ${ROOTFS_BOOTSTRAP_INSTALL} ${FEATURE_INSTALL}" PACKAGE_INSTALL_ATTEMPTONLY ?= "${FEATURE_INSTALL_OPTIONAL}" +DEPLOYDIR = "${DEPLOY_DIR_IMAGE}" + # Images are generally built explicitly, do not need to be part of world. EXCLUDE_FROM_WORLD = "1"
meta/classes/populate_sdk_base.bbclass +2 -0
@@ -26,6 +26,8 @@ SDK_DIR = "${WORKDIR}/sdk" SDK_OUTPUT = "${SDK_DIR}/image" SDK_DEPLOY = "${DEPLOY_DIR}/sdk" +DEPLOYDIR = "${SDK_DEPLOY}" + B_task-populate-sdk = "${SDK_DIR}" SDKTARGETSYSROOT = "${SDKPATH}/sysroots/${REAL_MULTIMACH_TARGET_SYS}"
Signature · 1 line
2.1.4
PATCH v3 2/6 sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set
In reply to Ed BartoshSSTATE_SKIP_CREATION variable will be used to skip creation of sstate .tgz files. It makes sense for image creation tasks as tarring images and keeping them in sstate would consume a lot of disk space.
--- meta/classes/sstate.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/sstate.bbclass +2 -0
@@ -566,6 +566,8 @@ def sstate_package(ss, d): for state in ss['dirs']: if not os.path.exists(state[1]): continue + if d.getVar('SSTATE_SKIP_CREATION', True) == '1': + continue srcbase = state[0].rstrip("/").rsplit('/', 1)[0] for walkroot, dirs, files in os.walk(state[1]): for file in files:
Signature · 1 line
2.1.4
PATCH v3 3/6 image: populate_sdk: deploy images to DEPLOYDIR
In reply to Ed BartoshChanged deployment directory from DEPLOY_DIR_IMAGE/SDK_DEPLOY to DEPLOYDIR to make sstate machinery to do final deployment and generate manifest.
Renamed variable deploy_dir to deploy_dir_image in selftest code to avoid confusion with DEPLOYDIR variable.
Updated the code of rootfs.py:Rootfs class to use DEPLOYDIR variable as it's now used as a new deployment destination.
--- meta/classes/image-live.bbclass | 12 +++--- meta/classes/image-vm.bbclass | 22 +++++------ meta/classes/image.bbclass | 6 +-- meta/classes/image_types.bbclass | 44 +++++++++++----------- meta/classes/image_types_uboot.bbclass | 2 +- meta/classes/populate_sdk_base.bbclass | 20 +++++----- meta/classes/rootfs-postcommands.bbclass | 4 +- meta/classes/syslinux.bbclass | 2 +- meta/lib/oe/rootfs.py | 6 +-- meta/lib/oeqa/selftest/imagefeatures.py | 4 +- .../images/build-appliance-image_15.0.0.bb | 8 ++-- 11 files changed, 65 insertions(+), 65 deletions(-)
meta/classes/image-live.bbclass +6 -6
@@ -43,7 +43,7 @@ ROOT_LIVE ?= "root=/dev/ram0" INITRD_IMAGE_LIVE ?= "core-image-minimal-initramfs" INITRD_LIVE ?= "${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_LIVE}-${MACHINE}.cpio.gz" -ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.ext4" +ROOTFS ?= "${DEPLOYDIR}/${IMAGE_LINK_NAME}.ext4" IMAGE_TYPEDEP_live = "ext4" IMAGE_TYPEDEP_iso = "ext4"
@@ -144,14 +144,14 @@ build_iso() { if [ "${PCBIOS}" = "1" ] && [ "${EFI}" != "1" ] ; then # PCBIOS only media mkisofs -V ${BOOTIMG_VOLUME_ID} \ - -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso \ + -o ${DEPLOYDIR}/${IMAGE_NAME}.iso \ -b ${ISO_BOOTIMG} -c ${ISO_BOOTCAT} \ $mkisofs_compress_opts \ ${MKISOFS_OPTIONS} $mkisofs_iso_level ${ISODIR} else # EFI only OR EFI+PCBIOS mkisofs -A ${BOOTIMG_VOLUME_ID} -V ${BOOTIMG_VOLUME_ID} \ - -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso \ + -o ${DEPLOYDIR}/${IMAGE_NAME}.iso \ -b ${ISO_BOOTIMG} -c ${ISO_BOOTCAT} \ $mkisofs_compress_opts ${MKISOFS_OPTIONS} $mkisofs_iso_level \ -eltorito-alt-boot -eltorito-platform efi \
@@ -160,7 +160,7 @@ build_iso() { isohybrid_args="-u" fi - isohybrid $isohybrid_args ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso + isohybrid $isohybrid_args ${DEPLOYDIR}/${IMAGE_NAME}.iso } build_fat_img() {
@@ -252,13 +252,13 @@ build_hddimg() { fi fi - build_fat_img ${HDDDIR} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg + build_fat_img ${HDDDIR} ${DEPLOYDIR}/${IMAGE_NAME}.hddimg if [ "${PCBIOS}" = "1" ]; then syslinux_hddimg_install fi - chmod 644 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg + chmod 644 ${DEPLOYDIR}/${IMAGE_NAME}.hddimg fi }
meta/classes/image-vm.bbclass +11 -11
@@ -33,14 +33,14 @@ IMAGE_TYPEDEP_hdddirect = "${VM_ROOTFS_TYPE}" IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect" VM_ROOTFS_TYPE ?= "ext4" -ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}" +ROOTFS ?= "${DEPLOYDIR}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}" # Used by bootloader LABELS_VM ?= "boot" ROOT_VM ?= "root=/dev/sda2" # Using an initramfs is optional. Enable it by setting INITRD_IMAGE_VM. INITRD_IMAGE_VM ?= "" -INITRD_VM ?= "${@'${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}" +INITRD_VM ?= "${@'${DEPLOYDIR}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}" do_bootdirectdisk[depends] += "${@'${INITRD_IMAGE_VM}:do_image_complete' if '${INITRD_IMAGE_VM}' else ''}" BOOTDD_VOLUME_ID ?= "boot"
@@ -52,7 +52,7 @@ DISK_SIGNATURE[vardepsexclude] = "DISK_SIGNATURE_GENERATED" build_boot_dd() { HDDDIR="${S}/hdd/boot" HDDIMG="${S}/hdd.image" - IMAGE=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect + IMAGE=${DEPLOYDIR}/${IMAGE_NAME}.hdddirect populate_kernel $HDDDIR
@@ -104,13 +104,13 @@ build_boot_dd() { dd if=$HDDIMG of=$IMAGE conv=notrunc seek=1 bs=512 dd if=${ROOTFS} of=$IMAGE conv=notrunc seek=$OFFSET bs=512 - cd ${DEPLOY_DIR_IMAGE} + cd ${DEPLOYDIR} - if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ]; then - rm -f $(readlink -f ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect) + if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ]; then + rm -f $(readlink -f ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect) fi - ln -sf ${IMAGE_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect + ln -sf ${IMAGE_NAME}.hdddirect ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect } python do_bootdirectdisk() {
@@ -145,13 +145,13 @@ DISK_SIGNATURE_GENERATED := "${@generate_disk_signature()}" run_qemu_img (){ type="$1" - qemu-img convert -O $type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.$type + qemu-img convert -O $type ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOYDIR}/${IMAGE_NAME}.$type - if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type ]; then - rm -f $(readlink -f ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type) + if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type ]; then + rm -f $(readlink -f ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type) fi - ln -sf ${IMAGE_NAME}.$type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type + ln -sf ${IMAGE_NAME}.$type ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type } create_vmdk_image () { run_qemu_img vmdk
meta/classes/image.bbclass +3 -3
@@ -120,7 +120,7 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','RM_OLD_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'DEPLOYDIR'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables)
@@ -442,7 +442,7 @@ python () { cmds.append("\t" + image_cmd) else: bb.fatal("No IMAGE_CMD defined for IMAGE_FSTYPES entry '%s' - possibly invalid type name or missing support class" % t) - cmds.append(localdata.expand("\tcd ${DEPLOY_DIR_IMAGE}")) + cmds.append(localdata.expand("\tcd ${DEPLOYDIR}")) # Since a copy of IMAGE_CMD_xxx will be inlined within do_image_xxx, # prevent a redundant copy of IMAGE_CMD_xxx being emitted as a function.
@@ -560,7 +560,7 @@ python set_image_size () { # python create_symlinks() { - deploy_dir = d.getVar('DEPLOY_DIR_IMAGE', True) + deploy_dir = d.getVar('DEPLOYDIR', True) img_name = d.getVar('IMAGE_NAME', True) link_name = d.getVar('IMAGE_LINK_NAME', True) manifest_name = d.getVar('IMAGE_MANIFEST', True)
meta/classes/image_types.bbclass +22 -22
119 lines not on this page · raw message
meta/classes/image_types_uboot.bbclass +1 -1
@@ -2,7 +2,7 @@ inherit image_types kernel-arch oe_mkimage () { mkimage -A ${UBOOT_ARCH} -O linux -T ramdisk -C $2 -n ${IMAGE_NAME} \ - -d ${DEPLOY_DIR_IMAGE}/$1 ${DEPLOY_DIR_IMAGE}/$1.u-boot + -d ${DEPLOYDIR}/$1 ${DEPLOYDIR}/$1.u-boot if [ x$3 = x"clean" ]; then rm $1 fi
meta/classes/populate_sdk_base.bbclass +10 -10
@@ -60,8 +60,8 @@ SDK_RELOCATE_AFTER_INSTALL ?= "1" SDKEXTPATH ?= "~/${@d.getVar('DISTRO', True)}_sdk" SDK_TITLE ?= "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} SDK" -SDK_TARGET_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.target.manifest" -SDK_HOST_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.host.manifest" +SDK_TARGET_MANIFEST = "${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.target.manifest" +SDK_HOST_MANIFEST = "${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.host.manifest" python write_target_sdk_manifest () { from oe.sdk import sdk_list_installed_packages from oe.utils import format_pkg_list
@@ -182,14 +182,14 @@ SDKTAROPTS = "--owner=root --group=root" fakeroot tar_sdk() { # Package it up - mkdir -p ${SDK_DEPLOY} + mkdir -p ${DEPLOYDIR} cd ${SDK_OUTPUT}/${SDKPATH} - tar ${SDKTAROPTS} -cf - . | pixz > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz + tar ${SDKTAROPTS} -cf - . | pixz > ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz } fakeroot create_shar() { # copy in the template shar extractor script - cp ${COREBASE}/meta/files/toolchain-shar-extract.sh ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + cp ${COREBASE}/meta/files/toolchain-shar-extract.sh ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh rm -f ${T}/pre_install_command ${T}/post_install_command
@@ -205,7 +205,7 @@ ${SDK_POST_INSTALL_COMMAND} EOF sed -i -e '/@SDK_PRE_INSTALL_COMMAND@/r ${T}/pre_install_command' \ -e '/@SDK_POST_INSTALL_COMMAND@/r ${T}/post_install_command' \ - ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh # substitute variables sed -i -e 's#@SDK_ARCH@#${SDK_ARCH}#g' \
@@ -217,16 +217,16 @@ EOF -e 's#@SDK_VERSION@#${SDK_VERSION}#g' \ -e '/@SDK_PRE_INSTALL_COMMAND@/d' \ -e '/@SDK_POST_INSTALL_COMMAND@/d' \ - ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh # add execution permission - chmod +x ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + chmod +x ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh # append the SDK tarball - cat ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + cat ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh # delete the old tarball, we don't need it anymore - rm ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz + rm ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz } populate_sdk_log_check() {
meta/classes/rootfs-postcommands.bbclass +2 -2
@@ -15,7 +15,7 @@ ROOTFS_POSTPROCESS_COMMAND += "rootfs_update_timestamp ; " ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", "read_only_rootfs_hook; ", "",d)}' # Write manifest -IMAGE_MANIFEST = "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.manifest" +IMAGE_MANIFEST = "${DEPLOYDIR}/${IMAGE_NAME}.rootfs.manifest" ROOTFS_POSTUNINSTALL_COMMAND =+ "write_image_manifest ; " # Set default postinst log file POSTINST_LOGFILE ?= "${localstatedir}/log/postinstall.log"
@@ -217,7 +217,7 @@ python write_image_manifest () { from oe.rootfs import image_list_installed_packages from oe.utils import format_pkg_list - deploy_dir = d.getVar('DEPLOY_DIR_IMAGE', True) + deploy_dir = d.getVar('DEPLOYDIR', True) link_name = d.getVar('IMAGE_LINK_NAME', True) manifest_name = d.getVar('IMAGE_MANIFEST', True)
meta/classes/syslinux.bbclass +1 -1
@@ -72,7 +72,7 @@ syslinux_hddimg_populate() { } syslinux_hddimg_install() { - syslinux ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg + syslinux ${DEPLOYDIR}/${IMAGE_NAME}.hddimg } syslinux_hdddirect_install() {
meta/lib/oe/rootfs.py +3 -3
@@ -19,7 +19,7 @@ class Rootfs(object, metaclass=ABCMeta): self.d = d self.pm = None self.image_rootfs = self.d.getVar('IMAGE_ROOTFS', True) - self.deploy_dir_image = self.d.getVar('DEPLOY_DIR_IMAGE', True) + self.deploydir = self.d.getVar('DEPLOYDIR', True) self.progress_reporter = progress_reporter self.install_order = Manifest.INSTALL_ORDER
@@ -182,12 +182,12 @@ class Rootfs(object, metaclass=ABCMeta): bb.utils.mkdirhier(self.image_rootfs) - bb.utils.mkdirhier(self.deploy_dir_image) + bb.utils.mkdirhier(self.deploydir) shutil.copytree(postinst_intercepts_dir, intercepts_dir) shutil.copy(self.d.expand("${COREBASE}/meta/files/deploydir_readme.txt"), - self.deploy_dir_image + + self.deploydir + "/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt") execute_pre_post_process(self.d, pre_process_cmds)
meta/lib/oeqa/selftest/imagefeatures.py +2 -2
@@ -113,9 +113,9 @@ class ImageFeatures(oeSelfTest): image_name = 'core-image-minimal' bitbake(image_name) - deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE') + deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE') link_name = get_bb_var('IMAGE_LINK_NAME', image_name) - image_path = os.path.join(deploy_dir, "%s.ext4" % link_name) + image_path = os.path.join(deploy_dir_image, "%s.ext4" % link_name) bmap_path = "%s.bmap" % image_path # check if result image and bmap file are in deploy directory
meta/recipes-core/images/build-appliance-image_15.0.0.bb +4 -4
@@ -32,7 +32,7 @@ BA_INCLUDE_SOURCES ??= "0" IMAGE_CMD_ext4_append () { # We don't need to reserve much space for root, 0.5% is more than enough - tune2fs -m 0.5 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext4 + tune2fs -m 0.5 ${DEPLOYDIR}/${IMAGE_NAME}.rootfs.ext4 } fakeroot do_populate_poky_src () {
@@ -101,9 +101,9 @@ create_bundle_files () { cd ${WORKDIR} mkdir -p Yocto_Build_Appliance cp *.vmx* Yocto_Build_Appliance - ln -sf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk - zip -r ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance - ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance.zip + ln -sf ${DEPLOYDIR}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk + zip -r ${DEPLOYDIR}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance + ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOYDIR}/Yocto_Build_Appliance.zip } create_bundle_files[vardepsexclude] = "DATETIME"
Signature · 1 line
2.1.4
PATCH v3 4/6 image.bbclass: put image_complete under sstate control
In reply to Ed BartoshAdding image_complete task should make sstate machinery to generate manifest for deployed images and do final deployment to DEPLOY_DIR_IMAGE.
Made sure DEPLOYDIR doesn't contain images from past deployments to prevent them to be included into sstate manifests.
Set stamp-extra-info flag for do_image_complete task. This flag is used in the name of sstate manifest. Setting it to predetermined value for image_complete should help to get correct manifest filenames when processing runQueueTask events.
--- meta/classes/image.bbclass | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
meta/classes/image.bbclass +7 -2
@@ -74,7 +74,7 @@ IMAGE_INSTALL[type] = "list" export PACKAGE_INSTALL ?= "${IMAGE_INSTALL} ${ROOTFS_BOOTSTRAP_INSTALL} ${FEATURE_INSTALL}" PACKAGE_INSTALL_ATTEMPTONLY ?= "${FEATURE_INSTALL_OPTIONAL}" -DEPLOYDIR = "${DEPLOY_DIR_IMAGE}" +DEPLOYDIR = "${WORKDIR}/deploy-${PN}-image-complete" # Images are generally built explicitly, do not need to be part of world. EXCLUDE_FROM_WORLD = "1"
@@ -251,7 +251,7 @@ fakeroot python do_rootfs () { progress_reporter.finish() } do_rootfs[dirs] = "${TOPDIR}" -do_rootfs[cleandirs] += "${S}" +do_rootfs[cleandirs] += "${S} ${DEPLOYDIR}" do_rootfs[umask] = "022" addtask rootfs before do_build
@@ -275,6 +275,11 @@ fakeroot python do_image_complete () { } do_image_complete[dirs] = "${TOPDIR}" do_image_complete[umask] = "022" +SSTATETASKS += "do_image_complete" +SSTATE_SKIP_CREATION_task-image-complete = '1' +do_image_complete[sstate-inputdirs] = "${DEPLOYDIR}" +do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_image_complete[stamp-extra-info] = "${MACHINE}" addtask do_image_complete after do_image before do_build # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
Signature · 1 line
2.1.4
PATCH v3 5/6 populate_sdk_base: put populate_sdk under sstate control
In reply to Ed BartoshAdding populate_sdk task to SSTATE_TASKS should make sstate machinery to generate manifest for deployed sdk artifacts and do final deployment to SDK_DEPLOY.
Set stamp-extra-info flag for do_populate_sdk task. This flag is used in the name of sstate manifest. Setting it to predetermined value for populate_sdk task should help to get correct manifest filenames when processing runQueueTask events.
--- meta/classes/populate_sdk_base.bbclass | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes/populate_sdk_base.bbclass +6 -1
@@ -26,7 +26,7 @@ SDK_DIR = "${WORKDIR}/sdk" SDK_OUTPUT = "${SDK_DIR}/image" SDK_DEPLOY = "${DEPLOY_DIR}/sdk" -DEPLOYDIR = "${SDK_DEPLOY}" +DEPLOYDIR = "${WORKDIR}/deploy-${PN}-populate-sdk" B_task-populate-sdk = "${SDK_DIR}"
@@ -117,6 +117,11 @@ fakeroot python do_populate_sdk() { populate_sdk(d) } +SSTATETASKS += "do_populate_sdk" +SSTATE_SKIP_CREATION_task-populate-sdk = '1' +do_populate_sdk[sstate-inputdirs] = "${DEPLOYDIR}" +do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}" +do_populate_sdk[stamp-extra-info] = "${MACHINE}" fakeroot create_sdk_files() { cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
Signature · 1 line
2.1.4
PATCH v3 6/6 toaster: fire TaskArtifacts event
In reply to Ed BartoshFire TaskArtifact MetaData event for deployment tasks when task either completed or skipped. Event contains full task id (recipe+task) and list of deployment artifacts from sstate manifest.
This should allow Toaster to always get notified about deployment artifacts produced by the build.
[YOCTO #9869]
--- meta/classes/toaster.bbclass | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
meta/classes/toaster.bbclass +17 -0
@@ -324,6 +324,20 @@ python toaster_buildhistory_dump() { } +# get list of artifacts from sstate manifest +python toaster_artifacts() { + if e.taskname in ["do_deploy", "do_image_complete", "do_populate_sdk", "do_populate_sdk_ext"]: + d2 = d.createCopy() + d2.setVar('FILE', e.taskfile) + d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}")) + manifest = oe.sstatesig.sstate_get_manifest_filename(e.taskname[3:], d2)[0] + if os.access(manifest, os.R_OK): + with open(manifest) as fmanifest: + artifacts = [fname.strip() for fname in fmanifest] + data = {"task": e.taskid, "artifacts": artifacts} + bb.event.fire(bb.event.MetadataEvent("TaskArtifacts", data), d2) +} + # set event handlers addhandler toaster_layerinfo_dumpdata toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -334,6 +348,9 @@ toaster_collect_task_stats[eventmask] = "bb.event.BuildCompleted bb.build.TaskSu addhandler toaster_buildhistory_dump toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted" +addhandler toaster_artifacts +toaster_artifacts[eventmask] = "bb.runqueue.runQueueTaskSkipped bb.runqueue.runQueueTaskCompleted" + do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata " do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
Signature · 1 line
2.1.4
Quoted 10 lines · Ed Bartosh
This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
The list of artifacts produced by deployment tasks (
do_deploy,do_image_completeand do_populate_sdk[_ext] is
obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to
handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
Quoted 8 lines
To generate manifests for
do_image_completeanddo_populate_sdkthey have been put under sstate control.To avoid storing big files(images and sdk installer) in sstate new variable
SSTATE_SKIP_CREATIONhas been
set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files
to sstate ifSSTATE_SKIP_CREATIONis set.
There were multiple issues here:
* Duplicate file overlap in DEPLOY_DIR/sdk causing build failures
* Overlap of the DEPLOYDIR namespace breaking builds (between do_image_complete, do_populate_sdk and do_populate_sdk_ext but could potentially conflict with a do_deploy task added to an image too)
* Breakage of do_populate_sdk_ext since it calls do_populate_sdk
* No sstate coverage of do_populate_sdk_ext
* no cleandirs on do_populate_sdk DEPLOYDIR
I've a revised patch set I'll test which addresses these issues.
Cheers,
Richard