PublishArtifacts.py: deal only with built toolchains, cp also md5 and manifests

From 8f28afbfc21dc84f076b0cc620e578f9607d1465 Mon Sep 17 00:00:00 2001
From: Robert Berger <[email protected]>
Date: Mon, 10 Oct 2016 06:26:06 +0000
Subject: [yocto-autobuilder][PATCH] PublishArtifacts.py: deal only with built toolchains, cp also
md5 and manifests

---
 .../autobuilder/buildsteps/PublishArtifacts.py             | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

index 633600c..46d7b40 100644
@@ -154,14 +154,16 @@ class PublishArtifacts(ShellCommand):
                 elif artifact == "toolchain":
                     artifact_name, deploy_image_dir = self.getDeployNames(artifact, buildername)
                     command=command+self.generateMD5cmd(artifact, deploy_image_dir)
-                    command=command+"mkdir -p " + os.path.join(DEST, X86TC_PUBLISH_DIR) + ";"
+                    command=command+"if [ -e " + os.path.join(deploy_image_dir, "poky-*i686-core-image*.sh ") + " ]; then " + \
+                                    "mkdir -p " + os.path.join(DEST, X86TC_PUBLISH_DIR) + "; "
                     command=command+"cp -R --no-dereference --preserve=links " + \
-                                    os.path.join(deploy_image_dir, "poky-*i686-core-image*.sh ") + \
-                                    os.path.join(DEST, X86TC_PUBLISH_DIR) + pipeline + ";"
-                    command=command+"mkdir -p " + os.path.join(DEST, X8664TC_PUBLISH_DIR) + ";"
+                                    os.path.join(deploy_image_dir, "poky-*i686-core-image* ") + \
+                                    os.path.join(DEST, X86TC_PUBLISH_DIR) + pipeline + "; fi;"
+                    command=command+"if [ -e " + os.path.join(deploy_image_dir, "poky-*x86_64-core-image*.sh ") + " ]; then " + \
+                                    "mkdir -p " + os.path.join(DEST, X8664TC_PUBLISH_DIR) + "; "
                     command=command+"cp -R --no-dereference --preserve=links " + \
-                                    os.path.join(deploy_image_dir, "poky-*x86_64-core-image*.sh ") + \
-                                    os.path.join(DEST, X8664TC_PUBLISH_DIR) + pipeline + ";"
+                                    os.path.join(deploy_image_dir, "poky-*x86_64-core-image* ") + \
+                                    os.path.join(DEST, X8664TC_PUBLISH_DIR) + pipeline + "; fi;"
                 elif artifact == "uninative":
                     artifact_name, deploy_image_dir = self.getDeployNames(artifact, buildername)
                     command=command+self.generateMD5cmd(artifact, deploy_image_dir)

2.7.4

Hi Robert,

Then "normal" method to send patches is via direct insertion into the email, rather than an attachment (see git-send-email). Also, it is missing a commit comment as the first text in the patch file. Otherwise, it looks good to me now.

-Bill

2016-10-09 23:44 GMT-07:00 <[email protected]>:

I'm going to pull this but please, in the future, follow the above.

-b

A few notes (not picking on this patch but it does point out some design failures and I want to at least get this on folks radar).

I've been unhappy with how DeployArtifacts work for sometime.

Long ago, when I wrote DeployArtifacts, it was "get it working, we'll refactor later" and that refactor never happened. So I'm up for ideas.

I would like a deploy infrastructure that:

a. Is distro/image name/etc agnostic (aka, poky and image names and paths hardcoded. ick.).
b. Is perhaps config file driven.
c. Had much less of the cruft that exists withing DeployArtifacts.

So. Thoughts on this? I'd be willing to hear suggestions on how to do this before I go and spend some time ripping this apart post release?

-b

Beth 'pidge' Flanagan <[email protected]>
toganlabs.com

It's quite useful to have a list of expected output of the build so that this step can fail if a component which should be published is missing.

I'd been planning to take a look at rewriting this build step after the release and my initial thoughts were towards something config file driven.

Thanks,

Joshua

Can you help me understand why you needed to create this patch?

We've run into some issues recently where toolchains we expected to be built weren't and the PublishArtifacts buildstep failing because they're missing is useful. With this change we'll no longer get that, right?

Thanks,

Joshua
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Hi,

Yes that's right (and not intended).

I would hope that you'll be able to detect such kind of problems before the PublishArtifacts buildstep because there should be some error in previous steps. The toolchain did not build!

I made this patch because I just want to build a 64 bit toolchain as opposed to both (like I used to do with some older version of Y-AB) and don't want the PublishArtifacts step to fail just because there is no 32-bit toolchain. There is no 32 bit toolchain on purpose.

As a bonus the patch also copies md5sums and friends over and not just the .sh files.

I do not want PublishArtifacts toolchain(s) to find out that something went wrong in previous build steps, but I would want it to copy over whatever was built (and not to fail if something I did not build is missing).

As a quick and dirty alternative I would also be happy with a solution based on additional PublishArtifacts steps like:

toolchain_i686 for 32 bit only (only the 32 bit toolchain)
toolchain_x86_64 for 64 bit only (only the 64 bit toolchain)

if that helps.

Then in my config file I just use toolchain_x86_64 instead of toolchain in my PublishArtifacts step to indicate that I just want the 64 bit version.

To publish both you could use toolchain_i686 and toolchain_x86_64 in the PublishArtifacts step.

I find it a bit confusing to write toolchain and to expect various 64 bit and 32 bit toolchains to be published.

BTW what happens here[1] if one toolchain does not build?

I think even without my patch you'll not be able to detect the problem, since BuildToolchainImages is being called 4 times so two 32 bit and two 64 bit toolchains should be built. You'll just get an error in PublishArtifacts before my patch if no 32 bit or no 64 bit toolchain is being built. Nobody counts how many should be there.

So I would say the best would be to count how many toolchains should be built (count BuildToolchainImages in combination with SDKMACHINE) and check if that's the case in PublishArtifacts. No idea if and how this can be done.

Regards,

Robert

[1] http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/tree/buildset-config.controller/nightly-arm.conf

Indeed, but software is buggy. This happened very recently on the public builders:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=10275

Toolchains *were* being built but they were being unstaged from the deploy directory by an unfortunate bug. We had errors logged due to failing cp the buildstep didn't fail.

This is a reasonable goal, but something I'd rather see addressed in the proposed PublishArtifacts rewrite. As we're trying to release morty right now I'd like to avoid changes to the AB behaviour until after the release.

Regards,

Joshua

Hi,

I'm absolutely happy with this and eagerly waiting for morty. Please keep me updated so, as my time permits, I can test the new Y-AB version.

Regards,

Robert