Skip to contentSkip to conversation

lib/oe/gpg_sign.py: Avoid race when creating .sig files in detach_sign

PATCH Tobias Hagelborn 1 participant 1 message Mar 23, 2023 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

Some context to this:

We run a shared state-cache and dockerized builds on large servers. Running ~30 builds and ~200 parallel tasks using shared signed-sstate gives this type of error messages (stemming from corruption of the .sig files created):

ERROR: packagegroup-<PACKAGE>-1.0-r0 do_populate_lic: GPG exited with code 2: gpg: signing failed: Bad passphrase

The provided solution with renaming the files into final destination has proven to eliminate the issue.

Cheers
Tobias

From: [email protected] <[email protected]> on behalf of Tobias Hagelborn <[email protected]>
Sent: Thursday, March 23, 2023 11:08 AM
To: [email protected] <[email protected]>
Subject: [OE-core] [PATCH] lib/oe/gpg_sign.py: Avoid race when creating .sig files in detach_sign

Move the signature file into place only after it is successfully signed.
This to avoid race and corrupted .sig files in cases multiple onging
builds write to a shared sstate-cache dir.

---
 meta/lib/oe/gpg_sign.py | 25 +++++++++++++++----------

 1 file changed, 15 insertions(+), 10 deletions(-)
meta/lib/oe/gpg_sign.py +0 -0
index 613dab8561..868846cdc5 100644
@@ -5,11 +5,12 @@

 #

 """Helper module for GPG signing"""
-import os

 import bb
-import subprocess
+import os
 import shlex
+import subprocess
+import tempfile

 class LocalSigner(object):
     """Class for handling local (on the build host) signing"""
@@ -73,8 +74,6 @@ class LocalSigner(object):
             cmd += ['--homedir', self.gpg_path]
         if armor:
             cmd += ['--armor']
-        if output_suffix:
-            cmd += ['-o', input_file + "." + output_suffix]
         if use_sha256:
             cmd += ['--digest-algo', "SHA256"]

@@ -83,19 +82,25 @@ class LocalSigner(object):
         if self.gpg_version > (2,1,):
             cmd += ['--pinentry-mode', 'loopback']

-        cmd += [input_file]
-
         try:
             if passphrase_file:
                 with open(passphrase_file) as fobj:
                     passphrase = fobj.readline();

-            job = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE) -            (_, stderr) = job.communicate(passphrase.encode("utf-8"))
+            output_file = input_file + "." + (output_suffix or 'sig')
+            with tempfile.TemporaryDirectory(dir=os.path.dirname(output_file)) as tmp_dir:
+                tmp_file = os.path.join(tmp_dir, os.path.basename(output_file))
+                cmd += ['-o', tmp_file]
+
+                cmd += [input_file]
+
+                job = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE) +                (_, stderr) = job.communicate(passphrase.encode("utf-8"))

-            if job.returncode:
-                bb.fatal("GPG exited with code %d: %s" % (job.returncode, stderr.decode("utf-8"))) +                if job.returncode:
+                    bb.fatal("GPG exited with code %d: %s" % (job.returncode, stderr.decode("utf-8")))
+                os.rename(tmp_file, output_file)
         except IOError as e:
             bb.error("IO error (%s): %s" % (e.errno, e.strerror))
             raise Exception("Failed to sign '%s'" % input_file)

Signature · 1 line
2.30.2