lib/oe/gpg_sign.py: Avoid race when creating .sig files in detach_sign
A message that is not in the archive (sent off-list or not yet received): [email protected]
Some context to this:
We run a shared state-cache and dockerized builds on large servers. Running ~30 builds and ~200 parallel tasks using shared signed-sstate gives this type of error messages (stemming from corruption of the .sig files created):
ERROR: packagegroup-<PACKAGE>-1.0-r0 do_populate_lic: GPG exited with code 2: gpg: signing failed: Bad passphrase
The provided solution with renaming the files into final destination has proven to eliminate the issue.
Cheers
Tobias
From: [email protected] <[email protected]> on behalf of Tobias Hagelborn <[email protected]>
Sent: Thursday, March 23, 2023 11:08 AM
To: [email protected] <[email protected]>
Subject: [OE-core] [PATCH] lib/oe/gpg_sign.py: Avoid race when creating .sig files in detach_sign
Move the signature file into place only after it is successfully signed.
This to avoid race and corrupted .sig files in cases multiple onging
builds write to a shared sstate-cache dir.
---
meta/lib/oe/gpg_sign.py | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
meta/lib/oe/gpg_sign.py +0 -0
@@ -5,11 +5,12 @@
#
"""Helper module for GPG signing"""
-import os
import bb
-import subprocess
+import os
import shlex
+import subprocess
+import tempfile
class LocalSigner(object):
"""Class for handling local (on the build host) signing"""
@@ -73,8 +74,6 @@ class LocalSigner(object):
cmd += ['--homedir', self.gpg_path]
if armor:
cmd += ['--armor']
- if output_suffix:
- cmd += ['-o', input_file + "." + output_suffix]
if use_sha256:
cmd += ['--digest-algo', "SHA256"]
@@ -83,19 +82,25 @@ class LocalSigner(object):
if self.gpg_version > (2,1,):
cmd += ['--pinentry-mode', 'loopback']
- cmd += [input_file]
-
try:
if passphrase_file:
with open(passphrase_file) as fobj:
passphrase = fobj.readline();
- job = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE) - (_, stderr) = job.communicate(passphrase.encode("utf-8"))
+ output_file = input_file + "." + (output_suffix or 'sig')
+ with tempfile.TemporaryDirectory(dir=os.path.dirname(output_file)) as tmp_dir:
+ tmp_file = os.path.join(tmp_dir, os.path.basename(output_file))
+ cmd += ['-o', tmp_file]
+
+ cmd += [input_file]
+
+ job = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE) + (_, stderr) = job.communicate(passphrase.encode("utf-8"))
- if job.returncode:
- bb.fatal("GPG exited with code %d: %s" % (job.returncode, stderr.decode("utf-8"))) + if job.returncode:
+ bb.fatal("GPG exited with code %d: %s" % (job.returncode, stderr.decode("utf-8")))
+ os.rename(tmp_file, output_file)
except IOError as e:
bb.error("IO error (%s): %s" % (e.errno, e.strerror))
raise Exception("Failed to sign '%s'" % input_file)
Signature · 1 line
2.30.2