Skip to contentSkip to conversation

package_manager: support for signed DEB package feeds

PATCH In review Ferry Toth 3 participants 17 messages Apr 3, 2022 – Apr 7, 2022

lore
Share & tools
lore

Ferry Toth

author
#0 lore raw

[PATCH v2 0/3] package_manager: support for signed DEB package feeds [PATCH v2 1/3] gpg-sign: Add parameters to gpg signature function [PATCH v2 2/3] package_manager: sign DEB package feeds [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds" enable signed deb package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

These patches makes deb a first class citizen as ipk and rpm.

Patches have been in use in meta-intel-edison since Gatesgarth,
see https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-repository.html

Changes in V2:
- Added runtime test for signed deb package feeds (Richard Purdie)

author
#1 lore raw

PATCH v2 1/3 gpg-sign: Add parameters to gpg signature function

From: Xavier Berger <[email protected]>

output_suffix: If defined, add output_suffix as file name extension.
use_sha256: If True, use sha256 for gpg as digest algorithm

---
 meta/lib/oe/gpg_sign.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/gpg_sign.py +5 -1
index 1bce6cb792..aa9bb49f2c 100644
@@ -58,7 +58,7 @@ class LocalSigner(object):
         for i in range(0, len(files), sign_chunk):
             subprocess.check_output(shlex.split(cmd + ' '.join(files[i:i+sign_chunk])), stderr=subprocess.STDOUT)
 
-    def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True):
+    def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True, output_suffix=None, use_sha256=False):
         """Create a detached signature of a file"""
 
         if passphrase_file and passphrase:
@@ -71,6 +71,10 @@ class LocalSigner(object):
             cmd += ['--homedir', self.gpg_path]
         if armor:
             cmd += ['--armor']
+        if output_suffix:
+            cmd += ['-o', input_file + "." + output_suffix]
+        if use_sha256:
+            cmd += ['--digest-algo', "SHA256"]
 
         #gpg > 2.1 supports password pipes only through the loopback interface
         #gpg < 2.1 errors out if given unknown parameters
Signature · 1 line
2.32.0
author
#2 lore raw

v2 cover *** SUBJECT HERE ***

In reply to Ferry Toth

From: Ferry Toth <[email protected]>

*** BLURB HERE ***

Ferry Toth (2):
package_manager: sign DEB package feeds
apt: add apt selftest to test signed package feeds

Xavier Berger (1):
gpg-sign: Add parameters to gpg signature function

 meta/lib/oe/gpg_sign.py                      |  6 +++-
 meta/lib/oe/package_manager/deb/__init__.py  | 19 ++++++++--
 meta/lib/oeqa/runtime/cases/apt.py           | 16 ++++++---
 meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
 4 files changed, 70 insertions(+), 9 deletions(-)
Signature · 1 line
2.32.0
author
#3 lore raw

PATCH v2 2/3 package_manager: sign DEB package feeds

In reply to Ferry Toth

From: Ferry Toth <[email protected]>

Implement debian package repository signature.
For each Release file created in repository subdirectory, a signature Release.gpg is created.

Signature is performed using gpg backend when the following variables are set in local.conf:
PACKAGE_CLASSES += "sign_package_feed" PACKAGE_FEED_GPG_NAME = "<Id of GPG key>" PACKAGE_FEED_GPG_PASSPHRASE_FILE="<path to password file>"

---
 meta/lib/oe/package_manager/deb/__init__.py | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
meta/lib/oe/package_manager/deb/__init__.py +16 -3
index 9f112ae25b..86ddb130ad 100644
@@ -53,6 +53,7 @@ class DpkgIndexer(Indexer):
 
         index_cmds = []
         deb_dirs_found = False
+        index_sign_files = set()
         for arch in arch_list:
             arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
@@ -62,7 +63,10 @@ class DpkgIndexer(Indexer):
 
             cmd += "%s -fcn Packages > Packages.gz;" % gzip
 
-            with open(os.path.join(arch_dir, "Release"), "w+") as release:
+            release_file = os.path.join(arch_dir, "Release")
+            index_sign_files.add(release_file)
+
+            with open(release_file, "w+") as release:
                 release.write("Label: %s\n" % arch)
 
             cmd += "PSEUDO_UNLOAD=1 %s release . >> Release" % apt_ftparchive
@@ -76,8 +80,17 @@ class DpkgIndexer(Indexer):
             return
 
         oe.utils.multiprocess_launch(create_index, index_cmds, self.d)
-        if self.d.getVar('PACKAGE_FEED_SIGN') == '1':
-            raise NotImplementedError('Package feed signing not implementd for dpkg')
+        if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
+            signer = get_signer(self.d, self.d.getVar('PACKAGE_FEED_GPG_BACKEND', True))
+        else:
+            signer = None
+        if signer:
+            for f in index_sign_files:
+                signer.detach_sign(f,
+                                   self.d.getVar('PACKAGE_FEED_GPG_NAME', True),
+                                   self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True),
+                                   output_suffix="gpg",
+                                   use_sha256=True)
 
 class PMPkgsList(PkgsList):
 
Signature · 1 line
2.32.0
author
#4 lore raw

PATCH v2 3/3 apt: add apt selftest to test signed package feeds

In reply to Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

---
 meta/lib/oeqa/runtime/cases/apt.py           | 16 ++++++---
 meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
 2 files changed, 49 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +11 -5
index 53745df93f..49f8714730 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
     @classmethod
     def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
         cls.repo_server = HTTPService(service_repo,
                                       '0.0.0.0', port=cls.tc.target.server_port,
                                       logger=cls.tc.logger)
@@ -32,13 +32,18 @@ class AptRepoTest(AptTest):
         cls.repo_server.stop()
 
     def setup_source_config_for_package_install(self):
-        apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
     def cleanup_source_config_for_package_install(self):
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
 
     @skipIfNotFeature('package-management',
                       'Test requires package-management to be in IMAGE_FEATURES')
@@ -47,7 +52,8 @@ class AptRepoTest(AptTest):
     @OEHasPackage(['apt'])
     def test_apt_install_from_repo(self):
         self.setup_source_config_for_package_install()
+        self.setup_key()
         self.pkg('update')
         self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+        self.pkg('install --yes run-postinsts-dev')
         self.cleanup_source_config_for_package_install()
meta/lib/oeqa/selftest/cases/runtime_test.py +38 -0
index 642f0eb637..7a75b95a99 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
         bitbake('core-image-full-cmdline socat')
         bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
     def test_testimage_virgl_gtk_sdl(self):
         """
         Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends
Signature · 1 line
2.32.0
#5 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Quoted 101 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

---
 meta/lib/oeqa/runtime/cases/apt.py           | 16 ++++++---
 meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
 2 files changed, 49 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +11 -5
index 53745df93f..49f8714730 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
     @classmethod
     def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
         cls.repo_server = HTTPService(service_repo,
                                       '0.0.0.0', port=cls.tc.target.server_port,
                                       logger=cls.tc.logger)
@@ -32,13 +32,18 @@ class AptRepoTest(AptTest):
         cls.repo_server.stop()
 
     def setup_source_config_for_package_install(self):
-        apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
     def cleanup_source_config_for_package_install(self):
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
 
     @skipIfNotFeature('package-management',
                       'Test requires package-management to be in IMAGE_FEATURES')
@@ -47,7 +52,8 @@ class AptRepoTest(AptTest):
     @OEHasPackage(['apt'])
     def test_apt_install_from_repo(self):
         self.setup_source_config_for_package_install()
+        self.setup_key()
         self.pkg('update')
         self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+        self.pkg('install --yes run-postinsts-dev')
         self.cleanup_source_config_for_package_install()
meta/lib/oeqa/selftest/cases/runtime_test.py +31 -0
index 642f0eb637..7a75b95a99 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
         bitbake('core-image-full-cmdline socat')
         bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home

meta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205

Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7 -0
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
     def test_testimage_virgl_gtk_sdl(self):
         """
         Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends

Thanks for working on this!

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975

however hopefully these shouldn't be too hard to fix?

The rest of the build is still running.

Cheers,

Richard

author
#6 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Hi,

Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 115 lines · Ferry Toth
Quoted 101 lines

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

---
  meta/lib/oeqa/runtime/cases/apt.py           | 16 ++++++---
  meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
  2 files changed, 49 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +11 -5
index 53745df93f..49f8714730 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
      @classmethod
      def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
          cls.repo_server = HTTPService(service_repo,
                                        '0.0.0.0', port=cls.tc.target.server_port,
                                        logger=cls.tc.logger)
@@ -32,13 +32,18 @@ class AptRepoTest(AptTest):
          cls.repo_server.stop()
 
      def setup_source_config_for_package_install(self):
-        apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
          apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
      def cleanup_source_config_for_package_install(self):
          apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
 
      @skipIfNotFeature('package-management',
                        'Test requires package-management to be in IMAGE_FEATURES')
@@ -47,7 +52,8 @@ class AptRepoTest(AptTest):
      @OEHasPackage(['apt'])
      def test_apt_install_from_repo(self):
          self.setup_source_config_for_package_install()
+        self.setup_key()
          self.pkg('update')
          self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+        self.pkg('install --yes run-postinsts-dev')
          self.cleanup_source_config_for_package_install()
meta/lib/oeqa/selftest/cases/runtime_test.py +31 -0
index 642f0eb637..7a75b95a99 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
          bitbake('core-image-full-cmdline socat')
          bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home

meta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205

Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7 -0
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
      def test_testimage_virgl_gtk_sdl(self):
          """
          Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends

Thanks for working on this!

Quoted 5 lines

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975

That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

What could cause this? Because the complaint is it can't remove the package because it was not installed.

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

Quoted 8 lines

however hopefully these shouldn't be too hard to fix?

The rest of the build is still running.

Cheers,

Richard

#7 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Quoted 133 lines · Ferry Toth

Hi,

Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 115 lines · Ferry Toth
Quoted 101 lines

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

---
  meta/lib/oeqa/runtime/cases/apt.py           | 16 ++++++---
  meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
  2 files changed, 49 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +11 -5
index 53745df93f..49f8714730 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
      @classmethod
      def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
          cls.repo_server = HTTPService(service_repo,
                                        '0.0.0.0', port=cls.tc.target.server_port,
                                        logger=cls.tc.logger)
@@ -32,13 +32,18 @@ class AptRepoTest(AptTest):
          cls.repo_server.stop()
 
      def setup_source_config_for_package_install(self):
-        apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
          apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
      def cleanup_source_config_for_package_install(self):
          apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
 
      @skipIfNotFeature('package-management',
                        'Test requires package-management to be in IMAGE_FEATURES')
@@ -47,7 +52,8 @@ class AptRepoTest(AptTest):
      @OEHasPackage(['apt'])
      def test_apt_install_from_repo(self):
          self.setup_source_config_for_package_install()
+        self.setup_key()
          self.pkg('update')
          self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+        self.pkg('install --yes run-postinsts-dev')
          self.cleanup_source_config_for_package_install()
meta/lib/oeqa/selftest/cases/runtime_test.py +31 -0
index 642f0eb637..7a75b95a99 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
          bitbake('core-image-full-cmdline socat')
          bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home

meta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205

Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7 -0
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
      def test_testimage_virgl_gtk_sdl(self):
          """
          Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends

Thanks for working on this!

Quoted 5 lines

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975

That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

Quoted 8 lines

What could cause this? Because the complaint is it can't remove the package because it was not installed.

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

I think the issue is you edited testimage which is a different set of tests which aren't just called by oe-selftest but by things like

"bitbake core-image-sato -c testimage"

as well. I'd suggest making the changes in testimage conditional on signing being configured.

Cheers,

Richard

author
#8 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Hi,

Op 04-04-2022 om 22:39 schreef Richard Purdie:

Quoted 135 lines · Ferry Toth
Quoted 134 lines

Hi,

Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 114 lines · Ferry Toth
Quoted 101 lines

From: Ferry Toth<[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

---
   meta/lib/oeqa/runtime/cases/apt.py           | 16 ++++++---
   meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
   2 files changed, 49 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +11 -5
index 53745df93f..49f8714730 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
       @classmethod
       def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
           cls.repo_server = HTTPService(service_repo,
                                         '0.0.0.0', port=cls.tc.target.server_port,
                                         logger=cls.tc.logger)
@@ -32,13 +32,18 @@ class AptRepoTest(AptTest):
           cls.repo_server.stop()
 
       def setup_source_config_for_package_install(self):
-        apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
           apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
       def cleanup_source_config_for_package_install(self):
           apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
 
       @skipIfNotFeature('package-management',
                         'Test requires package-management to be in IMAGE_FEATURES')
@@ -47,7 +52,8 @@ class AptRepoTest(AptTest):
       @OEHasPackage(['apt'])
       def test_apt_install_from_repo(self):
           self.setup_source_config_for_package_install()
+        self.setup_key()
           self.pkg('update')
           self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+        self.pkg('install --yes run-postinsts-dev')
           self.cleanup_source_config_for_package_install()
meta/lib/oeqa/selftest/cases/runtime_test.py +31 -0
index 642f0eb637..7a75b95a99 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
           bitbake('core-image-full-cmdline socat')
           bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth<[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home

meta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205

Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7 -0
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
       def test_testimage_virgl_gtk_sdl(self):
           """
           Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends

Thanks for working on this!

Quoted 5 lines

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975

That is weird, do I understand correctly that it fails on:

   apt-get remove --yes run-postinsts-dev

Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

What could cause this? Because the complaint is it can't remove the package because it was not installed.

Quoted 8 lines
Quoted 5 lines

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

I think the issue is you edited testimage which is a different set of tests which aren't just called by oe-selftest but by things like

That would be my first thought too, but...

because the failure seems to be on the line self.pkg('remove --yes run-postinsts-dev'),  that would mean the line self.pkg('update') passed.

And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week).

So, there may be a second thing wrong?

Do you know where I can find the log files referred to:

<..>tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/temp/log.do_testimage.35553

<..>tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/temp/log.do_testimage.35362

or could we do a 'quick' check by changing

        self.pkg('update')
        self.pkg('remove --yes run-postinsts-dev')
        self.pkg('install --yes run-postinsts-dev')
to
        self.pkg('update')
        self.pkg('install --yes run-postinsts-dev')
        self.pkg('remove --yes run-postinsts-dev')

?

Quoted 5 lines

"bitbake core-image-sato -c testimage"

as well. I'd suggest making the changes in testimage conditional on signing being configured.

Yes, regardless the above, we need to either make signing always enabled in all test cases or detect whether signing is used.

Do you have a hint if there is a variable to test in class AptRepoTest if PACKAGE_FEED_GPG_NAME has been set?

Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.

What would you prefer?

Quoted 5 lines

Cheers,

Richard

#9 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Quoted 29 lines · Ferry Toth

Op 04-04-2022 om 22:39 schreef Richard Purdie:

Quoted 27 lines · Ferry Toth

 Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 25 lines
Quoted 8 lines · Ferry Toth

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975

That is weird, do I understand correctly that it fails on:
  apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

What could cause this? Because the complaint is it can't remove the package because it was not installed.

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

Quoted 8 lines
Quoted 2 lines

I think the issue is you edited testimage which is a different set of tests which aren't just called by oe-selftest but by things like

That would be my first thought too, but...
because the failure seems to be on the line self.pkg('remove --yes run- postinsts-dev'),  that would mean the line self.pkg('update') passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?

I was easily able to reproduce this locally and it shows the setup_source_config_for_package_install() step fails and hence the sources aren't setup correctly, hence the update probably works.

Quoted 5 lines

Do you know where I can find the log files referred to:
<..>tmp/work/qemux86-poky-linux/core-image-sato/1.0- r0/temp/log.do_testimage.35553 <..>tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0- r0/temp/log.do_testimage.35362

We can get them off the autobuilder if needed but someone would have to manually go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with package_deb set as the backend.

Quoted 9 lines

or could we do a 'quick' check by changing self.pkg('update')
        self.pkg('remove --yes run-postinsts-dev') self.pkg('install --yes run-postinsts-dev') to
        self.pkg('update') self.pkg('install --yes run-postinsts-dev') self.pkg('remove --yes run-postinsts-dev') ?

I'm not convinced that would help us...

Quoted 6 lines

Quoted 5 lines

"bitbake core-image-sato -c testimage"

as well. I'd suggest making the changes in testimage conditional on signing being configured.

Quoted 8 lines

Yes, regardless the above, we need to either make signing always enabled in all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest if PACKAGE_FEED_GPG_NAME has been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?

We should be able to test self.tc.td.get('PACKAGE_FEED_GPG_NAME') in the test and handle accordingly?

I did merge the base changes into the release since I thought it was fair to get the fixes in before it was built. We just need to get the test sorted now, I think it is close.

Cheers,

Richard

author
#10 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Hi,

Op 06-04-2022 om 13:40 schreef Richard Purdie:

Quoted 36 lines · Ferry Toth
Quoted 27 lines

Op 04-04-2022 om 22:39 schreef Richard Purdie:

Quoted 25 lines · Ferry Toth

 Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 24 lines
Quoted 7 lines · Ferry Toth

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975

That is weird, do I understand correctly that it fails on:
  apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

What could cause this? Because the complaint is it can't remove the package because it was not installed.

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

Quoted 8 lines
Quoted 2 lines

I think the issue is you edited testimage which is a different set of tests which aren't just called by oe-selftest but by things like

That would be my first thought too, but...
because the failure seems to be on the line self.pkg('remove --yes run- postinsts-dev'),  that would mean the line self.pkg('update') passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?

Quoted 3 lines

I was easily able to reproduce this locally and it shows the setup_source_config_for_package_install() step fails and hence the sources aren't setup correctly, hence the update probably works.

not correct, hence works. You lost me here, but I'll try to reproduce.

Quoted 6 lines
Quoted 5 lines

Do you know where I can find the log files referred to:
<..>tmp/work/qemux86-poky-linux/core-image-sato/1.0- r0/temp/log.do_testimage.35553 <..>tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0- r0/temp/log.do_testimage.35362

We can get them off the autobuilder if needed but someone would have to manually

No, that would be too much work. I'll try to reproduce myself.

Quoted 2 lines

go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with package_deb set as the backend.

..in conf. But without PACKAGE_CLASSES, PACKAGE_FEED_GPG_NAME, PACKAGE_FEED_GPG_PASSPHRASE_FILE?

Quoted 10 lines
Quoted 9 lines

or could we do a 'quick' check by changing
        self.pkg('update')
        self.pkg('remove --yes run-postinsts-dev') self.pkg('install --yes run-postinsts-dev')
to
        self.pkg('update') self.pkg('install --yes run-postinsts-dev') self.pkg('remove --yes run-postinsts-dev')
?

I'm not convinced that would help us...

I'll try locally.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines

"bitbake core-image-sato -c testimage"

as well. I'd suggest making the changes in testimage conditional on signing being configured.

Quoted 8 lines

Yes, regardless the above, we need to either make signing always enabled in all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest if PACKAGE_FEED_GPG_NAME has been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?

Quoted 6 lines

We should be able to test self.tc.td.get('PACKAGE_FEED_GPG_NAME') in the test and handle accordingly?

I did merge the base changes into the release since I thought it was fair to get the fixes in before it was built. We just need to get the test sorted now, I think it is close.

Thanks for merging.

I'll fix the test, that's only fair.

One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?

Quoted 7 lines

Cheers,

Richard

#11 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Quoted 40 lines · Ferry Toth

Op 06-04-2022 om 13:40 schreef Richard Purdie:

Quoted 39 lines · Ferry Toth
Quoted 38 lines

Op 04-04-2022 om 22:39 schreef Richard Purdie:

Quoted 22 lines · Ferry Toth

 Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 21 lines
Quoted 9 lines · Ferry Toth

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/50
13/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/49
75

That is weird, do I understand correctly that it fails on:
  apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

What could cause this? Because the complaint is it can't remove the package because it was not installed.

Quoted 8 lines
Quoted 5 lines

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

I think the issue is you edited testimage which is a different set of tests
which aren't just called by oe-selftest but by things like

That would be my first thought too, but...
because the failure seems to be on the line self.pkg('remove --yes run- postinsts-dev'),  that would mean the line self.pkg('update') passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?

Quoted 4 lines
Quoted 3 lines

I was easily able to reproduce this locally and it shows the setup_source_config_for_package_install() step fails and hence the sources aren't setup correctly, hence the update probably works.

 not correct, hence works. You lost me here, but I'll try to reproduce.

I mean the command doesn't work correctly. In my local logs I see:

DEBUG: Command: cd /etc/apt/; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/http:\/\/192.168.7.1:46599/g' sources.list
Status: 1 Output: cp: can't stat 'sources.list': No such file or directory
sed: sources.list: No such file or directory

Quoted 6 lines

Quoted 2 lines

go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with package_deb set as the backend.

..in conf. But without PACKAGE_CLASSES, PACKAGE_FEED_GPG_NAME, PACKAGE_FEED_GPG_PASSPHRASE_FILE?

Yes.

Quoted 18 lines
Quoted 10 lines
Quoted 10 lines

Yes, regardless the above, we need to either make signing always enabled in
all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest if PACKAGE_FEED_GPG_NAME has been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?

Quoted 8 lines

We should be able to test self.tc.td.get('PACKAGE_FEED_GPG_NAME') in the test
and handle accordingly?

I did merge the base changes into the release since I thought it was fair to get
the fixes in before it was built. We just need to get the test sorted now, I think it is close.

Quoted 6 lines

Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?

No, the autobuilder runs all the tests in oe-selftest so it should be covered (and is why we saw the failures on the autobuilder).

Cheers,

Richard

author
#12 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Hi

Op 06-04-2022 om 17:23 schreef Richard Purdie:

Quoted 75 lines · Ferry Toth
Quoted 40 lines

Op 06-04-2022 om 13:40 schreef Richard Purdie:

Quoted 39 lines · Ferry Toth
Quoted 38 lines

Op 04-04-2022 om 22:39 schreef Richard Purdie:

Quoted 22 lines · Ferry Toth

 Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 21 lines
Quoted 9 lines · Ferry Toth

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/50
13/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/49
75

That is weird, do I understand correctly that it fails on:
  apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

What could cause this? Because the complaint is it can't remove the package because it was not installed.

Quoted 8 lines
Quoted 5 lines

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

I think the issue is you edited testimage which is a different set of tests
which aren't just called by oe-selftest but by things like

That would be my first thought too, but...
because the failure seems to be on the line self.pkg('remove --yes run- postinsts-dev'),  that would mean the line self.pkg('update') passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?

Quoted 4 lines
Quoted 3 lines

I was easily able to reproduce this locally and it shows the setup_source_config_for_package_install() step fails and hence the sources aren't setup correctly, hence the update probably works.

 not correct, hence works. You lost me here, but I'll try to reproduce.

I mean the command doesn't work correctly. In my local logs I see:

DEBUG: Command: cd /etc/apt/; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/http:\/\/192.168.7.1:46599/g' sources.list
Status: 1 Output: cp: can't stat 'sources.list': No such file or directory
sed: sources.list: No such file or directory

Quoted 5 lines
Quoted 3 lines

go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with package_deb set as the backend.

..in conf. But without PACKAGE_CLASSES, PACKAGE_FEED_GPG_NAME, PACKAGE_FEED_GPG_PASSPHRASE_FILE?

Yes.

Quoted 17 lines
Quoted 17 lines
Quoted 9 lines

Yes, regardless the above, we need to either make signing always enabled in
all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest if PACKAGE_FEED_GPG_NAME has been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?

We should be able to test self.tc.td.get('PACKAGE_FEED_GPG_NAME') in the test
and handle accordingly?

I did merge the base changes into the release since I thought it was fair to get
the fixes in before it was built. We just need to get the test sorted now, I think it is close.

Quoted 8 lines
Quoted 6 lines

Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?

No, the autobuilder runs all the tests in oe-selftest so it should be covered (and is why we saw the failures on the autobuilder).

I was running 'oe-selftest -K -r runtime_test.TestImage.test_testimage_apt' whereas buildbot seems to be running 'apt.AptRepoTest.test_apt_install_from_repo' directly.

However, test_testimage_apt is where keys are setup (keys found in meta-selftest). So, where/when is test_testimage_apt called (or test_testimage_dnf for that matter)?

Quoted 4 lines

Cheers,

Richard

#13 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Quoted 84 lines · Ferry Toth

Hi

Op 06-04-2022 om 17:23 schreef Richard Purdie:

Quoted 81 lines · Ferry Toth
Quoted 40 lines

Op 06-04-2022 om 13:40 schreef Richard Purdie:

Quoted 39 lines · Ferry Toth
Quoted 38 lines

Op 04-04-2022 om 22:39 schreef Richard Purdie:

Quoted 22 lines · Ferry Toth

 Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 21 lines
Quoted 9 lines · Ferry Toth

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/50
13/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/49
75

That is weird, do I understand correctly that it fails on:
  apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

What could cause this? Because the complaint is it can't remove the package because it was not installed.

Quoted 8 lines
Quoted 5 lines

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

I think the issue is you edited testimage which is a different set of tests
which aren't just called by oe-selftest but by things like

That would be my first thought too, but...
because the failure seems to be on the line self.pkg('remove --yes run- postinsts-dev'),  that would mean the line self.pkg('update') passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?

Quoted 4 lines
Quoted 3 lines

I was easily able to reproduce this locally and it shows the setup_source_config_for_package_install() step fails and hence the sources aren't setup correctly, hence the update probably works.

 not correct, hence works. You lost me here, but I'll try to reproduce.

I mean the command doesn't work correctly. In my local logs I see:

DEBUG: Command: cd /etc/apt/; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/http:\/\/192.168.7.1:46599/g' sources.list
Status: 1 Output: cp: can't stat 'sources.list': No such file or directory
sed: sources.list: No such file or directory

Quoted 5 lines
Quoted 3 lines

go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with package_deb set as the backend.

..in conf. But without PACKAGE_CLASSES, PACKAGE_FEED_GPG_NAME, PACKAGE_FEED_GPG_PASSPHRASE_FILE?

Yes.

Quoted 23 lines
Quoted 9 lines
Quoted 9 lines

Yes, regardless the above, we need to either make signing always enabled in
all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest if PACKAGE_FEED_GPG_NAME has been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?

Quoted 8 lines

We should be able to test self.tc.td.get('PACKAGE_FEED_GPG_NAME') in the test
and handle accordingly?

I did merge the base changes into the release since I thought it was fair to get
the fixes in before it was built. We just need to get the test sorted now, I think it is close.

Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?

Quoted 6 lines
Quoted 2 lines

No, the autobuilder runs all the tests in oe-selftest so it should be covered (and is why we saw the failures on the autobuilder).

I was running 'oe-selftest -K -r runtime_test.TestImage.test_testimage_apt' whereas buildbot seems to be running 'apt.AptRepoTest.test_apt_install_from_repo' directly.

We have several types of test. There are two types in play here,

"oe-selftest -K -r runtime_test.TestImage.test_testimage_apt"

and

"bitbake core-image-sato -c testimage"

The latter testimage tests are often run every time we create images and running testimage will trigger 'apt.AptRepoTest.test_apt_install_from_repo' if the image has apt present and is built using debian package management.

We also run oe-selftest which triggers a testimage of it's own for a specfic test case.

So we'd expect the normal testimage calls to not have the package signing enabled and then we'd have the oe-selftest which specifically tests signing.

Quoted 3 lines

However, test_testimage_apt is where keys are setup (keys found in meta-selftest). So, where/when is test_testimage_apt called (or test_testimage_dnf for that matter)?

Those are called by the oe-selftest call on the autobuilder. We run the oe- selftest with a mask on the autobuilder so pretty much all of them run.

Cheers,

Richard

author
#14 lore raw

Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Op 06-04-2022 om 23:05 schreef Richard Purdie:

Quoted 71 lines · Ferry Toth
Quoted 70 lines

Hi

Op 06-04-2022 om 17:23 schreef Richard Purdie:

Quoted 67 lines · Ferry Toth
Quoted 40 lines

Op 06-04-2022 om 13:40 schreef Richard Purdie:

Quoted 39 lines · Ferry Toth
Quoted 38 lines

Op 04-04-2022 om 22:39 schreef Richard Purdie:

Quoted 22 lines · Ferry Toth

 Op 04-04-2022 om 15:58 schreef Richard Purdie:

Quoted 21 lines
Quoted 9 lines · Ferry Toth

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/50
13/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/49
75

That is weird, do I understand correctly that it fails on:
  apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev

That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.

What could cause this? Because the complaint is it can't remove the package because it was not installed.

Quoted 8 lines
Quoted 5 lines

It would be trivial to remove the line

*) self.pkg('remove --yes run-postinsts-dev')

but how could it have passed the test before?

I think the issue is you edited testimage which is a different set of tests
which aren't just called by oe-selftest but by things like

That would be my first thought too, but...
because the failure seems to be on the line self.pkg('remove --yes run- postinsts-dev'),  that would mean the line self.pkg('update') passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?

Quoted 4 lines
Quoted 3 lines

I was easily able to reproduce this locally and it shows the setup_source_config_for_package_install() step fails and hence the sources aren't setup correctly, hence the update probably works.

 not correct, hence works. You lost me here, but I'll try to reproduce.

I mean the command doesn't work correctly. In my local logs I see:

DEBUG: Command: cd /etc/apt/; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/http:\/\/192.168.7.1:46599/g' sources.list
Status: 1 Output: cp: can't stat 'sources.list': No such file or directory
sed: sources.list: No such file or directory

Quoted 5 lines
Quoted 3 lines

go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with package_deb set as the backend.

..in conf. But without PACKAGE_CLASSES, PACKAGE_FEED_GPG_NAME, PACKAGE_FEED_GPG_PASSPHRASE_FILE?

Yes.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Yes, regardless the above, we need to either make signing always enabled in
all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest if PACKAGE_FEED_GPG_NAME has been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

We should be able to test self.tc.td.get('PACKAGE_FEED_GPG_NAME') in the test
and handle accordingly?

I'll try this for autodetecting the test case.

Quoted 16 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

I did merge the base changes into the release since I thought it was fair to get
the fixes in before it was built. We just need to get the test sorted now, I think it is close.

Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?

Quoted 5 lines
Quoted 2 lines

No, the autobuilder runs all the tests in oe-selftest so it should be covered (and is why we saw the failures on the autobuilder).

I was running 'oe-selftest -K -r runtime_test.TestImage.test_testimage_apt' whereas buildbot seems to be running 'apt.AptRepoTest.test_apt_install_from_repo' directly.

Quoted 7 lines

We have several types of test. There are two types in play here,

"oe-selftest -K -r runtime_test.TestImage.test_testimage_apt"

and

"bitbake core-image-sato -c testimage"

Thanks for the explanation. Yesterday evening I built Sato and was able to reproduce  the issue.

(that was a big build with half of build time spent on building Rust).

core-image-sato does not have a /etc/apt/sources.list, the selftest public key is not on the image and gpg is not installed. This explains the errors in the log.

Quoted 6 lines

The latter testimage tests are often run every time we create images and running testimage will trigger 'apt.AptRepoTest.test_apt_install_from_repo' if the image has apt present and is built using debian package management.

We also run oe-selftest which triggers a testimage of it's own for a specfic test case.

Got it.

Quoted 8 lines

So we'd expect the normal testimage calls to not have the package signing enabled and then we'd have the oe-selftest which specifically tests signing.

Quoted 3 lines

However, test_testimage_apt is where keys are setup (keys found in meta-selftest). So, where/when is test_testimage_apt called (or test_testimage_dnf for that matter)?

Those are called by the oe-selftest call on the autobuilder. We run the oe- selftest with a mask on the autobuilder so pretty much all of them run.

After looking at more detail at autobuilder schedules I get it.

Quoted 5 lines

Cheers,

Richard

#15 lore raw

Re: [OE-core] [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

In reply to Richard Purdie

Hello,

Quoted 118 lines · Richard Purdie
Quoted 101 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

---
 meta/lib/oeqa/runtime/cases/apt.py           | 16 ++++++---
 meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
 2 files changed, 49 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +11 -5
index 53745df93f..49f8714730 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
     @classmethod
     def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
         cls.repo_server = HTTPService(service_repo,
                                       '0.0.0.0', port=cls.tc.target.server_port,
                                       logger=cls.tc.logger)
@@ -32,13 +32,18 @@ class AptRepoTest(AptTest):
         cls.repo_server.stop()
 
     def setup_source_config_for_package_install(self):
-        apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
     def cleanup_source_config_for_package_install(self):
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
 
     @skipIfNotFeature('package-management',
                       'Test requires package-management to be in IMAGE_FEATURES')
@@ -47,7 +52,8 @@ class AptRepoTest(AptTest):
     @OEHasPackage(['apt'])
     def test_apt_install_from_repo(self):
         self.setup_source_config_for_package_install()
+        self.setup_key()
         self.pkg('update')
         self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+        self.pkg('install --yes run-postinsts-dev')
         self.cleanup_source_config_for_package_install()
meta/lib/oeqa/selftest/cases/runtime_test.py +31 -0
index 642f0eb637..7a75b95a99 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
         bitbake('core-image-full-cmdline socat')
         bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home

meta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205

Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7 -0
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
     def test_testimage_virgl_gtk_sdl(self):
         """
         Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends

Thanks for working on this!

Looking at the patches I wondered if this would break testimage and unfortunately it does:

Quoted 6 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975

however hopefully these shouldn't be too hard to fix?

The rest of the build is still running.

I missed it at the time but I believe this is also the cause of:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3352/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: GPG exited with code 2: gpg: can't connect to the agent: IPC connect call failed
gpg: skipped "testuser": No secret key
gpg: signing failed: No secret key

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
author
#16 lore raw

Re: [OE-core] [PATCH v2 3/3] apt: add apt selftest to test signed package feeds

Hi

Op 06-04-2022 om 12:10 schreef Alexandre Belloni:

Quoted 125 lines

Hello,

Quoted 122 lines · Richard Purdie
Quoted 101 lines · Ferry Toth

From: Ferry Toth<[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

---
  meta/lib/oeqa/runtime/cases/apt.py           | 16 ++++++---
  meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
  2 files changed, 49 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +11 -5
index 53745df93f..49f8714730 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
      @classmethod
      def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
          cls.repo_server = HTTPService(service_repo,
                                        '0.0.0.0', port=cls.tc.target.server_port,
                                        logger=cls.tc.logger)
@@ -32,13 +32,18 @@ class AptRepoTest(AptTest):
          cls.repo_server.stop()
 
      def setup_source_config_for_package_install(self):
-        apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
          apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
      def cleanup_source_config_for_package_install(self):
          apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
 
      @skipIfNotFeature('package-management',
                        'Test requires package-management to be in IMAGE_FEATURES')
@@ -47,7 +52,8 @@ class AptRepoTest(AptTest):
      @OEHasPackage(['apt'])
      def test_apt_install_from_repo(self):
          self.setup_source_config_for_package_install()
+        self.setup_key()
          self.pkg('update')
          self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+        self.pkg('install --yes run-postinsts-dev')
          self.cleanup_source_config_for_package_install()
meta/lib/oeqa/selftest/cases/runtime_test.py +31 -0
index 642f0eb637..7a75b95a99 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
          bitbake('core-image-full-cmdline socat')
          bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth<[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home

meta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205

Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7 -0
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
      def test_testimage_virgl_gtk_sdl(self):
          """
          Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends

Thanks for working on this!

Looking at the patches I wondered if this would break testimage and unfortunately it does:

https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975

however hopefully these shouldn't be too hard to fix?

Quoted 8 lines
Quoted 1 line

The rest of the build is still running.

I missed it at the time but I believe this is also the cause of:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3352/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: GPG exited with code 2: gpg: can't connect to the agent: IPC connect call failed
gpg: skipped "testuser": No secret key
gpg: signing failed: No secret key

This seems related but not exact the same.

It seems do_package_index wants to generate a signed deb repo but no key is provided. But IIUC you have PACKAGE_CLASSES = "package_rpm", so why is runtime_test.TestImage.test_testimage_apt run?