Skip to contentSkip to conversation

package_manager: support for signed DEB package feeds

PATCH Superseded Ferry Toth 3 participants 11 messages Apr 11, 2022 – Apr 13, 2022

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Ferry Toth

author
#0 lore raw

[PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds" (already in master) enable signed deb package feeds. When called from
oe-selftest -r runtime_test.TestImage.test_testimage_apt this patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

Changes in V3:
- When called from bitbake core-image-sato -c testimage package feed is unsigned. Auto-detect this case and behave as before (Richard Purdie)

Changes in V2:
- Added runtime test for signed deb package feeds (Richard Purdie)

author
#1 lore raw

PATCH v3 1/1 apt: add apt selftest to test signed package feeds

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

---
 meta/lib/oeqa/runtime/cases/apt.py           | 38 ++++++++++++++++----
 meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
 2 files changed, 69 insertions(+), 7 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +31 -7
index 53745df93f..574a34f148 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
     @classmethod
     def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
         cls.repo_server = HTTPService(service_repo,
                                       '0.0.0.0', port=cls.tc.target.server_port,
                                       logger=cls.tc.logger)
@@ -34,20 +34,44 @@ class AptRepoTest(AptTest):
     def setup_source_config_for_package_install(self):
         apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s/all ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+
+    def setup_source_config_for_package_install_signed(self):
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
     def cleanup_source_config_for_package_install(self):
         apt_get_sourceslist_dir = '/etc/apt/'
         self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
 
+    def cleanup_source_config_for_package_install_signed(self):
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
+
     @skipIfNotFeature('package-management',
                       'Test requires package-management to be in IMAGE_FEATURES')
     @skipIfNotDataVar('IMAGE_PKGTYPE', 'deb',
                       'DEB is not the primary package manager')
     @OEHasPackage(['apt'])
     def test_apt_install_from_repo(self):
-        self.setup_source_config_for_package_install()
-        self.pkg('update')
-        self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
-        self.cleanup_source_config_for_package_install()
+        if not self.tc.td.get('PACKAGE_FEED_GPG_NAME'):
+            self.setup_source_config_for_package_install()
+            self.pkg('update')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+            self.cleanup_source_config_for_package_install()
+        else:
+            # when we are here a key has been set to sign the package feed and
+            # public key and gnupg installed on the image by test_testimage_apt
+            self.setup_source_config_for_package_install_signed()
+            self.setup_key()
+            self.pkg('update')
+            self.pkg('install --yes run-postinsts-dev')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.cleanup_source_config_for_package_install_signed()
meta/lib/oeqa/selftest/cases/runtime_test.py +38 -0
index 2ad89490fc..3ece617cb0 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
         bitbake('core-image-full-cmdline socat')
         bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
     def test_testimage_virgl_gtk_sdl(self):
         """
         Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends
Signature · 1 line
2.32.0
#2 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Hello,

Quoted 11 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed
ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

Quoted 126 lines
---
 meta/lib/oeqa/runtime/cases/apt.py           | 38 ++++++++++++++++----
 meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
 2 files changed, 69 insertions(+), 7 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +31 -7
index 53745df93f..574a34f148 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
     @classmethod
     def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
         cls.repo_server = HTTPService(service_repo,
                                       '0.0.0.0', port=cls.tc.target.server_port,
                                       logger=cls.tc.logger)
@@ -34,20 +34,44 @@ class AptRepoTest(AptTest):
     def setup_source_config_for_package_install(self):
         apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
         apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s/all ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+
+    def setup_source_config_for_package_install_signed(self):
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
     def cleanup_source_config_for_package_install(self):
         apt_get_sourceslist_dir = '/etc/apt/'
         self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
 
+    def cleanup_source_config_for_package_install_signed(self):
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
+
     @skipIfNotFeature('package-management',
                       'Test requires package-management to be in IMAGE_FEATURES')
     @skipIfNotDataVar('IMAGE_PKGTYPE', 'deb',
                       'DEB is not the primary package manager')
     @OEHasPackage(['apt'])
     def test_apt_install_from_repo(self):
-        self.setup_source_config_for_package_install()
-        self.pkg('update')
-        self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
-        self.cleanup_source_config_for_package_install()
+        if not self.tc.td.get('PACKAGE_FEED_GPG_NAME'):
+            self.setup_source_config_for_package_install()
+            self.pkg('update')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+            self.cleanup_source_config_for_package_install()
+        else:
+            # when we are here a key has been set to sign the package feed and
+            # public key and gnupg installed on the image by test_testimage_apt
+            self.setup_source_config_for_package_install_signed()
+            self.setup_key()
+            self.pkg('update')
+            self.pkg('install --yes run-postinsts-dev')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.cleanup_source_config_for_package_install_signed()
meta/lib/oeqa/selftest/cases/runtime_test.py +38 -0
index 2ad89490fc..3ece617cb0 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
         bitbake('core-image-full-cmdline socat')
         bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
     def test_testimage_virgl_gtk_sdl(self):
         """
         Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends
Signature · 1 line
2.32.0
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
author
#3 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Hi

Op 12-04-2022 om 16:16 schreef Alexandre Belloni:

Quoted 9 lines

Hello,

Quoted 6 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Quoted 7 lines
Quoted 5 lines

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

That is disappointing.

Quoted 5 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed

In fact package_index is failing, which is outside this patch code.

Quoted 4 lines

ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

I would have expected gnupg-native, but the log line above shows hosttools is being used. But the same would happen for signed rpm and ipk feeds right?

Did we get the correct one tested? I see 55173d in next and then reverted by Richard. But that was v2.

Quoted 128 lines
Quoted 126 lines
---
  meta/lib/oeqa/runtime/cases/apt.py           | 38 ++++++++++++++++----
  meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
  2 files changed, 69 insertions(+), 7 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +31 -7
index 53745df93f..574a34f148 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
 
      @classmethod
      def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
          cls.repo_server = HTTPService(service_repo,
                                        '0.0.0.0', port=cls.tc.target.server_port,
                                        logger=cls.tc.logger)
@@ -34,20 +34,44 @@ class AptRepoTest(AptTest):
      def setup_source_config_for_package_install(self):
          apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
          apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s/all ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+
+    def setup_source_config_for_package_install_signed(self):
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
 
      def cleanup_source_config_for_package_install(self):
          apt_get_sourceslist_dir = '/etc/apt/'
          self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
 
+    def cleanup_source_config_for_package_install_signed(self):
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
+
      @skipIfNotFeature('package-management',
                        'Test requires package-management to be in IMAGE_FEATURES')
      @skipIfNotDataVar('IMAGE_PKGTYPE', 'deb',
                        'DEB is not the primary package manager')
      @OEHasPackage(['apt'])
      def test_apt_install_from_repo(self):
-        self.setup_source_config_for_package_install()
-        self.pkg('update')
-        self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
-        self.cleanup_source_config_for_package_install()
+        if not self.tc.td.get('PACKAGE_FEED_GPG_NAME'):
+            self.setup_source_config_for_package_install()
+            self.pkg('update')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+            self.cleanup_source_config_for_package_install()
+        else:
+            # when we are here a key has been set to sign the package feed and
+            # public key and gnupg installed on the image by test_testimage_apt
+            self.setup_source_config_for_package_install_signed()
+            self.setup_key()
+            self.pkg('update')
+            self.pkg('install --yes run-postinsts-dev')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.cleanup_source_config_for_package_install_signed()
meta/lib/oeqa/selftest/cases/runtime_test.py +38 -0
index 2ad89490fc..3ece617cb0 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
          bitbake('core-image-full-cmdline socat')
          bitbake('-c testimage core-image-full-cmdline')
 
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
      def test_testimage_virgl_gtk_sdl(self):
          """
          Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends
Signature · 1 line
2.32.0
#4 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Quoted 32 lines · Ferry Toth

Hi

Op 12-04-2022 om 16:16 schreef Alexandre Belloni:

Quoted 9 lines

Hello,

Quoted 6 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Quoted 7 lines
Quoted 5 lines

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

That is disappointing.

Quoted 5 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed

In fact package_index is failing, which is outside this patch code.

Quoted 2 lines

ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

Quoted 9 lines
Quoted 2 lines

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

I would have expected gnupg-native, but the log line above shows hosttools is being used. But the same would happen for signed rpm and ipk feeds right?

Did we get the correct one tested? I see 55173d in next and then reverted by Richard. But that was v2.

This was https://git.yoctoproject.org/poky-contrib/commit/?id=5abda438ce762fc7b8e065e3e9063820c758918e

Just to be sure, I've started on ubuntu1604 both master and this branch, we'll see if this reproduces.

Quoted 124 lines
Quoted 124 lines
Quoted 122 lines
---
  meta/lib/oeqa/runtime/cases/apt.py           | 38 ++++++++++++++++----
  meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++
  2 files changed, 69 insertions(+), 7 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +31 -7
index 53745df93f..574a34f148 100644
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest):
      @classmethod
      def setUpClass(cls):
-        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
+        service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '')
          cls.repo_server = HTTPService(service_repo,
                                        '0.0.0.0', port=cls.tc.target.server_port,
                                        logger=cls.tc.logger)
@@ -34,20 +34,44 @@ class AptRepoTest(AptTest):
      def setup_source_config_for_package_install(self):
          apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port)
          apt_get_sourceslist_dir = '/etc/apt/'
-        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+        self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s/all ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server))
+
+    def setup_source_config_for_package_install_signed(self):
+        apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port)
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server))
      def cleanup_source_config_for_package_install(self):
          apt_get_sourceslist_dir = '/etc/apt/'
          self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir))
+    def cleanup_source_config_for_package_install_signed(self):
+        apt_get_sourceslist_dir = '/etc/apt/'
+        self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir))
+
+    def setup_key(self):
+        # the key is found on the target /etc/pki/packagefeed-gpg/
+        # named PACKAGEFEED-GPG-KEY-poky-branch
+        self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg'))
+
      @skipIfNotFeature('package-management',
                        'Test requires package-management to be in IMAGE_FEATURES')
      @skipIfNotDataVar('IMAGE_PKGTYPE', 'deb',
                        'DEB is not the primary package manager')
      @OEHasPackage(['apt'])
      def test_apt_install_from_repo(self):
-        self.setup_source_config_for_package_install()
-        self.pkg('update')
-        self.pkg('remove --yes run-postinsts-dev')
-        self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
-        self.cleanup_source_config_for_package_install()
+        if not self.tc.td.get('PACKAGE_FEED_GPG_NAME'):
+            self.setup_source_config_for_package_install()
+            self.pkg('update')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.pkg('install --yes --allow-unauthenticated run-postinsts-dev')
+            self.cleanup_source_config_for_package_install()
+        else:
+            # when we are here a key has been set to sign the package feed and
+            # public key and gnupg installed on the image by test_testimage_apt
+            self.setup_source_config_for_package_install_signed()
+            self.setup_key()
+            self.pkg('update')
+            self.pkg('install --yes run-postinsts-dev')
+            self.pkg('remove --yes run-postinsts-dev')
+            self.cleanup_source_config_for_package_install_signed()
meta/lib/oeqa/selftest/cases/runtime_test.py +38 -0
index 2ad89490fc..3ece617cb0 100644
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase):
          bitbake('core-image-full-cmdline socat')
          bitbake('-c testimage core-image-full-cmdline')
+    def test_testimage_apt(self):
+        """
+        Summary: Check package feeds functionality for apt
+        Expected: 1. Check that remote package feeds can be accessed
+        Product: oe-core
+        Author: Ferry Toth <[email protected]>
+        """
+        if get_bb_var('DISTRO') == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
+
+        features = 'INHERIT += "testimage"\n'
+        features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n'
+        # We don't yet know what the server ip and port will be - they will be patched
+        # in at the start of the on-image test
+        features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_deb"\n'
+        # We need  gnupg on the target to install keys
+        features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n'
+
+        bitbake('gnupg-native -c addto_recipe_sysroot')
+
+        # Enable package feed signing
+        self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-")
+        self.track_for_cleanup(self.gpg_home)
+        signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing')
+        runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True)
+        features += 'INHERIT += "sign_package_feed"\n'
+        features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
+        features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
+        features += 'GPG_PATH = "%s"\n' % self.gpg_home
+        features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
+        self.write_config(features)
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-full-cmdline socat')
+        bitbake('-c testimage core-image-full-cmdline')
+
      def test_testimage_virgl_gtk_sdl(self):
          """
          Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends
Signature · 1 line
2.32.0
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Quoted 39 lines · Alexandre Belloni
Quoted 38 lines · Ferry Toth

Hi

Op 12-04-2022 om 16:16 schreef Alexandre Belloni:

Quoted 9 lines

Hello,

Quoted 6 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Quoted 7 lines
Quoted 5 lines

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

That is disappointing.

Quoted 5 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed

In fact package_index is failing, which is outside this patch code.

Quoted 4 lines

ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

I would have expected gnupg-native, but the log line above shows hosttools is being used. But the same would happen for signed rpm and ipk feeds right?

Quoted 8 lines
Quoted 3 lines

Did we get the correct one tested? I see 55173d in next and then reverted by Richard. But that was v2.

This was https://git.yoctoproject.org/poky-contrib/commit/?id=5abda438ce762fc7b8e065e3e9063820c758918e

Just to be sure, I've started on ubuntu1604 both master and this branch, we'll see if this reproduces.

Firstly, this is occurring in the newly added test so this is being triggered by the new code. I suspect what is happening is that gnupg-native isn't being built before the test and this means that it is falling back to the system gpg. The system gpg is too old on that worker so it fails.

You can probably reproduce locally by not having a gpg on your build system (move it out the way temporarily?).

If I'm right (and I'm just guessing), the fix is to add the missing dependency to ensure gpg is one we've built.

Cheers,

Richard

author
#6 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Hi,

Op 12-04-2022 om 23:51 schreef Richard Purdie:

Quoted 37 lines · Alexandre Belloni
Quoted 36 lines · Ferry Toth
Quoted 35 lines

Hi

Op 12-04-2022 om 16:16 schreef Alexandre Belloni:

Quoted 9 lines

Hello,

Quoted 6 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Quoted 7 lines
Quoted 5 lines

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

That is disappointing.

Quoted 5 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed

In fact package_index is failing, which is outside this patch code.

Quoted 4 lines

ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

I would have expected gnupg-native, but the log line above shows hosttools is being used. But the same would happen for signed rpm and ipk feeds right?

Did we get the correct one tested? I see 55173d in next and then reverted by Richard. But that was v2.

This was https://git.yoctoproject.org/poky-contrib/commit/?id=5abda438ce762fc7b8e065e3e9063820c758918e

This is the correct one.

Quoted 7 lines
Quoted 2 lines

Just to be sure, I've started on ubuntu1604 both master and this branch, we'll see if this reproduces.

Firstly, this is occurring in the newly added test so this is being triggered by the new code. I suspect what is happening is that gnupg-native isn't being built before the test and this means that it is falling back to the system gpg. The system gpg is too old on that worker so it fails.

Certainly

Quoted 2 lines

You can probably reproduce locally by not having a gpg on your build system (move it out the way temporarily?).

Thanks for the tip. Not sure if I can remove the package, but IIUC it's the executable that needs to be present so I can just move it out of the way.

Quoted 2 lines

If I'm right (and I'm just guessing), the fix is to add the missing dependency to ensure gpg is one we've built.

I know how to add dependency in a recipe, but where to add here?

I already have 'bitbake('gnupg-native -c addto_recipe_sysroot')' Should I run 'bitbake('gnupg-native')' before that?

I copied these lines from test_testimage_dnf, shouldn't that have similar problems?

Quoted 6 lines

Cheers,

Richard

#7 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Quoted 70 lines · Ferry Toth

Hi,

Op 12-04-2022 om 23:51 schreef Richard Purdie:

Quoted 37 lines · Alexandre Belloni
Quoted 36 lines · Ferry Toth
Quoted 35 lines

Hi

Op 12-04-2022 om 16:16 schreef Alexandre Belloni:

Quoted 9 lines

Hello,

Quoted 6 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Quoted 7 lines
Quoted 5 lines

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

That is disappointing.

Quoted 5 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed

In fact package_index is failing, which is outside this patch code.

Quoted 4 lines

ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

I would have expected gnupg-native, but the log line above shows hosttools is being used. But the same would happen for signed rpm and ipk feeds right?

Did we get the correct one tested? I see 55173d in next and then reverted by Richard. But that was v2.

This was https://git.yoctoproject.org/poky-contrib/commit/?id=5abda438ce762fc7b8e065e3e9063820c758918e

This is the correct one.

Quoted 7 lines
Quoted 2 lines

Just to be sure, I've started on ubuntu1604 both master and this branch, we'll see if this reproduces.

Firstly, this is occurring in the newly added test so this is being triggered by the new code. I suspect what is happening is that gnupg-native isn't being built before the test and this means that it is falling back to the system gpg. The system gpg is too old on that worker so it fails.

Certainly

Quoted 2 lines

You can probably reproduce locally by not having a gpg on your build system (move it out the way temporarily?).

Thanks for the tip. Not sure if I can remove the package, but IIUC it's the executable that needs to be present so I can just move it out of the way.

Quoted 2 lines

If I'm right (and I'm just guessing), the fix is to add the missing dependency to ensure gpg is one we've built.

Quoted 8 lines

I know how to add dependency in a recipe, but where to add here?

I already have 'bitbake('gnupg-native -c addto_recipe_sysroot')' Should I run 'bitbake('gnupg-native')' before that?

I copied these lines from test_testimage_dnf, shouldn't that have similar problems?

sign_rpm.bbclass has PACKAGE_WRITE_DEPS += "gnupg-native", doesn't that solve this issue?

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Quoted 76 lines · Alexandre Belloni
Quoted 75 lines · Ferry Toth

Hi,

Op 12-04-2022 om 23:51 schreef Richard Purdie:

Quoted 37 lines · Alexandre Belloni
Quoted 36 lines · Ferry Toth
Quoted 35 lines

Hi

Op 12-04-2022 om 16:16 schreef Alexandre Belloni:

Quoted 9 lines

Hello,

Quoted 6 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Quoted 7 lines
Quoted 5 lines

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

That is disappointing.

Quoted 5 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed

In fact package_index is failing, which is outside this patch code.

Quoted 4 lines

ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

I would have expected gnupg-native, but the log line above shows hosttools is being used. But the same would happen for signed rpm and ipk feeds right?

Did we get the correct one tested? I see 55173d in next and then reverted by Richard. But that was v2.

This was https://git.yoctoproject.org/poky-contrib/commit/?id=5abda438ce762fc7b8e065e3e9063820c758918e

This is the correct one.

Quoted 7 lines
Quoted 2 lines

Just to be sure, I've started on ubuntu1604 both master and this branch, we'll see if this reproduces.

Firstly, this is occurring in the newly added test so this is being triggered by the new code. I suspect what is happening is that gnupg-native isn't being built before the test and this means that it is falling back to the system gpg. The system gpg is too old on that worker so it fails.

Certainly

Quoted 2 lines

You can probably reproduce locally by not having a gpg on your build system (move it out the way temporarily?).

Thanks for the tip. Not sure if I can remove the package, but IIUC it's the executable that needs to be present so I can just move it out of the way.

Quoted 2 lines

If I'm right (and I'm just guessing), the fix is to add the missing dependency to ensure gpg is one we've built.

I know how to add dependency in a recipe, but where to add here?

I already have 'bitbake('gnupg-native -c addto_recipe_sysroot')' Should I run 'bitbake('gnupg-native')' before that?

Quoted 6 lines
Quoted 3 lines

I copied these lines from test_testimage_dnf, shouldn't that have similar problems?

sign_rpm.bbclass has PACKAGE_WRITE_DEPS += "gnupg-native", doesn't that solve this issue?

Perhaps sign_package_feed.bbclass needs something like:

PACKAGEINDEXDEPS += "gnupg-native:do_populate_sysroot"

I'm not sure why/how it works in the rpm case but it does seem like the dependency is missing in the deb one.

Cheers,

Richard

author
#9 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Hi,

Op 13-04-2022 om 09:00 schreef Richard Purdie:

Quoted 55 lines · Alexandre Belloni
Quoted 54 lines · Ferry Toth
Quoted 53 lines

Hi,

Op 12-04-2022 om 23:51 schreef Richard Purdie:

Quoted 33 lines · Alexandre Belloni
Quoted 32 lines · Ferry Toth
Quoted 31 lines

Hi

Op 12-04-2022 om 16:16 schreef Alexandre Belloni:

Quoted 9 lines

Hello,

Quoted 6 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Quoted 6 lines
Quoted 5 lines

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

That is disappointing.

Quoted 5 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed

In fact package_index is failing, which is outside this patch code.

Quoted 4 lines

ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

I would have expected gnupg-native, but the log line above shows hosttools is being used. But the same would happen for signed rpm and ipk feeds right?

Did we get the correct one tested? I see 55173d in next and then reverted by Richard. But that was v2.

This was https://git.yoctoproject.org/poky-contrib/commit/?id=5abda438ce762fc7b8e065e3e9063820c758918e

This is the correct one.

Quoted 6 lines
Quoted 2 lines

Just to be sure, I've started on ubuntu1604 both master and this branch, we'll see if this reproduces.

Firstly, this is occurring in the newly added test so this is being triggered by the new code. I suspect what is happening is that gnupg-native isn't being built before the test and this means that it is falling back to the system gpg. The system gpg is too old on that worker so it fails.

Certainly

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

You can probably reproduce locally by not having a gpg on your build system (move it out the way temporarily?).

Thanks for the tip. Not sure if I can remove the package, but IIUC it's the executable that needs to be present so I can just move it out of the way.

Quoted 2 lines

If I'm right (and I'm just guessing), the fix is to add the missing dependency to ensure gpg is one we've built.

You are right

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I know how to add dependency in a recipe, but where to add here?

I already have 'bitbake('gnupg-native -c addto_recipe_sysroot')' Should I run 'bitbake('gnupg-native')' before that?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I copied these lines from test_testimage_dnf, shouldn't that have similar problems?

sign_rpm.bbclass has PACKAGE_WRITE_DEPS += "gnupg-native", doesn't that solve this issue?

Perhaps sign_package_feed.bbclass needs something like:

PACKAGEINDEXDEPS += "gnupg-native:do_populate_sysroot"

I added this to the end of 'meta/classes/package_deb.bbclass' and that works.

Do you agree this is the right place?

Should I squash with this patch or send in as a separate patch (fixes 0b4231b5 <https://git.yoctoproject.org/poky/commit/?id=0b4231b597618e18668b8340f4209cd364b2b2d0> "package_manager: sign DEB package feeds")?

Quoted 8 lines

I'm not sure why/how it works in the rpm case but it does seem like the dependency is missing in the deb one.

Cheers,

Richard

#10 lore raw

Re: [OE-core] [PATCH v3 1/1] apt: add apt selftest to test signed package feeds

Quoted 76 lines · Ferry Toth

Hi,

Op 13-04-2022 om 09:00 schreef Richard Purdie:

Quoted 55 lines · Alexandre Belloni
Quoted 54 lines · Ferry Toth
Quoted 53 lines

Hi,

Op 12-04-2022 om 23:51 schreef Richard Purdie:

Quoted 33 lines · Alexandre Belloni
Quoted 32 lines · Ferry Toth
Quoted 31 lines

Hi

Op 12-04-2022 om 16:16 schreef Alexandre Belloni:

Quoted 9 lines

Hello,

Quoted 6 lines · Ferry Toth

From: Ferry Toth <[email protected]>

Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.

Quoted 6 lines
Quoted 5 lines

Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.

This went through the autobuilders and it seems this still fails:

That is disappointing.

Quoted 5 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3437/steps/15/logs/stdio

ERROR: package-index-1.0-r0 do_package_index: Could not get gpg version: Command '['/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg', '--agent-program=/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/hosttools/gpg-agent|--auto-expand-secmem', '--version', '--no-permission-warning']' returned non-zero exit status 2.
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-34525/tmp/work/core2-64-poky-linux/package-index/1.0-r0/temp/log.do_package_index.53841
NOTE: recipe package-index-1.0-r0: task do_package_index: Failed

In fact package_index is failing, which is outside this patch code.

Quoted 4 lines

ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-core/meta/package-index.bb:do_package_index) failed with exit code '1'

This was ubuntu 16.04 so maybe gpg on the distro is too old (1.4.20) but I'm not sure as I think you are using gnupg-native.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

I would have expected gnupg-native, but the log line above shows hosttools is being used. But the same would happen for signed rpm and ipk feeds right?

Did we get the correct one tested? I see 55173d in next and then reverted by Richard. But that was v2.

This was https://git.yoctoproject.org/poky-contrib/commit/?id=5abda438ce762fc7b8e065e3e9063820c758918e

This is the correct one.

Quoted 6 lines
Quoted 2 lines

Just to be sure, I've started on ubuntu1604 both master and this branch, we'll see if this reproduces.

Firstly, this is occurring in the newly added test so this is being triggered by the new code. I suspect what is happening is that gnupg-native isn't being built before the test and this means that it is falling back to the system gpg. The system gpg is too old on that worker so it fails.

Certainly

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

You can probably reproduce locally by not having a gpg on your build system (move it out the way temporarily?).

Thanks for the tip. Not sure if I can remove the package, but IIUC it's the executable that needs to be present so I can just move it out of the way.

Quoted 2 lines

If I'm right (and I'm just guessing), the fix is to add the missing dependency to ensure gpg is one we've built.

You are right

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

I know how to add dependency in a recipe, but where to add here?

I already have 'bitbake('gnupg-native -c addto_recipe_sysroot')' Should I run 'bitbake('gnupg-native')' before that?

I copied these lines from test_testimage_dnf, shouldn't that have similar problems?

sign_rpm.bbclass has PACKAGE_WRITE_DEPS += "gnupg-native", doesn't that solve this issue?

Quoted 8 lines
Quoted 3 lines

Perhaps sign_package_feed.bbclass needs something like:

PACKAGEINDEXDEPS += "gnupg-native:do_populate_sysroot"

I added this to the end of 'meta/classes/package_deb.bbclass' and that works.

Do you agree this is the right place?

No. That builds gpg pieces even when signing isn't enabled so I don't think that is right.

Quoted 4 lines

Should I squash with this patch or send in as a separate patch (fixes 0b4231b5
<https://git.yoctoproject.org/poky/commit/?id=0b4231b597618e18668b8340f4209cd364b2b2d0>
"package_manager: sign DEB package feeds")?

I think we can likely make this change a separate commit since it will have it's own explanation with it.

Cheers,

Richard