package_manager: support for signed DEB package feeds
[PATCH v2 0/3] package_manager: support for signed DEB package feeds [PATCH v2 1/3] gpg-sign: Add parameters to gpg signature function [PATCH v2 2/3] package_manager: sign DEB package feeds [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.
Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign deb package feeds" enable signed deb package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.
These patches makes deb a first class citizen as ipk and rpm.
Patches have been in use in meta-intel-edison since Gatesgarth,
see https://edison-fw.github.io/meta-intel-edison/5.0-Creating-a-deb-repository.html
Changes in V2:
- Added runtime test for signed deb package feeds (Richard Purdie)
PATCH v2 3/3 apt: add apt selftest to test signed package feeds
From: Ferry Toth <[email protected]>
Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.
Patches "gpg-sign: Add parameters to gpg signature function" and "package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from the test_testimage_dnf test. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.
--- meta/lib/oeqa/runtime/cases/apt.py | 16 ++++++--- meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++ 2 files changed, 49 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +11 -5
@@ -21,7 +21,7 @@ class AptRepoTest(AptTest): @classmethod def setUpClass(cls): - service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all') + service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '') cls.repo_server = HTTPService(service_repo, '0.0.0.0', port=cls.tc.target.server_port, logger=cls.tc.logger)
@@ -32,13 +32,18 @@ class AptRepoTest(AptTest): cls.repo_server.stop() def setup_source_config_for_package_install(self): - apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port) + apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port) apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server)) + self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server)) def cleanup_source_config_for_package_install(self): apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir)) + self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir)) + + def setup_key(self): + # the key is found on the target /etc/pki/packagefeed-gpg/ + # named PACKAGEFEED-GPG-KEY-poky-branch + self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg')) @skipIfNotFeature('package-management', 'Test requires package-management to be in IMAGE_FEATURES')
@@ -47,7 +52,8 @@ class AptRepoTest(AptTest): @OEHasPackage(['apt']) def test_apt_install_from_repo(self): self.setup_source_config_for_package_install() + self.setup_key() self.pkg('update') self.pkg('remove --yes run-postinsts-dev') - self.pkg('install --yes --allow-unauthenticated run-postinsts-dev') + self.pkg('install --yes run-postinsts-dev') self.cleanup_source_config_for_package_install()
meta/lib/oeqa/selftest/cases/runtime_test.py +38 -0
@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase): bitbake('core-image-full-cmdline socat') bitbake('-c testimage core-image-full-cmdline') + def test_testimage_apt(self): + """ + Summary: Check package feeds functionality for apt + Expected: 1. Check that remote package feeds can be accessed + Product: oe-core + Author: Ferry Toth <[email protected]> + """ + if get_bb_var('DISTRO') == 'poky-tiny': + self.skipTest('core-image-full-cmdline not buildable for poky-tiny') + + features = 'INHERIT += "testimage"\n' + features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n' + # We don't yet know what the server ip and port will be - they will be patched + # in at the start of the on-image test + features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_deb"\n' + # We need gnupg on the target to install keys + features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n' + + bitbake('gnupg-native -c addto_recipe_sysroot') + + # Enable package feed signing + self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-") + self.track_for_cleanup(self.gpg_home) + signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing') + runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True) + features += 'INHERIT += "sign_package_feed"\n' + features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n' + features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase') + features += 'GPG_PATH = "%s"\n' % self.gpg_home + features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home + self.write_config(features) + + # Build core-image-sato and testimage + bitbake('core-image-full-cmdline socat') + bitbake('-c testimage core-image-full-cmdline') + def test_testimage_virgl_gtk_sdl(self): """ Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends
Signature · 1 line
2.32.0
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
Quoted 101 lines · Ferry Toth
From: Ferry Toth <[email protected]>
Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.
Patches "gpg-sign: Add parameters to gpg signature function" and "
package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from thetest_testimage_dnftest. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.Signed-off-by Ferry Toth <[email protected]>--- meta/lib/oeqa/runtime/cases/apt.py | 16 ++++++--- meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++ 2 files changed, 49 insertions(+), 5 deletions(-)meta/lib/oeqa/runtime/cases/apt.py +11
-5@@ -21,7 +21,7 @@ class AptRepoTest(AptTest): @classmethod def setUpClass(cls): - service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all') + service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '') cls.repo_server = HTTPService(service_repo, '0.0.0.0', port=cls.tc.target.server_port, logger=cls.tc.logger)@@ -32,13 +32,18 @@ class AptRepoTest(AptTest): cls.repo_server.stop() def setup_source_config_for_package_install(self): - apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port) + apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port) apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server)) + self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server)) def cleanup_source_config_for_package_install(self): apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir)) + self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir)) + + def setup_key(self): + # the key is found on the target /etc/pki/packagefeed-gpg/ + # named PACKAGEFEED-GPG-KEY-poky-branch + self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg')) @skipIfNotFeature('package-management', 'Test requires package-management to be in IMAGE_FEATURES')@@ -47,7 +52,8 @@ class AptRepoTest(AptTest): @OEHasPackage(['apt']) def test_apt_install_from_repo(self): self.setup_source_config_for_package_install() + self.setup_key() self.pkg('update') self.pkg('remove --yes run-postinsts-dev') - self.pkg('install --yes --allow-unauthenticated run-postinsts-dev') + self.pkg('install --yes run-postinsts-dev') self.cleanup_source_config_for_package_install()meta/lib/oeqa/selftest/cases/runtime_test.py +31
-0@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase): bitbake('core-image-full-cmdline socat') bitbake('-c testimage core-image-full-cmdline') + def test_testimage_apt(self): + """ + Summary: Check package feeds functionality for apt + Expected: 1. Check that remote package feeds can be accessed + Product: oe-core + Author: Ferry Toth <[email protected]> + """ + if get_bb_var('DISTRO') == 'poky-tiny': + self.skipTest('core-image-full-cmdline not buildable for poky-tiny') + + features = 'INHERIT += "testimage"\n' + features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n' + # We don't yet know what the server ip and port will be - they will be patched + # in at the start of the on-image test + features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_deb"\n' + # We need gnupg on the target to install keys + features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n' + + bitbake('gnupg-native -c addto_recipe_sysroot') + + # Enable package feed signing + self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-") + self.track_for_cleanup(self.gpg_home) + signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing') + runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True) + features += 'INHERIT += "sign_package_feed"\n' + features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n' + features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase') + features += 'GPG_PATH = "%s"\n' % self.gpg_home
meta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205
Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7
-0+ features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home + self.write_config(features) + + # Build core-image-sato and testimage + bitbake('core-image-full-cmdline socat') + bitbake('-c testimage core-image-full-cmdline') + def test_testimage_virgl_gtk_sdl(self): """ Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontends
Thanks for working on this!
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975
however hopefully these shouldn't be too hard to fix?
The rest of the build is still running.
Cheers,
Richard
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
Hi,
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 115 lines · Ferry Toth
Quoted 101 lines
From: Ferry Toth <[email protected]>
Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.
Patches "gpg-sign: Add parameters to gpg signature function" and "
package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from thetest_testimage_dnftest. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.Signed-off-by Ferry Toth <[email protected]>--- meta/lib/oeqa/runtime/cases/apt.py | 16 ++++++--- meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++ 2 files changed, 49 insertions(+), 5 deletions(-)meta/lib/oeqa/runtime/cases/apt.py +11
-5@@ -21,7 +21,7 @@ class AptRepoTest(AptTest): @classmethod def setUpClass(cls): - service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all') + service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '') cls.repo_server = HTTPService(service_repo, '0.0.0.0', port=cls.tc.target.server_port, logger=cls.tc.logger)@@ -32,13 +32,18 @@ class AptRepoTest(AptTest): cls.repo_server.stop() def setup_source_config_for_package_install(self): - apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port) + apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port) apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server)) + self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server)) def cleanup_source_config_for_package_install(self): apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir)) + self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir)) + + def setup_key(self): + # the key is found on the target /etc/pki/packagefeed-gpg/ + # named PACKAGEFEED-GPG-KEY-poky-branch + self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg')) @skipIfNotFeature('package-management', 'Test requires package-management to be in IMAGE_FEATURES')@@ -47,7 +52,8 @@ class AptRepoTest(AptTest): @OEHasPackage(['apt']) def test_apt_install_from_repo(self): self.setup_source_config_for_package_install() + self.setup_key() self.pkg('update') self.pkg('remove --yes run-postinsts-dev') - self.pkg('install --yes --allow-unauthenticated run-postinsts-dev') + self.pkg('install --yes run-postinsts-dev') self.cleanup_source_config_for_package_install()meta/lib/oeqa/selftest/cases/runtime_test.py +31
-0@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase): bitbake('core-image-full-cmdline socat') bitbake('-c testimage core-image-full-cmdline') + def test_testimage_apt(self): + """ + Summary: Check package feeds functionality for apt + Expected: 1. Check that remote package feeds can be accessed + Product: oe-core + Author: Ferry Toth <[email protected]> + """ + if get_bb_var('DISTRO') == 'poky-tiny': + self.skipTest('core-image-full-cmdline not buildable for poky-tiny') + + features = 'INHERIT += "testimage"\n' + features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n' + # We don't yet know what the server ip and port will be - they will be patched + # in at the start of the on-image test + features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_deb"\n' + # We need gnupg on the target to install keys + features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n' + + bitbake('gnupg-native -c addto_recipe_sysroot') + + # Enable package feed signing + self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-") + self.track_for_cleanup(self.gpg_home) + signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing') + runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True) + features += 'INHERIT += "sign_package_feed"\n' + features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n' + features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase') + features += 'GPG_PATH = "%s"\n' % self.gpg_homemeta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205
Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7
-0+ features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home + self.write_config(features) + + # Build core-image-sato and testimage + bitbake('core-image-full-cmdline socat') + bitbake('-c testimage core-image-full-cmdline') + def test_testimage_virgl_gtk_sdl(self): """ Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontendsThanks for working on this!
Quoted 5 lines
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975
That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-dev
That is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
What could cause this? Because the complaint is it can't remove the package because it was not installed.
It would be trivial to remove the line
*) self.pkg('remove --yes run-postinsts-dev')
but how could it have passed the test before?
Quoted 8 lines
however hopefully these shouldn't be too hard to fix?
The rest of the build is still running.
Cheers,
Richard
PATCH v2 1/3 gpg-sign: Add parameters to gpg signature function
In reply to Ferry TothFrom: Xavier Berger <[email protected]>
output_suffix: If defined, add output_suffix as file name extension.use_sha256: If True, use sha256 for gpg as digest algorithm
--- meta/lib/oe/gpg_sign.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/gpg_sign.py +5 -1
@@ -58,7 +58,7 @@ class LocalSigner(object): for i in range(0, len(files), sign_chunk): subprocess.check_output(shlex.split(cmd + ' '.join(files[i:i+sign_chunk])), stderr=subprocess.STDOUT) - def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True): + def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True, output_suffix=None, use_sha256=False): """Create a detached signature of a file""" if passphrase_file and passphrase:
@@ -71,6 +71,10 @@ class LocalSigner(object): cmd += ['--homedir', self.gpg_path] if armor: cmd += ['--armor'] + if output_suffix: + cmd += ['-o', input_file + "." + output_suffix] + if use_sha256: + cmd += ['--digest-algo', "SHA256"] #gpg > 2.1 supports password pipes only through the loopback interface #gpg < 2.1 errors out if given unknown parameters
Signature · 1 line
2.32.0
v2 cover *** SUBJECT HERE ***
In reply to Ferry TothFrom: Ferry Toth <[email protected]>
*** BLURB HERE ***
Ferry Toth (2):
package_manager: sign DEB package feeds
apt: add apt selftest to test signed package feeds
Xavier Berger (1):
gpg-sign: Add parameters to gpg signature function
meta/lib/oe/gpg_sign.py | 6 +++- meta/lib/oe/package_manager/deb/__init__.py | 19 ++++++++-- meta/lib/oeqa/runtime/cases/apt.py | 16 ++++++--- meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++ 4 files changed, 70 insertions(+), 9 deletions(-)
Signature · 1 line
2.32.0
PATCH v2 2/3 package_manager: sign DEB package feeds
In reply to Ferry TothFrom: Ferry Toth <[email protected]>
Implement debian package repository signature.
For each Release file created in repository subdirectory, a signature Release.gpg is created.
Signature is performed using gpg backend when the following variables are set in local.conf:PACKAGE_CLASSES += "sign_package_feed" PACKAGE_FEED_GPG_NAME = "<Id of GPG key>" PACKAGE_FEED_GPG_PASSPHRASE_FILE="<path to password file>"
--- meta/lib/oe/package_manager/deb/__init__.py | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
meta/lib/oe/package_manager/deb/__init__.py +16 -3
@@ -53,6 +53,7 @@ class DpkgIndexer(Indexer): index_cmds = [] deb_dirs_found = False + index_sign_files = set() for arch in arch_list: arch_dir = os.path.join(self.deploy_dir, arch) if not os.path.isdir(arch_dir):
@@ -62,7 +63,10 @@ class DpkgIndexer(Indexer): cmd += "%s -fcn Packages > Packages.gz;" % gzip - with open(os.path.join(arch_dir, "Release"), "w+") as release: + release_file = os.path.join(arch_dir, "Release") + index_sign_files.add(release_file) + + with open(release_file, "w+") as release: release.write("Label: %s\n" % arch) cmd += "PSEUDO_UNLOAD=1 %s release . >> Release" % apt_ftparchive
@@ -76,8 +80,17 @@ class DpkgIndexer(Indexer): return oe.utils.multiprocess_launch(create_index, index_cmds, self.d) - if self.d.getVar('PACKAGE_FEED_SIGN') == '1': - raise NotImplementedError('Package feed signing not implementd for dpkg') + if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1': + signer = get_signer(self.d, self.d.getVar('PACKAGE_FEED_GPG_BACKEND', True)) + else: + signer = None + if signer: + for f in index_sign_files: + signer.detach_sign(f, + self.d.getVar('PACKAGE_FEED_GPG_NAME', True), + self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True), + output_suffix="gpg", + use_sha256=True) class PMPkgsList(PkgsList):
Signature · 1 line
2.32.0
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
In reply to Ferry TothQuoted 133 lines · Ferry Toth
Hi,
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 115 lines · Ferry Toth
Quoted 101 lines
From: Ferry Toth <[email protected]>
Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.
Patches "gpg-sign: Add parameters to gpg signature function" and "
package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from thetest_testimage_dnftest. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.Signed-off-by Ferry Toth <[email protected]>--- meta/lib/oeqa/runtime/cases/apt.py | 16 ++++++--- meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++ 2 files changed, 49 insertions(+), 5 deletions(-)meta/lib/oeqa/runtime/cases/apt.py +11
-5@@ -21,7 +21,7 @@ class AptRepoTest(AptTest): @classmethod def setUpClass(cls): - service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all') + service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '') cls.repo_server = HTTPService(service_repo, '0.0.0.0', port=cls.tc.target.server_port, logger=cls.tc.logger)@@ -32,13 +32,18 @@ class AptRepoTest(AptTest): cls.repo_server.stop() def setup_source_config_for_package_install(self): - apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port) + apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port) apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server)) + self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server)) def cleanup_source_config_for_package_install(self): apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir)) + self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir)) + + def setup_key(self): + # the key is found on the target /etc/pki/packagefeed-gpg/ + # named PACKAGEFEED-GPG-KEY-poky-branch + self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg')) @skipIfNotFeature('package-management', 'Test requires package-management to be in IMAGE_FEATURES')@@ -47,7 +52,8 @@ class AptRepoTest(AptTest): @OEHasPackage(['apt']) def test_apt_install_from_repo(self): self.setup_source_config_for_package_install() + self.setup_key() self.pkg('update') self.pkg('remove --yes run-postinsts-dev') - self.pkg('install --yes --allow-unauthenticated run-postinsts-dev') + self.pkg('install --yes run-postinsts-dev') self.cleanup_source_config_for_package_install()meta/lib/oeqa/selftest/cases/runtime_test.py +31
-0@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase): bitbake('core-image-full-cmdline socat') bitbake('-c testimage core-image-full-cmdline') + def test_testimage_apt(self): + """ + Summary: Check package feeds functionality for apt + Expected: 1. Check that remote package feeds can be accessed + Product: oe-core + Author: Ferry Toth <[email protected]> + """ + if get_bb_var('DISTRO') == 'poky-tiny': + self.skipTest('core-image-full-cmdline not buildable for poky-tiny') + + features = 'INHERIT += "testimage"\n' + features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n' + # We don't yet know what the server ip and port will be - they will be patched + # in at the start of the on-image test + features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_deb"\n' + # We need gnupg on the target to install keys + features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n' + + bitbake('gnupg-native -c addto_recipe_sysroot') + + # Enable package feed signing + self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-") + self.track_for_cleanup(self.gpg_home) + signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing') + runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True) + features += 'INHERIT += "sign_package_feed"\n' + features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n' + features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase') + features += 'GPG_PATH = "%s"\n' % self.gpg_homemeta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205
Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7
-0+ features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home + self.write_config(features) + + # Build core-image-sato and testimage + bitbake('core-image-full-cmdline socat') + bitbake('-c testimage core-image-full-cmdline') + def test_testimage_virgl_gtk_sdl(self): """ Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontendsThanks for working on this!
Quoted 5 lines
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-devThat is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
Quoted 8 lines
What could cause this? Because the complaint is it can't remove the package because it was not installed.
It would be trivial to remove the line
*)
self.pkg('remove --yes run-postinsts-dev')but how could it have passed the test before?
I think the issue is you edited testimage which is a different set of tests which aren't just called by oe-selftest but by things like
"bitbake core-image-sato -c testimage"
as well. I'd suggest making the changes in testimage conditional on signing being configured.
Cheers,
Richard
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
Hi,
Op 04-04-2022 om 22:39 schreef Richard Purdie:
Quoted 135 lines · Ferry Toth
Quoted 134 lines
Hi,
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 114 lines · Ferry Toth
Quoted 101 lines
From: Ferry Toth<[email protected]>
Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.
Patches "gpg-sign: Add parameters to gpg signature function" and "
package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from thetest_testimage_dnftest. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.Signed-off-by Ferry Toth<[email protected]>--- meta/lib/oeqa/runtime/cases/apt.py | 16 ++++++--- meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++ 2 files changed, 49 insertions(+), 5 deletions(-)meta/lib/oeqa/runtime/cases/apt.py +11
-5@@ -21,7 +21,7 @@ class AptRepoTest(AptTest): @classmethod def setUpClass(cls): - service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all') + service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '') cls.repo_server = HTTPService(service_repo, '0.0.0.0', port=cls.tc.target.server_port, logger=cls.tc.logger)@@ -32,13 +32,18 @@ class AptRepoTest(AptTest): cls.repo_server.stop() def setup_source_config_for_package_install(self): - apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port) + apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port) apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server)) + self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server)) def cleanup_source_config_for_package_install(self): apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir)) + self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir)) + + def setup_key(self): + # the key is found on the target /etc/pki/packagefeed-gpg/ + # named PACKAGEFEED-GPG-KEY-poky-branch + self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg')) @skipIfNotFeature('package-management', 'Test requires package-management to be in IMAGE_FEATURES')@@ -47,7 +52,8 @@ class AptRepoTest(AptTest): @OEHasPackage(['apt']) def test_apt_install_from_repo(self): self.setup_source_config_for_package_install() + self.setup_key() self.pkg('update') self.pkg('remove --yes run-postinsts-dev') - self.pkg('install --yes --allow-unauthenticated run-postinsts-dev') + self.pkg('install --yes run-postinsts-dev') self.cleanup_source_config_for_package_install()meta/lib/oeqa/selftest/cases/runtime_test.py +31
-0@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase): bitbake('core-image-full-cmdline socat') bitbake('-c testimage core-image-full-cmdline') + def test_testimage_apt(self): + """ + Summary: Check package feeds functionality for apt + Expected: 1. Check that remote package feeds can be accessed + Product: oe-core + Author: Ferry Toth<[email protected]> + """ + if get_bb_var('DISTRO') == 'poky-tiny': + self.skipTest('core-image-full-cmdline not buildable for poky-tiny') + + features = 'INHERIT += "testimage"\n' + features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n' + # We don't yet know what the server ip and port will be - they will be patched + # in at the start of the on-image test + features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_deb"\n' + # We need gnupg on the target to install keys + features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n' + + bitbake('gnupg-native -c addto_recipe_sysroot') + + # Enable package feed signing + self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-") + self.track_for_cleanup(self.gpg_home) + signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing') + runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True) + features += 'INHERIT += "sign_package_feed"\n' + features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n' + features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase') + features += 'GPG_PATH = "%s"\n' % self.gpg_homemeta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205
Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7
-0+ features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home + self.write_config(features) + + # Build core-image-sato and testimage + bitbake('core-image-full-cmdline socat') + bitbake('-c testimage core-image-full-cmdline') + def test_testimage_virgl_gtk_sdl(self): """ Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontendsThanks for working on this!
Quoted 5 lines
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-devReading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-devThat is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
What could cause this? Because the complaint is it can't remove the package because it was not installed.
Quoted 8 lines
Quoted 5 lines
It would be trivial to remove the line
*)
self.pkg('remove --yes run-postinsts-dev')but how could it have passed the test before?
I think the issue is you edited testimage which is a different set of tests which aren't just called by oe-selftest but by things like
That would be my first thought too, but...
because the failure seems to be on the line self.pkg('remove --yes run-postinsts-dev'), that would mean the line self.pkg('update') passed.
And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week).
So, there may be a second thing wrong?
Do you know where I can find the log files referred to:
<..>tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/temp/log.do_testimage.35553
<..>tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0-r0/temp/log.do_testimage.35362
or could we do a 'quick' check by changing
self.pkg('update')
self.pkg('remove --yes run-postinsts-dev')
self.pkg('install --yes run-postinsts-dev')
to
self.pkg('update')
self.pkg('install --yes run-postinsts-dev')
self.pkg('remove --yes run-postinsts-dev')?
Quoted 5 lines
"bitbake core-image-sato -c testimage"
as well. I'd suggest making the changes in testimage conditional on signing being configured.
Yes, regardless the above, we need to either make signing always enabled in all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest if PACKAGE_FEED_GPG_NAME has been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?
Quoted 5 lines
Cheers,
Richard
Re: [OE-core] [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
In reply to Richard PurdieHello,
Quoted 118 lines · Richard Purdie
Quoted 101 lines · Ferry Toth
From: Ferry Toth <[email protected]>
Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.
Patches "gpg-sign: Add parameters to gpg signature function" and "
package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from thetest_testimage_dnftest. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.Signed-off-by Ferry Toth <[email protected]>--- meta/lib/oeqa/runtime/cases/apt.py | 16 ++++++--- meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++ 2 files changed, 49 insertions(+), 5 deletions(-)meta/lib/oeqa/runtime/cases/apt.py +11
-5@@ -21,7 +21,7 @@ class AptRepoTest(AptTest): @classmethod def setUpClass(cls): - service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all') + service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '') cls.repo_server = HTTPService(service_repo, '0.0.0.0', port=cls.tc.target.server_port, logger=cls.tc.logger)@@ -32,13 +32,18 @@ class AptRepoTest(AptTest): cls.repo_server.stop() def setup_source_config_for_package_install(self): - apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port) + apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port) apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server)) + self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server)) def cleanup_source_config_for_package_install(self): apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir)) + self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir)) + + def setup_key(self): + # the key is found on the target /etc/pki/packagefeed-gpg/ + # named PACKAGEFEED-GPG-KEY-poky-branch + self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg')) @skipIfNotFeature('package-management', 'Test requires package-management to be in IMAGE_FEATURES')@@ -47,7 +52,8 @@ class AptRepoTest(AptTest): @OEHasPackage(['apt']) def test_apt_install_from_repo(self): self.setup_source_config_for_package_install() + self.setup_key() self.pkg('update') self.pkg('remove --yes run-postinsts-dev') - self.pkg('install --yes --allow-unauthenticated run-postinsts-dev') + self.pkg('install --yes run-postinsts-dev') self.cleanup_source_config_for_package_install()meta/lib/oeqa/selftest/cases/runtime_test.py +31
-0@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase): bitbake('core-image-full-cmdline socat') bitbake('-c testimage core-image-full-cmdline') + def test_testimage_apt(self): + """ + Summary: Check package feeds functionality for apt + Expected: 1. Check that remote package feeds can be accessed + Product: oe-core + Author: Ferry Toth <[email protected]> + """ + if get_bb_var('DISTRO') == 'poky-tiny': + self.skipTest('core-image-full-cmdline not buildable for poky-tiny') + + features = 'INHERIT += "testimage"\n' + features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n' + # We don't yet know what the server ip and port will be - they will be patched + # in at the start of the on-image test + features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_deb"\n' + # We need gnupg on the target to install keys + features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n' + + bitbake('gnupg-native -c addto_recipe_sysroot') + + # Enable package feed signing + self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-") + self.track_for_cleanup(self.gpg_home) + signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing') + runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True) + features += 'INHERIT += "sign_package_feed"\n' + features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n' + features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase') + features += 'GPG_PATH = "%s"\n' % self.gpg_homemeta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205
Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7
-0+ features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home + self.write_config(features) + + # Build core-image-sato and testimage + bitbake('core-image-full-cmdline socat') + bitbake('-c testimage core-image-full-cmdline') + def test_testimage_virgl_gtk_sdl(self): """ Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontendsThanks for working on this!
Looking at the patches I wondered if this would break testimage and unfortunately it does:
Quoted 6 lines
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975however hopefully these shouldn't be too hard to fix?
The rest of the build is still running.
I missed it at the time but I believe this is also the cause of:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3352/steps/15/logs/stdio
ERROR: package-index-1.0-r0 do_package_index: GPG exited with code 2: gpg: can't connect to the agent: IPC connect call failed
gpg: skipped "testuser": No secret key
gpg: signing failed: No secret key
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
In reply to Ferry TothQuoted 29 lines · Ferry Toth
Op 04-04-2022 om 22:39 schreef Richard Purdie:
Quoted 27 lines · Ferry Toth
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 25 lines
Quoted 8 lines · Ferry Toth
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-devThat is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
What could cause this? Because the complaint is it can't remove the package because it was not installed.
It would be trivial to remove the line
*)
self.pkg('remove --yes run-postinsts-dev')but how could it have passed the test before?
Quoted 8 lines
Quoted 2 lines
I think the issue is you edited testimage which is a different set of tests which aren't just called by oe-selftest but by things like
That would be my first thought too, but...
because the failure seems to be on the lineself.pkg('remove --yes run- postinsts-dev'), that would mean the lineself.pkg('update')passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?
I was easily able to reproduce this locally and it shows the setup_source_config_for_package_install() step fails and hence the sources aren't setup correctly, hence the update probably works.
Quoted 5 lines
Do you know where I can find the log files referred to:
<..>tmp/work/qemux86-poky-linux/core-image-sato/1.0- r0/temp/log.do_testimage.35553 <..>tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0- r0/temp/log.do_testimage.35362
We can get them off the autobuilder if needed but someone would have to manually go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with package_deb set as the backend.
Quoted 9 lines
or could we do a 'quick' check by changing
self.pkg('update')
self.pkg('remove --yes run-postinsts-dev')self.pkg('install --yes run-postinsts-dev')to
self.pkg('update')self.pkg('install --yes run-postinsts-dev')self.pkg('remove --yes run-postinsts-dev')?
I'm not convinced that would help us...
Quoted 6 lines
Quoted 5 lines
"bitbake core-image-sato -c testimage"
as well. I'd suggest making the changes in testimage conditional on signing being configured.
Quoted 8 lines
Yes, regardless the above, we need to either make signing always enabled in all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest ifPACKAGE_FEED_GPG_NAMEhas been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?
We should be able to test self.tc.td.get('PACKAGE_FEED_GPG_NAME') in the test and handle accordingly?
I did merge the base changes into the release since I thought it was fair to get the fixes in before it was built. We just need to get the test sorted now, I think it is close.
Cheers,
Richard
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
In reply to Ferry TothQuoted 40 lines · Ferry Toth
Op 06-04-2022 om 13:40 schreef Richard Purdie:
Quoted 39 lines · Ferry Toth
Quoted 38 lines
Op 04-04-2022 om 22:39 schreef Richard Purdie:
Quoted 22 lines · Ferry Toth
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 21 lines
Quoted 9 lines · Ferry Toth
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/50
13/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/49
75That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-devThat is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
What could cause this? Because the complaint is it can't remove the package because it was not installed.
Quoted 8 lines
Quoted 5 lines
It would be trivial to remove the line
*)
self.pkg('remove --yes run-postinsts-dev')but how could it have passed the test before?
I think the issue is you edited testimage which is a different set of tests
which aren't just called by oe-selftest but by things likeThat would be my first thought too, but...
because the failure seems to be on the lineself.pkg('remove --yes run- postinsts-dev'), that would mean the lineself.pkg('update')passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?
Quoted 4 lines
Quoted 3 lines
I was easily able to reproduce this locally and it shows the
setup_source_config_for_package_install()step fails and hence the sources aren't setup correctly, hence the update probably works.not correct, hence works. You lost me here, but I'll try to reproduce.
I mean the command doesn't work correctly. In my local logs I see:
DEBUG: Command: cd /etc/apt/; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/http:\/\/192.168.7.1:46599/g' sources.list
Status: 1 Output: cp: can't stat 'sources.list': No such file or directory
sed: sources.list: No such file or directory
Quoted 6 lines
Quoted 2 lines
go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with
package_debset as the backend...in conf. But without
PACKAGE_CLASSES,PACKAGE_FEED_GPG_NAME,PACKAGE_FEED_GPG_PASSPHRASE_FILE?
Yes.
Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
Yes, regardless the above, we need to either make signing always enabled in
all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest ifPACKAGE_FEED_GPG_NAMEhas been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?Quoted 8 lines
We should be able to test
self.tc.td.get('PACKAGE_FEED_GPG_NAME')in the test
and handle accordingly?I did merge the base changes into the release since I thought it was fair to get
the fixes in before it was built. We just need to get the test sorted now, I think it is close.
Quoted 6 lines
Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?
No, the autobuilder runs all the tests in oe-selftest so it should be covered (and is why we saw the failures on the autobuilder).
Cheers,
Richard
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
In reply to Richard PurdieHi,
Op 06-04-2022 om 13:40 schreef Richard Purdie:
Quoted 36 lines · Ferry Toth
Quoted 27 lines
Op 04-04-2022 om 22:39 schreef Richard Purdie:
Quoted 25 lines · Ferry Toth
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 24 lines
Quoted 7 lines · Ferry Toth
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-devThat is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
What could cause this? Because the complaint is it can't remove the package because it was not installed.
It would be trivial to remove the line
*)
self.pkg('remove --yes run-postinsts-dev')but how could it have passed the test before?
Quoted 8 lines
Quoted 2 lines
I think the issue is you edited testimage which is a different set of tests which aren't just called by oe-selftest but by things like
That would be my first thought too, but...
because the failure seems to be on the lineself.pkg('remove --yes run- postinsts-dev'), that would mean the lineself.pkg('update')passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?
Quoted 3 lines
I was easily able to reproduce this locally and it shows the
setup_source_config_for_package_install()step fails and hence the sources aren't setup correctly, hence the update probably works.
not correct, hence works. You lost me here, but I'll try to reproduce.
Quoted 6 lines
Quoted 5 lines
Do you know where I can find the log files referred to:
<..>tmp/work/qemux86-poky-linux/core-image-sato/1.0- r0/temp/log.do_testimage.35553 <..>tmp/work/qemux86-poky-linux/core-image-sato-sdk/1.0- r0/temp/log.do_testimage.35362We can get them off the autobuilder if needed but someone would have to manually
No, that would be too much work. I'll try to reproduce myself.
Quoted 2 lines
go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with
package_debset as the backend.
..in conf. But without PACKAGE_CLASSES, PACKAGE_FEED_GPG_NAME, PACKAGE_FEED_GPG_PASSPHRASE_FILE?
Quoted 10 lines
Quoted 9 lines
or could we do a 'quick' check by changing
self.pkg('update')
self.pkg('remove --yes run-postinsts-dev')self.pkg('install --yes run-postinsts-dev')
to
self.pkg('update')self.pkg('install --yes run-postinsts-dev')self.pkg('remove --yes run-postinsts-dev')
?I'm not convinced that would help us...
I'll try locally.
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
"bitbake core-image-sato -c testimage"
as well. I'd suggest making the changes in testimage conditional on signing being configured.
Quoted 8 lines
Yes, regardless the above, we need to either make signing always enabled in all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest ifPACKAGE_FEED_GPG_NAMEhas been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?
Quoted 6 lines
We should be able to test
self.tc.td.get('PACKAGE_FEED_GPG_NAME')in the test and handle accordingly?I did merge the base changes into the release since I thought it was fair to get the fixes in before it was built. We just need to get the test sorted now, I think it is close.
Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?
Quoted 7 lines
Cheers,
Richard
Re: [OE-core] [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
In reply to Alexandre BelloniHi
Op 06-04-2022 om 12:10 schreef Alexandre Belloni:
Quoted 125 lines
Hello,
Quoted 122 lines · Richard Purdie
Quoted 101 lines · Ferry Toth
From: Ferry Toth<[email protected]>
Since Gatesgarth apt (1.8.2) has become more strict and doesn’t allow unsigned repositories by default. Currently when building images this requirement is worked around by using [allow-insecure=yes] and equivalently when performing selftest.
Patches "gpg-sign: Add parameters to gpg signature function" and "
package_manager: sign DEB package feeds" enable signed DEB package feeds. This patch adds a runtime test for apt derived from thetest_testimage_dnftest. It creates a signed deb package feed, runs a qemu image to install the key and performs some package management. To be able to install the key the gnupg package is added to the testimage.Signed-off-by Ferry Toth<[email protected]>--- meta/lib/oeqa/runtime/cases/apt.py | 16 ++++++--- meta/lib/oeqa/selftest/cases/runtime_test.py | 38 ++++++++++++++++++++ 2 files changed, 49 insertions(+), 5 deletions(-)meta/lib/oeqa/runtime/cases/apt.py +11
-5@@ -21,7 +21,7 @@ class AptRepoTest(AptTest): @classmethod def setUpClass(cls): - service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all') + service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], '') cls.repo_server = HTTPService(service_repo, '0.0.0.0', port=cls.tc.target.server_port, logger=cls.tc.logger)@@ -32,13 +32,18 @@ class AptRepoTest(AptTest): cls.repo_server.stop() def setup_source_config_for_package_install(self): - apt_get_source_server = 'http://%s:%s/' % (self.tc.target.server_ip, self.repo_server.port) + apt_get_source_server = 'http:\/\/%s:%s' % (self.tc.target.server_ip, self.repo_server.port) apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; echo deb [ allow-insecure=yes ] %s ./ > sources.list' % (apt_get_sourceslist_dir, apt_get_source_server)) + self.target.run("cd %s; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/%s/g' sources.list" % (apt_get_sourceslist_dir, apt_get_source_server)) def cleanup_source_config_for_package_install(self): apt_get_sourceslist_dir = '/etc/apt/' - self.target.run('cd %s; rm sources.list' % (apt_get_sourceslist_dir)) + self.target.run('cd %s; mv sources.list.bak sources.list' % (apt_get_sourceslist_dir)) + + def setup_key(self): + # the key is found on the target /etc/pki/packagefeed-gpg/ + # named PACKAGEFEED-GPG-KEY-poky-branch + self.target.run('cd %s; apt-key add P*' % ('/etc/pki/packagefeed-gpg')) @skipIfNotFeature('package-management', 'Test requires package-management to be in IMAGE_FEATURES')@@ -47,7 +52,8 @@ class AptRepoTest(AptTest): @OEHasPackage(['apt']) def test_apt_install_from_repo(self): self.setup_source_config_for_package_install() + self.setup_key() self.pkg('update') self.pkg('remove --yes run-postinsts-dev') - self.pkg('install --yes --allow-unauthenticated run-postinsts-dev') + self.pkg('install --yes run-postinsts-dev') self.cleanup_source_config_for_package_install()meta/lib/oeqa/selftest/cases/runtime_test.py +31
-0@@ -162,6 +162,44 @@ class TestImage(OESelftestTestCase): bitbake('core-image-full-cmdline socat') bitbake('-c testimage core-image-full-cmdline') + def test_testimage_apt(self): + """ + Summary: Check package feeds functionality for apt + Expected: 1. Check that remote package feeds can be accessed + Product: oe-core + Author: Ferry Toth<[email protected]> + """ + if get_bb_var('DISTRO') == 'poky-tiny': + self.skipTest('core-image-full-cmdline not buildable for poky-tiny') + + features = 'INHERIT += "testimage"\n' + features += 'TEST_SUITES = "ping ssh apt.AptRepoTest.test_apt_install_from_repo"\n' + # We don't yet know what the server ip and port will be - they will be patched + # in at the start of the on-image test + features += 'PACKAGE_FEED_URIS = "http://bogus_ip:bogus_port"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_deb"\n' + # We need gnupg on the target to install keys + features += 'IMAGE_INSTALL:append:pn-core-image-full-cmdline = " gnupg"\n' + + bitbake('gnupg-native -c addto_recipe_sysroot') + + # Enable package feed signing + self.gpg_home = tempfile.mkdtemp(prefix="oeqa-feed-sign-") + self.track_for_cleanup(self.gpg_home) + signing_key_dir = os.path.join(self.testlayer_path, 'files', 'signing') + runCmd('gpgconf --list-dirs --homedir %s; gpg -v --batch --homedir %s --import %s' % (self.gpg_home, self.gpg_home, os.path.join(signing_key_dir, 'key.secret')), native_sysroot=get_bb_var("RECIPE_SYSROOT_NATIVE", "gnupg-native"), shell=True) + features += 'INHERIT += "sign_package_feed"\n' + features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n' + features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase') + features += 'GPG_PATH = "%s"\n' % self.gpg_homemeta/lib/oeqa/selftest/cases/runtime_test.py class TestImage(OESelftestTestCase): · line 205
Quoted 10 lines
meta/lib/oeqa/selftest/cases/runtime_test.py +7
-0+ features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home + self.write_config(features) + + # Build core-image-sato and testimage + bitbake('core-image-full-cmdline socat') + bitbake('-c testimage core-image-full-cmdline') + def test_testimage_virgl_gtk_sdl(self): """ Summary: Check host-assisted accelerate OpenGL functionality in qemu with gtk and SDL frontendsThanks for working on this!
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/5013/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/4975however hopefully these shouldn't be too hard to fix?
Quoted 8 lines
Quoted 1 line
The rest of the build is still running.
I missed it at the time but I believe this is also the cause of:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3352/steps/15/logs/stdio
ERROR: package-index-1.0-r0
do_package_index: GPG exited with code 2: gpg: can't connect to the agent: IPC connect call failed
gpg: skipped "testuser": No secret key
gpg: signing failed: No secret key
This seems related but not exact the same.
It seems do_package_index wants to generate a signed deb repo but no key is provided. But IIUC you have PACKAGE_CLASSES = "package_rpm", so why is runtime_test.TestImage.test_testimage_apt run?
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
In reply to Richard PurdieHi
Op 06-04-2022 om 17:23 schreef Richard Purdie:
Quoted 75 lines · Ferry Toth
Quoted 40 lines
Op 06-04-2022 om 13:40 schreef Richard Purdie:
Quoted 39 lines · Ferry Toth
Quoted 38 lines
Op 04-04-2022 om 22:39 schreef Richard Purdie:
Quoted 22 lines · Ferry Toth
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 21 lines
Quoted 9 lines · Ferry Toth
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/50
13/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/49
75That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-devThat is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
What could cause this? Because the complaint is it can't remove the package because it was not installed.
Quoted 8 lines
Quoted 5 lines
It would be trivial to remove the line
*)
self.pkg('remove --yes run-postinsts-dev')but how could it have passed the test before?
I think the issue is you edited testimage which is a different set of tests
which aren't just called by oe-selftest but by things likeThat would be my first thought too, but...
because the failure seems to be on the lineself.pkg('remove --yes run- postinsts-dev'), that would mean the lineself.pkg('update')passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?Quoted 4 lines
Quoted 3 lines
I was easily able to reproduce this locally and it shows the
setup_source_config_for_package_install()step fails and hence the sources aren't setup correctly, hence the update probably works.not correct, hence works. You lost me here, but I'll try to reproduce.
I mean the command doesn't work correctly. In my local logs I see:
DEBUG: Command: cd
/etc/apt/; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/http:\/\/192.168.7.1:46599/g' sources.list
Status: 1 Output: cp: can't stat 'sources.list': No such file or directory
sed: sources.list: No such file or directoryQuoted 5 lines
Quoted 3 lines
go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with
package_debset as the backend...in conf. But without
PACKAGE_CLASSES,PACKAGE_FEED_GPG_NAME,PACKAGE_FEED_GPG_PASSPHRASE_FILE?Yes.
Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
Yes, regardless the above, we need to either make signing always enabled in
all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest ifPACKAGE_FEED_GPG_NAMEhas been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?We should be able to test
self.tc.td.get('PACKAGE_FEED_GPG_NAME')in the test
and handle accordingly?I did merge the base changes into the release since I thought it was fair to get
the fixes in before it was built. We just need to get the test sorted now, I think it is close.
Quoted 8 lines
Quoted 6 lines
Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?No, the autobuilder runs all the tests in oe-selftest so it should be covered (and is why we saw the failures on the autobuilder).
I was running 'oe-selftest -K -r runtime_test.TestImage.test_testimage_apt' whereas buildbot seems to be running 'apt.AptRepoTest.test_apt_install_from_repo' directly.
However, test_testimage_apt is where keys are setup (keys found in meta-selftest). So, where/when is test_testimage_apt called (or test_testimage_dnf for that matter)?
Quoted 4 lines
Cheers,
Richard
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
Quoted 84 lines · Ferry Toth
Hi
Op 06-04-2022 om 17:23 schreef Richard Purdie:
Quoted 81 lines · Ferry Toth
Quoted 40 lines
Op 06-04-2022 om 13:40 schreef Richard Purdie:
Quoted 39 lines · Ferry Toth
Quoted 38 lines
Op 04-04-2022 om 22:39 schreef Richard Purdie:
Quoted 22 lines · Ferry Toth
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 21 lines
Quoted 9 lines · Ferry Toth
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/50
13/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/49
75That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-devThat is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
What could cause this? Because the complaint is it can't remove the package because it was not installed.
Quoted 8 lines
Quoted 5 lines
It would be trivial to remove the line
*)
self.pkg('remove --yes run-postinsts-dev')but how could it have passed the test before?
I think the issue is you edited testimage which is a different set of tests
which aren't just called by oe-selftest but by things likeThat would be my first thought too, but...
because the failure seems to be on the lineself.pkg('remove --yes run- postinsts-dev'), that would mean the lineself.pkg('update')passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?Quoted 4 lines
Quoted 3 lines
I was easily able to reproduce this locally and it shows the
setup_source_config_for_package_install()step fails and hence the sources aren't setup correctly, hence the update probably works.not correct, hence works. You lost me here, but I'll try to reproduce.
I mean the command doesn't work correctly. In my local logs I see:
DEBUG: Command: cd
/etc/apt/; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/http:\/\/192.168.7.1:46599/g' sources.list
Status: 1 Output: cp: can't stat 'sources.list': No such file or directory
sed: sources.list: No such file or directoryQuoted 5 lines
Quoted 3 lines
go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with
package_debset as the backend...in conf. But without
PACKAGE_CLASSES,PACKAGE_FEED_GPG_NAME,PACKAGE_FEED_GPG_PASSPHRASE_FILE?Yes.
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Yes, regardless the above, we need to either make signing always enabled in
all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest ifPACKAGE_FEED_GPG_NAMEhas been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?Quoted 8 lines
We should be able to test
self.tc.td.get('PACKAGE_FEED_GPG_NAME')in the test
and handle accordingly?I did merge the base changes into the release since I thought it was fair to get
the fixes in before it was built. We just need to get the test sorted now, I think it is close.Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?
Quoted 6 lines
Quoted 2 lines
No, the autobuilder runs all the tests in oe-selftest so it should be covered (and is why we saw the failures on the autobuilder).
I was running 'oe-selftest -K -r runtime_test.TestImage.test_testimage_apt' whereas buildbot seems to be running 'apt.AptRepoTest.test_apt_install_from_repo' directly.
We have several types of test. There are two types in play here,
"oe-selftest -K -r runtime_test.TestImage.test_testimage_apt"
and
"bitbake core-image-sato -c testimage"
The latter testimage tests are often run every time we create images and running testimage will trigger 'apt.AptRepoTest.test_apt_install_from_repo' if the image has apt present and is built using debian package management.
We also run oe-selftest which triggers a testimage of it's own for a specfic test case.
So we'd expect the normal testimage calls to not have the package signing enabled and then we'd have the oe-selftest which specifically tests signing.
Quoted 3 lines
However,
test_testimage_aptis where keys are setup (keys found in meta-selftest). So, where/when istest_testimage_aptcalled (ortest_testimage_dnffor that matter)?
Those are called by the oe-selftest call on the autobuilder. We run the oe- selftest with a mask on the autobuilder so pretty much all of them run.
Cheers,
Richard
Re: [PATCH v2 3/3] apt: add apt selftest to test signed package feeds
Op 06-04-2022 om 23:05 schreef Richard Purdie:
Quoted 71 lines · Ferry Toth
Quoted 70 lines
Hi
Op 06-04-2022 om 17:23 schreef Richard Purdie:
Quoted 67 lines · Ferry Toth
Quoted 40 lines
Op 06-04-2022 om 13:40 schreef Richard Purdie:
Quoted 39 lines · Ferry Toth
Quoted 38 lines
Op 04-04-2022 om 22:39 schreef Richard Purdie:
Quoted 22 lines · Ferry Toth
Op 04-04-2022 om 15:58 schreef Richard Purdie:
Quoted 21 lines
Quoted 9 lines · Ferry Toth
Looking at the patches I wondered if this would break testimage and unfortunately it does:
https://autobuilder.yoctoproject.org/typhoon/#/builders/50/builds/50
13/s
teps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/49
75That is weird, do I understand correctly that it fails on:
apt-get remove --yes run-postinsts-dev
Reading package lists...
Building dependency tree...
E: Unable to locate package run-postinsts-devThat is actually *) one line I didn't touch. I did note while testing that I saw this exact message, however that was not counted as a fail.
What could cause this? Because the complaint is it can't remove the package because it was not installed.
Quoted 8 lines
Quoted 5 lines
It would be trivial to remove the line
*)
self.pkg('remove --yes run-postinsts-dev')but how could it have passed the test before?
I think the issue is you edited testimage which is a different set of tests
which aren't just called by oe-selftest but by things likeThat would be my first thought too, but...
because the failure seems to be on the lineself.pkg('remove --yes run- postinsts-dev'), that would mean the lineself.pkg('update')passed. And that should only pass if it finds a signed repository and has the key installed (and believe me, I saw a log of that in the last week). So, there may be a second thing wrong?Quoted 4 lines
Quoted 3 lines
I was easily able to reproduce this locally and it shows the
setup_source_config_for_package_install()step fails and hence the sources aren't setup correctly, hence the update probably works.not correct, hence works. You lost me here, but I'll try to reproduce.
I mean the command doesn't work correctly. In my local logs I see:
DEBUG: Command: cd
/etc/apt/; cp sources.list sources.list.bak; sed -i 's/\[trusted=yes\] http:\/\/bogus_ip:bogus_port/http:\/\/192.168.7.1:46599/g' sources.list
Status: 1 Output: cp: can't stat 'sources.list': No such file or directory
sed: sources.list: No such file or directoryQuoted 5 lines
Quoted 3 lines
go in and find/share them. The issue does locally reproduce for me with a "bitbake core-image-sato -c testimage" with
package_debset as the backend...in conf. But without
PACKAGE_CLASSES,PACKAGE_FEED_GPG_NAME,PACKAGE_FEED_GPG_PASSPHRASE_FILE?Yes.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Yes, regardless the above, we need to either make signing always enabled in
all test cases or detect whether signing is used.
Do you have a hint if there is a variable to test in class AptRepoTest ifPACKAGE_FEED_GPG_NAMEhas been set?
Otherwise I could just duplicate code and create apt.AptRepoTest.test_apt_install_from_repo_signed.
What would you prefer?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
We should be able to test
self.tc.td.get('PACKAGE_FEED_GPG_NAME')in the test
and handle accordingly?
I'll try this for autodetecting the test case.
Quoted 16 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
I did merge the base changes into the release since I thought it was fair to get
the fixes in before it was built. We just need to get the test sorted now, I think it is close.Thanks for merging.
I'll fix the test, that's only fair.
One thing, the test "test_testimage_apt" is new. It needs to be scheduled somewhere (where "test_testimage_dnf" is called i guess), I didn't add that. Is that correct?Quoted 5 lines
Quoted 2 lines
No, the autobuilder runs all the tests in oe-selftest so it should be covered (and is why we saw the failures on the autobuilder).
I was running 'oe-selftest -K -r runtime_test.TestImage.test_testimage_apt' whereas buildbot seems to be running 'apt.AptRepoTest.test_apt_install_from_repo' directly.
Quoted 7 lines
We have several types of test. There are two types in play here,
"oe-selftest -K -r runtime_test.TestImage.test_testimage_apt"
and
"bitbake core-image-sato -c testimage"
Thanks for the explanation. Yesterday evening I built Sato and was able to reproduce the issue.
(that was a big build with half of build time spent on building Rust).
core-image-sato does not have a /etc/apt/sources.list, the selftest public key is not on the image and gpg is not installed. This explains the errors in the log.
Quoted 6 lines
The latter testimage tests are often run every time we create images and running testimage will trigger 'apt.AptRepoTest.test_apt_install_from_repo' if the image has apt present and is built using debian package management.
We also run oe-selftest which triggers a testimage of it's own for a specfic test case.
Got it.
Quoted 8 lines
So we'd expect the normal testimage calls to not have the package signing enabled and then we'd have the oe-selftest which specifically tests signing.
Quoted 3 lines
However,
test_testimage_aptis where keys are setup (keys found in meta-selftest). So, where/when istest_testimage_aptcalled (ortest_testimage_dnffor that matter)?Those are called by the oe-selftest call on the autobuilder. We run the oe- selftest with a mask on the autobuilder so pretty much all of them run.
After looking at more detail at autobuilder schedules I get it.
Quoted 5 lines
Cheers,
Richard