rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable
PATCH 1/3 rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable
I've used a previous patch (which was never merged) by Humberto Ibarra <[email protected]> as a model for how to do runtime testing of this feature (e.g. we need to boot an image, run dnf on it, and check that it is indeed able to access the remote repo over http). Here's his original commit message:
=====
Testing that feeds specified with PACKAGE_FEED_URIS var are set correctly has two parts. First a build with this var set is required, and then smart update needs to be issued in the running taget.
The previous is not a common selftest practice because this is a simple test, but requires building and running a specific image, which takes a lot of time. testimage is not a good fit either, since the images tested there do not have the PACKAGE_FEED_URIS var set.
For this test, the runtime-test module is being used, which is a selftest module but runs a testimage command. The var and test environment were set in runtime-perf.py and the actual test is done in a new testcase added to meta-selftest layer.
=====
[YOCTO #10872]
--- meta-selftest/lib/oeqa/runtime/cases/selftest.py | 35 ++++++++++++++++++++++++ meta/lib/oe/package_manager.py | 12 +++++++- meta/lib/oeqa/selftest/runtime-test.py | 5 ++++ 3 files changed, 51 insertions(+), 1 deletion(-)
meta-selftest/lib/oeqa/runtime/cases/selftest.py +35 -0
@@ -1,5 +1,7 @@ from oeqa.runtime.case import OERuntimeTestCase from oeqa.core.decorator.depends import OETestDepends +from oeqa.runtime.cases.dnf import DnfTest +from oeqa.utils.httpserver import HTTPService class Selftest(OERuntimeTestCase):
@@ -29,3 +31,36 @@ class Selftest(OERuntimeTestCase): (status, output) = self.target.run("socat -V") self.assertNotEqual(status, 0, msg="socat is still installed") + + +class DnfSelftest(DnfTest): + + @classmethod + def setUpClass(cls): + cls.repo_server = HTTPService(os.path.join(cls.tc.td['WORKDIR'], 'oe-rootfs-repo'), + cls.tc.target.server_ip) + cls.repo_server.port = 8080 + cls.repo_server.start() + + @classmethod + def tearDownClass(cls): + cls.repo_server.stop() + + @OETestDepends(['ssh.SSHTest.test_ssh']) + def test_verify_package_feeds(self): + """ + Summary: Check correct setting of PACKAGE_FEED_URIS var + Expected: 1. Feeds were correctly set for dnf + 2. Update recovers packages from host's repo + Author: Humberto Ibarra <[email protected]> + Author: Alexander Kanavin <[email protected]> + """ + import re + + output_makecache = self.dnf('makecache') + self.assertTrue(re.match(r".*Metadata cache created", output_makecache, re.DOTALL) is not None, msg = "dnf makecache failed: %s" %(output_makecache)) + + output_repoinfo = self.dnf('repoinfo') + matchobj = re.match(r".*Repo-pkgs\s*:\s*(?P<n_pkgs>[0-9]+)", output_repoinfo, re.DOTALL) + self.assertTrue(matchobj is not None, msg = "Could not find the amount of packages in dnf repoinfo output: %s" %(output_repoinfo)) + self.assertTrue(int(matchobj.group('n_pkgs')) > 0, msg = "Amount of remote packages is not more than zero: %s\n" %(output_repoinfo))
meta/lib/oe/package_manager.py +11 -1
@@ -533,10 +533,20 @@ class RpmPM(PackageManager): bb.utils.unlockfile(lf) def insert_feeds_uris(self, feed_uris, feed_base_paths, feed_archs): + from urllib.parse import urlparse + if feed_uris == "": return - raise NotImplementedError("Adding remote dnf feeds not yet supported.") + bb.utils.mkdirhier(oe.path.join(self.target_rootfs, "etc", "yum.repos.d")) + remote_uris = self.construct_uris(feed_uris.split(), feed_base_paths.split()) + for uri in remote_uris: + repo_name = "oe-remote-repo" + "-".join(urlparse(uri).path.split("/")) + if feed_archs is not None: + repo_uris = [uri + "/" + arch for arch in feed_archs] + else: + repo_uris = [uri] + open(oe.path.join(self.target_rootfs, "etc", "yum.repos.d", repo_name + ".repo"), 'w').write("[%s]\nbaseurl=%s\n" % (repo_name, " ".join(repo_uris))) def _prepare_pkg_transaction(self): os.environ['D'] = self.target_rootfs
meta/lib/oeqa/selftest/runtime-test.py +5 -0
@@ -108,14 +108,19 @@ class TestImage(oeSelfTest): Summary: Check install packages functionality for testimage/testexport. Expected: 1. Import tests from a directory other than meta. 2. Check install/uninstall of socat. + 3. Check that remote package feeds can be accessed Product: oe-core Author: Mariano Lopez <[email protected]> + Author: Alexander Kanavin <[email protected]> """ if get_bb_var('DISTRO') == 'poky-tiny': self.skipTest('core-image-full-cmdline not buildable for poky-tiny') features = 'INHERIT += "testimage"\n' features += 'TEST_SUITES = "ping ssh selftest"\n' + features += 'PACKAGE_FEED_URIS = "http://192.168.7.1:8080"\n' + features += 'EXTRA_IMAGE_FEATURES += "package-management"\n' + features += 'PACKAGE_CLASSES = "package_rpm"' self.write_config(features) # Build core-image-sato and testimage
Signature · 1 line
2.11.0
PATCH 2/3 lib/oe/package_manager: import rpm signing key to rpmdb
From: Markus Lehtonen <[email protected]>
Import the gpg key used in rpm signing into rpmdb. This makes it possible again to create images when rpm signing is enabled.
Also, instruct dnf to enforce signature check if rpm signing is enabled.
--- meta/lib/oe/package_manager.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
meta/lib/oe/package_manager.py +10 -2
@@ -520,7 +520,14 @@ class RpmPM(PackageManager): open(platformconfdir + "macros", 'a').write("%_prefer_color 7") if self.d.getVar('RPM_SIGN_PACKAGES') == '1': - raise NotImplementedError("Signature verification with rpm not yet supported.") + pubkey_path = self.d.getVar('RPM_GPG_PUBKEY') + rpm_bin = bb.utils.which(os.getenv('PATH'), "rpmkeys") + cmd = [rpm_bin, '--root=%s' % self.target_rootfs, '--import', pubkey_path] + try: + subprocess.check_output(cmd, stderr=subprocess.STDOUT) + except subprocess.CalledProcessError as e: + bb.fatal("Importing GPG key failed. Command '%s' " + "returned %d:\n%s" % (' '.join(cmd), e.returncode, e.output.decode("utf-8"))) def create_configs(self): self._configure_dnf()
@@ -570,7 +577,8 @@ class RpmPM(PackageManager): output = self._invoke_dnf((["--skip-broken"] if attempt_only else []) + (["-x", ",".join(exclude_pkgs)] if len(exclude_pkgs) > 0 else []) + (["--setopt=install_weak_deps=False"] if self.d.getVar('NO_RECOMMENDATIONS') == 1 else []) + - ["--nogpgcheck", "install"] + + (["--nogpgcheck"] if self.d.getVar('RPM_SIGN_PACKAGES') != '1' else ["--setopt=gpgcheck=True"]) + + ["install"] + pkgs) failed_scriptlets_pkgnames = collections.OrderedDict()
Signature · 1 line
2.11.0
PATCH 3/3 selftest/signing.py: verify that images can be created from signed packages
In reply to Alexander Kanavin--- meta/lib/oeqa/selftest/signing.py | 8 ++++++++ 1 file changed, 8 insertions(+)
meta/lib/oeqa/selftest/signing.py +8 -0
@@ -34,8 +34,10 @@ class Signing(oeSelfTest): """ Summary: Test that packages can be signed in the package feed Expected: Package should be signed with the correct key + Expected: Images can be created from signed packages Product: oe-core Author: Daniel Istrate <[email protected]> + Author: Alexander Kanavin <[email protected]> AutomatedBy: Daniel Istrate <[email protected]> """ import oe.packagedata
@@ -85,6 +87,12 @@ class Signing(oeSelfTest): self.assertIn('rsa sha1 (md5) pgp md5 OK', ret.output, 'Package signed incorrectly.') shutil.rmtree(rpmdb) + #Check that an image can be built from signed packages + self.add_command_to_tearDown('bitbake -c clean core-image-minimal') + bitbake('-c clean core-image-minimal') + bitbake('core-image-minimal') + + @testcase(1382) def test_signing_sstate_archive(self): """
Signature · 1 line
2.11.0
Quoted 2 lines · Alexander Kanavin
+ cls.repo_server.port = 8080
What if parallel selftests are running at once? Can this service be started before the image is created, so the right port can be written into the image?
Ross
Quoted 7 lines · Ross
+ cls.repo_server.port = 8080
What if parallel selftests are running at once? Can this service be started before the image is created, so the right port can be written into the image?
I don't know. I only followed what Humberto wrote in the original patch; we need to ask him. I'm also not entirely certain that hardcoding the ip address will always work, and did not get an answer to that.
Another, more hackish option is to monkey patch the ephemeral port before running dnf into the repo config on the image.
Alex
Quoted 9 lines · Alexander Kanavin
Quoted 7 lines · Ross
+ cls.repo_server.port = 8080
What if parallel selftests are running at once? Can this service be started before the image is created, so the right port can be written into the image?
Quoted 6 lines
I don't know. I only followed what Humberto wrote in the original patch; we need to ask him. I'm also not entirely certain that hardcoding the ip address will always work, and did not get an answer to that.
Another, more hackish option is to monkey patch the ephemeral port before running dnf into the repo config on the image.
Or perhaps use ssh with port forwarding? Then the host-side port can be picked randomly, while on the target side it can be fixed.
I was about to suggest qemu slirp networking because that also does port mapping, but I think ssh port forwarding is better.
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Sorry, missed the previous mail.
I am not sure how to improve the hardcoding there, I don't like it either, but couldn't find an alternative. The thing is that we need the ip of the host as seen from the target to run testimage, but we don’t have the exact value for that one until qemu is up. The issue here is that we need to give this ip to the conf in the PACKAGE_FEED_URIS before building so we can get an image with this var to later use in testimage.
If someone has a better idea to get rid of this hardcoding, it would be great to know.
-----Original Message-----
From: Alexander Kanavin [mailto:[email protected]]
Sent: Wednesday, March 15, 2017 5:19 AM
To: Burton, Ross <[email protected]>
Cc: OE-core <[email protected]>; Ibarra Lopez, Humberto <[email protected]>
Subject: Re: [OE-core] [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable
Quoted 7 lines · Ross
+ cls.repo_server.port = 8080
What if parallel selftests are running at once? Can this service be started before the image is created, so the right port can be written into the image?
I don't know. I only followed what Humberto wrote in the original patch; we need to ask him. I'm also not entirely certain that hardcoding the ip address will always work, and did not get an answer to that.
Another, more hackish option is to monkey patch the ephemeral port before running dnf into the repo config on the image.
Alex
Quoted 10 lines · Humberto
Sorry, missed the previous mail.
I am not sure how to improve the hardcoding there, I don't like it either, but couldn't find an alternative. The thing is that we need the ip of the host as seen from the target to run testimage, but we don’t have the exact value for that one until qemu is up. The issue here is that we need to give this ip to the conf in the
PACKAGE_FEED_URISbefore building so we can get an image with this var to later use in testimage.
Quoted 2 lines
If someone has a better idea to get rid of this hardcoding, it would be great to know.
We could run a sed command inside the image after it boots but before triggering the tests to put the correct IP/port number into the file?
Not a perfect test but simple and probably good enough for this?
Cheers,
Richard
Yeah, it does sound off but we could use it. We could put a mock value in the ip to the PACKAGE_FEED_URIS var to build this image and once it boths update that value with the real one. We would have to check that the mock value was correctly set before updating so we can test the var being correctly set; and then check that the feeds are updated correctly once a valid ip is there. I will look into this.
Humberto
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Wednesday, March 15, 2017 10:08 AM
To: Ibarra Lopez, Humberto <[email protected]>; Alexander Kanavin <[email protected]>; Burton, Ross <[email protected]>
Cc: OE-core <[email protected]>
Subject: Re: [OE-core] [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable
Quoted 10 lines · Humberto
Sorry, missed the previous mail.
I am not sure how to improve the hardcoding there, I don't like it either, but couldn't find an alternative. The thing is that we need the ip of the host as seen from the target to run testimage, but we don’t have the exact value for that one until qemu is up. The issue here is that we need to give this ip to the conf in the
PACKAGE_FEED_URISbefore building so we can get an image with this var to later use in testimage.
Quoted 2 lines
If someone has a better idea to get rid of this hardcoding, it would be great to know.
We could run a sed command inside the image after it boots but before triggering the tests to put the correct IP/port number into the file?
Not a perfect test but simple and probably good enough for this?
Cheers,
Richard