Skip to contentSkip to conversation

rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

PATCH In review Alexander Kanavin 5 participants 9 messages Mar 14, 2017 – Mar 15, 2017

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

I've used a previous patch (which was never merged) by Humberto Ibarra <[email protected]> as a model for how to do runtime testing of this feature (e.g. we need to boot an image, run dnf on it, and check that it is indeed able to access the remote repo over http). Here's his original commit message:

=====
Testing that feeds specified with PACKAGE_FEED_URIS var are set correctly has two parts. First a build with this var set is required, and then smart update needs to be issued in the running taget.

The previous is not a common selftest practice because this is a simple test, but requires building and running a specific image, which takes a lot of time. testimage is not a good fit either, since the images tested there do not have the PACKAGE_FEED_URIS var set.

For this test, the runtime-test module is being used, which is a selftest module but runs a testimage command. The var and test environment were set in runtime-perf.py and the actual test is done in a new testcase added to meta-selftest layer.
=====

[YOCTO #10872]

---
 meta-selftest/lib/oeqa/runtime/cases/selftest.py | 35 ++++++++++++++++++++++++
 meta/lib/oe/package_manager.py                   | 12 +++++++-
 meta/lib/oeqa/selftest/runtime-test.py           |  5 ++++
 3 files changed, 51 insertions(+), 1 deletion(-)
meta-selftest/lib/oeqa/runtime/cases/selftest.py +35 -0
index 19de7406236..e872ed85b12 100644
@@ -1,5 +1,7 @@
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
+from oeqa.runtime.cases.dnf import DnfTest
+from oeqa.utils.httpserver import HTTPService
 
 class Selftest(OERuntimeTestCase):
 
@@ -29,3 +31,36 @@ class Selftest(OERuntimeTestCase):
 
         (status, output) = self.target.run("socat -V")
         self.assertNotEqual(status, 0, msg="socat is still installed")
+
+
+class DnfSelftest(DnfTest):
+
+    @classmethod
+    def setUpClass(cls):
+        cls.repo_server = HTTPService(os.path.join(cls.tc.td['WORKDIR'], 'oe-rootfs-repo'),
+                                      cls.tc.target.server_ip)
+        cls.repo_server.port = 8080
+        cls.repo_server.start()
+
+    @classmethod
+    def tearDownClass(cls):
+        cls.repo_server.stop()
+
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
+    def test_verify_package_feeds(self):
+        """
+        Summary: Check correct setting of PACKAGE_FEED_URIS var
+        Expected: 1. Feeds were correctly set for dnf
+                  2. Update recovers packages from host's repo
+        Author: Humberto Ibarra <[email protected]>
+        Author: Alexander Kanavin <[email protected]>
+        """
+        import re
+
+        output_makecache = self.dnf('makecache')
+        self.assertTrue(re.match(r".*Metadata cache created", output_makecache, re.DOTALL) is not None, msg = "dnf makecache failed: %s" %(output_makecache))
+
+        output_repoinfo = self.dnf('repoinfo')
+        matchobj = re.match(r".*Repo-pkgs\s*:\s*(?P<n_pkgs>[0-9]+)", output_repoinfo, re.DOTALL)
+        self.assertTrue(matchobj is not None, msg = "Could not find the amount of packages in dnf repoinfo output: %s" %(output_repoinfo))
+        self.assertTrue(int(matchobj.group('n_pkgs')) > 0, msg = "Amount of remote packages is not more than zero: %s\n" %(output_repoinfo))
meta/lib/oe/package_manager.py +11 -1
index d51609189d0..b016bc32dc5 100644
@@ -533,10 +533,20 @@ class RpmPM(PackageManager):
         bb.utils.unlockfile(lf)
 
     def insert_feeds_uris(self, feed_uris, feed_base_paths, feed_archs):
+        from urllib.parse import urlparse
+
         if feed_uris == "":
             return
 
-        raise NotImplementedError("Adding remote dnf feeds not yet supported.")
+        bb.utils.mkdirhier(oe.path.join(self.target_rootfs, "etc", "yum.repos.d"))
+        remote_uris = self.construct_uris(feed_uris.split(), feed_base_paths.split())
+        for uri in remote_uris:
+            repo_name = "oe-remote-repo" + "-".join(urlparse(uri).path.split("/"))
+            if feed_archs is not None:
+                repo_uris = [uri + "/" + arch for arch in feed_archs]
+            else:
+                repo_uris = [uri]
+            open(oe.path.join(self.target_rootfs, "etc", "yum.repos.d", repo_name + ".repo"), 'w').write("[%s]\nbaseurl=%s\n" % (repo_name, " ".join(repo_uris)))
 
     def _prepare_pkg_transaction(self):
         os.environ['D'] = self.target_rootfs
meta/lib/oeqa/selftest/runtime-test.py +5 -0
index e8b483d7f8b..d23cb7cf46f 100644
@@ -108,14 +108,19 @@ class TestImage(oeSelfTest):
         Summary: Check install packages functionality for testimage/testexport.
         Expected: 1. Import tests from a directory other than meta.
                   2. Check install/uninstall of socat.
+                  3. Check that remote package feeds can be accessed
         Product: oe-core
         Author: Mariano Lopez <[email protected]>
+        Author: Alexander Kanavin <[email protected]>
         """
         if get_bb_var('DISTRO') == 'poky-tiny':
             self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
 
         features = 'INHERIT += "testimage"\n'
         features += 'TEST_SUITES = "ping ssh selftest"\n'
+        features += 'PACKAGE_FEED_URIS = "http://192.168.7.1:8080"\n'
+        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
+        features += 'PACKAGE_CLASSES = "package_rpm"'
         self.write_config(features)
 
         # Build core-image-sato and testimage
Signature · 1 line
2.11.0
#1 lore raw

PATCH 2/3 lib/oe/package_manager: import rpm signing key to rpmdb

From: Markus Lehtonen <[email protected]>

Import the gpg key used in rpm signing into rpmdb. This makes it possible again to create images when rpm signing is enabled.

Also, instruct dnf to enforce signature check if rpm signing is enabled.

---
 meta/lib/oe/package_manager.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
meta/lib/oe/package_manager.py +10 -2
index b016bc32dc5..eeb4c76071a 100644
@@ -520,7 +520,14 @@ class RpmPM(PackageManager):
             open(platformconfdir + "macros", 'a').write("%_prefer_color 7")
 
         if self.d.getVar('RPM_SIGN_PACKAGES') == '1':
-            raise NotImplementedError("Signature verification with rpm not yet supported.")
+            pubkey_path = self.d.getVar('RPM_GPG_PUBKEY')
+            rpm_bin = bb.utils.which(os.getenv('PATH'), "rpmkeys")
+            cmd = [rpm_bin, '--root=%s' % self.target_rootfs, '--import', pubkey_path]
+            try:
+                subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+            except subprocess.CalledProcessError as e:
+                bb.fatal("Importing GPG key failed. Command '%s' "
+                        "returned %d:\n%s" % (' '.join(cmd), e.returncode, e.output.decode("utf-8")))
 
     def create_configs(self):
         self._configure_dnf()
@@ -570,7 +577,8 @@ class RpmPM(PackageManager):
         output = self._invoke_dnf((["--skip-broken"] if attempt_only else []) +
                          (["-x", ",".join(exclude_pkgs)] if len(exclude_pkgs) > 0 else []) +
                          (["--setopt=install_weak_deps=False"] if self.d.getVar('NO_RECOMMENDATIONS') == 1 else []) +
-                         ["--nogpgcheck", "install"] +
+                         (["--nogpgcheck"] if self.d.getVar('RPM_SIGN_PACKAGES') != '1' else ["--setopt=gpgcheck=True"]) +
+                         ["install"] +
                          pkgs)
 
         failed_scriptlets_pkgnames = collections.OrderedDict()
Signature · 1 line
2.11.0
#2 lore raw

PATCH 3/3 selftest/signing.py: verify that images can be created from signed packages

In reply to Alexander Kanavin
---
 meta/lib/oeqa/selftest/signing.py | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/lib/oeqa/selftest/signing.py +8 -0
index 38b2fca8cf9..8a106aa80f6 100644
@@ -34,8 +34,10 @@ class Signing(oeSelfTest):
         """
         Summary:     Test that packages can be signed in the package feed
         Expected:    Package should be signed with the correct key
+        Expected:    Images can be created from signed packages
         Product:     oe-core
         Author:      Daniel Istrate <[email protected]>
+        Author:      Alexander Kanavin <[email protected]>
         AutomatedBy: Daniel Istrate <[email protected]>
         """
         import oe.packagedata
@@ -85,6 +87,12 @@ class Signing(oeSelfTest):
         self.assertIn('rsa sha1 (md5) pgp md5 OK', ret.output, 'Package signed incorrectly.')
         shutil.rmtree(rpmdb)
 
+        #Check that an image can be built from signed packages
+        self.add_command_to_tearDown('bitbake -c clean core-image-minimal')
+        bitbake('-c clean core-image-minimal')
+        bitbake('core-image-minimal')
+
+
     @testcase(1382)
     def test_signing_sstate_archive(self):
         """
Signature · 1 line
2.11.0
#3 lore raw
In reply to Alexander Kanavin
Quoted 2 lines · Alexander Kanavin

+ cls.repo_server.port = 8080

What if parallel selftests are running at once? Can this service be started before the image is created, so the right port can be written into the image?

Ross

#4 lore raw
Quoted 7 lines · Ross

+ cls.repo_server.port = 8080

What if parallel selftests are running at once? Can this service be started before the image is created, so the right port can be written into the image?

I don't know. I only followed what Humberto wrote in the original patch; we need to ask him. I'm also not entirely certain that hardcoding the ip address will always work, and did not get an answer to that.

Another, more hackish option is to monkey patch the ephemeral port before running dnf into the repo config on the image.

Alex

#5 lore raw
Quoted 9 lines · Alexander Kanavin
Quoted 7 lines · Ross

+ cls.repo_server.port = 8080

What if parallel selftests are running at once? Can this service be started before the image is created, so the right port can be written into the image?

Quoted 6 lines

I don't know. I only followed what Humberto wrote in the original patch; we need to ask him. I'm also not entirely certain that hardcoding the ip address will always work, and did not get an answer to that.

Another, more hackish option is to monkey patch the ephemeral port before running dnf into the repo config on the image.

Or perhaps use ssh with port forwarding? Then the host-side port can be picked randomly, while on the target side it can be fixed.

I was about to suggest qemu slirp networking because that also does port mapping, but I think ssh port forwarding is better.

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#6 lore raw
In reply to Alexander Kanavin

Sorry, missed the previous mail.

I am not sure how to improve the hardcoding there, I don't like it either, but couldn't find an alternative. The thing is that we need the ip of the host as seen from the target to run testimage, but we don’t have the exact value for that one until qemu is up. The issue here is that we need to give this ip to the conf in the PACKAGE_FEED_URIS before building so we can get an image with this var to later use in testimage.

If someone has a better idea to get rid of this hardcoding, it would be great to know.

-----Original Message-----
From: Alexander Kanavin [mailto:[email protected]]
Sent: Wednesday, March 15, 2017 5:19 AM
To: Burton, Ross <[email protected]>
Cc: OE-core <[email protected]>; Ibarra Lopez, Humberto <[email protected]>
Subject: Re: [OE-core] [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

Quoted 7 lines · Ross

+ cls.repo_server.port = 8080

What if parallel selftests are running at once? Can this service be started before the image is created, so the right port can be written into the image?

I don't know. I only followed what Humberto wrote in the original patch; we need to ask him. I'm also not entirely certain that hardcoding the ip address will always work, and did not get an answer to that.

Another, more hackish option is to monkey patch the ephemeral port before running dnf into the repo config on the image.

Alex

#7 lore raw
Quoted 10 lines · Humberto

Sorry, missed the previous mail.

I am not sure how to improve the hardcoding there, I don't like it either, but couldn't find an alternative. The thing is that we need the ip of the host as seen from the target to run testimage, but we don’t have the exact value for that one until qemu is up. The issue here is that we need to give this ip to the conf in the PACKAGE_FEED_URIS before building so we can get an image with this var to later use in testimage.

Quoted 2 lines

If someone has a better idea to get rid of this hardcoding, it would be great to know.

We could run a sed command inside the image after it boots but before triggering the tests to put the correct IP/port number into the file?

Not a perfect test but simple and probably good enough for this?

Cheers,

Richard

#8 lore raw

Yeah, it does sound off but we could use it. We could put a mock value in the ip to the PACKAGE_FEED_URIS var to build this image and once it boths update that value with the real one. We would have to check that the mock value was correctly set before updating so we can test the var being correctly set; and then check that the feeds are updated correctly once a valid ip is there. I will look into this.

Humberto

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Wednesday, March 15, 2017 10:08 AM
To: Ibarra Lopez, Humberto <[email protected]>; Alexander Kanavin <[email protected]>; Burton, Ross <[email protected]>
Cc: OE-core <[email protected]>
Subject: Re: [OE-core] [PATCH 1/3] rpm: add support for remote package feeds via PACKAGE_FEED_URIS variable

Quoted 10 lines · Humberto

Sorry, missed the previous mail.

I am not sure how to improve the hardcoding there, I don't like it either, but couldn't find an alternative. The thing is that we need the ip of the host as seen from the target to run testimage, but we don’t have the exact value for that one until qemu is up. The issue here is that we need to give this ip to the conf in the PACKAGE_FEED_URIS before building so we can get an image with this var to later use in testimage.

Quoted 2 lines

If someone has a better idea to get rid of this hardcoding, it would be great to know.

We could run a sed command inside the image after it boots but before triggering the tests to put the correct IP/port number into the file?

Not a perfect test but simple and probably good enough for this?

Cheers,

Richard