Skip to contentSkip to conversation

support OEQA inside container

PATCH New André Draszik 1 participant 5 messages Dec 12, 2019

lore
Share & tools
lore
#0 lore raw

cover support OEQA inside container

Hi,

With these patches it is possible to run bitbake inside a docker container and still successfully execute the OEQA tests that require the target to talk to a temporary http server spawned on the machine running bitake, e.g. using crops.

This wasn't possible so far because:
* we need to map a port from outside the container into the container for the target to be able to reach the http server
* we need to pass the IP address of the docker host to the target, e.g. in the OPKG repository URL, but we can't bind the http server to that IP address, as the http server is running inside the container with a different IP address - the bind would fail

To make this work, TEST_SERVER_IP needs to be set to the IP address of the docker host, and the port that is mapped into the container can now also be specified in TEST_SERVER_IP, similar to TEST_TARGET_IP

Cheers,
Andre'

#1 lore raw

PATCH 1/4 oeqa/utils/httpserver: allow to pass in listening port

Being able to specify the listening port is useful when running OEQA from within a docker container, e.g. crops or any other solution.
In that case, a port on the outside must be mapped to a specific port inside the container. If no port is specified for the http server module in this case, the http server would choose a random port, which is unlikely to be mapped and thus won't be reachable from the outside.

---
 meta/lib/oeqa/utils/httpserver.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oeqa/utils/httpserver.py +2 -2
index aa435590f0..58d3c3b3f8 100644
@@ -22,10 +22,10 @@ class HTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
 
 class HTTPService(object):
 
-    def __init__(self, root_dir, host='', logger=None):
+    def __init__(self, root_dir, host='', port=0, logger=None):
         self.root_dir = root_dir
         self.host = host
-        self.port = 0
+        self.port = port
         self.logger = logger
 
     def start(self):
Signature · 1 line
2.24.0
#2 lore raw

PATCH 2/4 oeqa/runtime/context.py: support listening port in TEST_SERVER_IP

In reply to André Draszik

Similar to the existing possibility to specify a port in TEST_TARGET_IP, allow TEST_SERVER_IP to also contain a port.

The intention is for this port to be passed into e.g. the http server from the apt / dnf / opkg tests, or any other (custom) tests that might need the target to connect to a service spawned by bitbake / oeqa, where bitbake is actually running inside a docker container.

---
 meta/lib/oeqa/runtime/context.py | 6 ++++++
 1 file changed, 6 insertions(+)
meta/lib/oeqa/runtime/context.py +6 -0
index ef738a3359..2ecb1a8f01 100644
@@ -98,6 +98,12 @@ class OERuntimeTestContextExecutor(OETestContextExecutor):
                 target_ip = target_ip_port[0]
                 kwargs['port'] = target_ip_port[1]
 
+        if server_ip:
+            server_ip_port = server_ip.split(':')
+            if len(server_ip_port) == 2:
+                server_ip = server_ip_port[0]
+                kwargs['server_port'] = int(server_ip_port[1])
+
         if target_type == 'simpleremote':
             target = OESSHTarget(logger, target_ip, server_ip, **kwargs)
         elif target_type == 'qemu':
Signature · 1 line
2.24.0
#3 lore raw

PATCH 3/4 oeqa/target/ssh oeqa/target/qemu: expose server listening port to tests

In reply to André Draszik

Allow tests to access the listening port as just introduced. Note that when using qemu this infrastructure shouldn't be needed, but we still need to set the port to 0 so that a listening port is determined automatically (e.g. by the python http server).

---
 meta/lib/oeqa/core/target/qemu.py | 1 +
 meta/lib/oeqa/core/target/ssh.py  | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)
meta/lib/oeqa/core/target/qemu.py +1 -0
index 081c627b01..758703c0d1 100644
@@ -24,6 +24,7 @@ class OEQemuTarget(OESSHTarget):
                 user, port)
 
         self.server_ip = server_ip
+        self.server_port = 0
         self.machine = machine
         self.rootfs = rootfs
         self.kernel = kernel
meta/lib/oeqa/core/target/ssh.py +2 -1
index 51032ef1a9..63fc9468b3 100644
@@ -15,7 +15,7 @@ from . import OETarget
 
 class OESSHTarget(OETarget):
     def __init__(self, logger, ip, server_ip, timeout=300, user='root',
-                 port=None, **kwargs):
+                 port=None, server_port=0, **kwargs):
         if not logger:
             logger = logging.getLogger('target')
             logger.setLevel(logging.INFO)
@@ -30,6 +30,7 @@ class OESSHTarget(OETarget):
         super(OESSHTarget, self).__init__(logger)
         self.ip = ip
         self.server_ip = server_ip
+        self.server_port = server_port
         self.timeout = timeout
         self.user = user
         ssh_options = [
Signature · 1 line
2.24.0
#4 lore raw

PATCH 4/4 oeqa/runtime/apt dnf opkg: support running from within docker container

In reply to André Draszik

If the user specified an http port to use for serving files, use that instead of a random one. At the same time, have the http server bind to all interfaces.

Binding to the server_ip might not always be possible, e.g. in the case of running bitbake / oeqa from within a docker container. In this case, the ip address is valid outside the container, but not inside, and hence can't be bound to. So switch to simply binding to all interfaces.

---
 meta/lib/oeqa/runtime/cases/apt.py  | 4 +++-
 meta/lib/oeqa/runtime/cases/dnf.py  | 3 ++-
 meta/lib/oeqa/runtime/cases/opkg.py | 4 +++-
 3 files changed, 8 insertions(+), 3 deletions(-)
meta/lib/oeqa/runtime/cases/apt.py +3 -1
index 74a940d80f..c5378d90c3 100644
@@ -22,7 +22,9 @@ class AptRepoTest(AptTest):
     @classmethod
     def setUpClass(cls):
         service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_DEB'], 'all')
-        cls.repo_server = HTTPService(service_repo, cls.tc.target.server_ip, logger=cls.tc.logger)
+        cls.repo_server = HTTPService(service_repo,
+                                      '0.0.0.0', port=cls.tc.target.server_port,
+                                      logger=cls.tc.logger)
         cls.repo_server.start()
 
     @classmethod
meta/lib/oeqa/runtime/cases/dnf.py +2 -1
index de3759995e..f40c63026e 100644
@@ -53,7 +53,8 @@ class DnfRepoTest(DnfTest):
     @classmethod
     def setUpClass(cls):
         cls.repo_server = HTTPService(os.path.join(cls.tc.td['WORKDIR'], 'oe-testimage-repo'),
-                                      cls.tc.target.server_ip, logger=cls.tc.logger)
+                                      '0.0.0.0', port=cls.tc.target.server_port,
+                                      logger=cls.tc.logger)
         cls.repo_server.start()
 
     @classmethod
meta/lib/oeqa/runtime/cases/opkg.py +3 -1
index 750706161b..9cfee1cd88 100644
@@ -25,7 +25,9 @@ class OpkgRepoTest(OpkgTest):
         if cls.tc.td["MULTILIB_VARIANTS"]:
             allarchfeed = cls.tc.td["TUNE_PKGARCH"]
         service_repo = os.path.join(cls.tc.td['DEPLOY_DIR_IPK'], allarchfeed)
-        cls.repo_server = HTTPService(service_repo, cls.tc.target.server_ip, logger=cls.tc.logger)
+        cls.repo_server = HTTPService(service_repo,
+                                      '0.0.0.0', port=cls.tc.target.server_port,
+                                      logger=cls.tc.logger)
         cls.repo_server.start()
 
     @classmethod
Signature · 1 line
2.24.0