Allows to copy symlinks to DUTs with testimage
cover Allows to copy symlinks to DUTs with testimage
From: Mariano Lopez <[email protected]>
Currently when using the install functionality in the DUT without package manager, if you copy a symlink the function will fail.
These series will change the behaviour and will allow to recreate the symlinks in the DUT.
[YOCTO #9932]
The following changes since commit 039f47ad197a9a53109c9f3deadd9c35e62c056d:
uclibc: remove meta-yocto-bsp append (2016-07-26 08:56:32 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib mariano/bug9932
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug9932
Mariano Lopez (3):
oeqa/utils/commands.py: Add oeqa_copy function oeqa/oetest.py: Allow to export packages using symlinks oeqa/utils/sshcontrol.py: Allows to copy symlinks to target
meta/classes/testexport.bbclass | 22 ++++------------------ meta/lib/oeqa/oetest.py | 16 ++++++++++++---- meta/lib/oeqa/utils/commands.py | 9 +++++++++ meta/lib/oeqa/utils/sshcontrol.py | 9 +++++++-- 4 files changed, 32 insertions(+), 24 deletions(-)
Signature · 1 line
2.6.6
PATCH 1/3 oeqa/utils/commands.py: Add oeqa_copy function
From: Mariano Lopez <[email protected]>
This new function is just a call to shutil.copy2() with follow_symlinks set to False. This function is needed when using shutil.copytree() to preserve the symlinks.
[YOCTO #9932]
--- meta/lib/oeqa/utils/commands.py | 9 +++++++++ 1 file changed, 9 insertions(+)
meta/lib/oeqa/utils/commands.py +9 -0
@@ -18,6 +18,7 @@ from oeqa.utils import CommandError from oeqa.utils import ftools import re import contextlib +import shutil # Export test doesn't require bb try: import bb
@@ -273,3 +274,11 @@ def updateEnv(env_file): for line in result.output.split("\0"): (key, _, value) = line.partition("=") os.environ[key] = value + +def oeqa_copy(src, dst): + """ + Copy files and symlinks trying to preserve metadata. + """ + + shutil.copy2(src, dst, follow_symlinks=False) +
Signature · 1 line
2.6.6
Re: [PATCH 1/3] oeqa/utils/commands.py: Add oeqa_copy function
meta/lib/oeqa/utils/commands.py def updateEnv(env_file): · line 283
Quoted 7 lines · On 26 July 2016 at 08:39, <[email protected]> wrote:
meta/lib/oeqa/utils/commands.py +6
-0+def oeqa_copy(src, dst): + """ + Copy files and symlinks trying to preserve metadata. + """ + + shutil.copy2(src, dst, follow_symlinks=False)
Why not just use shutil directly instead of having a wrapper function?
Ross
Re: [PATCH 1/3] oeqa/utils/commands.py: Add oeqa_copy function
Quoted 11 lines · Ross
On 26 July 2016 at 08:39, <[email protected] <mailto:[email protected]>> wrote:
+def
oeqa_copy(src, dst):
+ """
+ Copy files and symlinks trying to preserve metadata.
+ """
+
+shutil.copy2(src, dst, follow_symlinks=False)
Quoted 1 line
Why not just use shutil directly instead of having a wrapper function?
shutil.copytree didn't play nice with the symlinks, this is why I added this function, nevertheless, I like Joshua's approach of use oe.path.copytree(), so this function won't be needed.
Quoted 2 lines
Ross
Mariano
PATCH 2/3 oeqa/oetest.py: Allow to export packages using symlinks
In reply to [email protected]From: Mariano Lopez <[email protected]>
Currently packages that contains symlinks can't be extracted and exported. This allows to export extracted such packages.
A nice side effect is improved readability.
[YOCTO #9932]
--- meta/classes/testexport.bbclass | 22 ++++------------------ meta/lib/oeqa/oetest.py | 16 ++++++++++++---- 2 files changed, 16 insertions(+), 22 deletions(-)
meta/classes/testexport.bbclass +4 -18
@@ -47,6 +47,7 @@ def exportTests(d,tc): import shutil import pkgutil import re + from oeqa.utils.commands import oeqa_copy exportpath = d.getVar("TEST_EXPORT_DIR", True)
@@ -132,27 +133,12 @@ def exportTests(d,tc): create_tarball(d, "testexport.tar.gz", d.getVar("TEST_EXPORT_DIR", True)) # Copy packages needed for runtime testing - export_pkg_dir = os.path.join(d.getVar("TEST_EXPORT_DIR", True), "packages") test_pkg_dir = d.getVar("TEST_NEEDED_PACKAGES_DIR", True) - need_pkg_dir = False - for root, subdirs, files in os.walk(test_pkg_dir): - for subdir in subdirs: - tmp_dir = os.path.join(root.replace(test_pkg_dir, "").lstrip("/"), subdir) - new_dir = os.path.join(export_pkg_dir, tmp_dir) - bb.utils.mkdirhier(new_dir) - - for f in files: - need_pkg_dir = True - src_f = os.path.join(root, f) - dst_f = os.path.join(export_pkg_dir, root.replace(test_pkg_dir, "").lstrip("/"), f) - shutil.copy2(src_f, dst_f) - - if need_pkg_dir: + if os.listdir(test_pkg_dir): + export_pkg_dir = os.path.join(d.getVar("TEST_EXPORT_DIR", True), "packages") + shutil.copytree(test_pkg_dir, export_pkg_dir, copy_function=oeqa_copy) # Create tar file for packages needed by the DUT create_tarball(d, "testexport_packages_%s.tar.gz" % d.getVar("MACHINE", True), export_pkg_dir) - else: - # Remov packages dir from exported test - bb.utils.remove(export_pkg_dir, True) # Copy SDK if d.getVar("TEST_EXPORT_SDK_ENABLED", True) == "1":
meta/lib/oeqa/oetest.py +12 -4
@@ -26,6 +26,7 @@ try: import oeqa.sdkext except ImportError: pass +from oeqa.utils.commands import oeqa_copy from oeqa.utils.decorators import LogResults, gettag, getResults from oeqa.utils import avoid_paths_in_environ
@@ -447,6 +448,8 @@ class RuntimeTestContext(TestContext): modules = self.getTestModules() bbpaths = self.d.getVar("BBPATH", True).split(":") + shutil.rmtree(self.d.getVar("TEST_EXTRACTED_DIR", True)) + shutil.rmtree(self.d.getVar("TEST_PACKAGED_DIR", True)) for module in modules: json_file = self._getJsonFile(module) if json_file:
@@ -481,13 +484,18 @@ class RuntimeTestContext(TestContext): dst_dir = os.path.join(packaged_path) # Extract package and copy it to TEST_EXTRACTED_DIR - if extract and not os.path.exists(dst_dir): - pkg_dir = self._extract_in_tmpdir(pkg) - shutil.copytree(pkg_dir, dst_dir) + pkg_dir = self._extract_in_tmpdir(pkg) + if extract: + + # Same package used for more than one test, + # don't need to extract again. + if os.path.exists(dst_dir): + continue + shutil.copytree(pkg_dir, dst_dir, copy_function=oeqa_copy) shutil.rmtree(pkg_dir) # Copy package to TEST_PACKAGED_DIR - elif not extract: + else: self._copy_package(pkg) def _getJsonFile(self, module):
Signature · 1 line
2.6.6
PATCH 3/3 oeqa/utils/sshcontrol.py: Allows to copy symlinks to target
In reply to [email protected]From: Mariano Lopez <[email protected]>
Currently when copying a symlink to the target it will fail throwing an exception. This will recreate symlinks from the system performing the tests to the device under tests.
[YOCTO #9932]
--- meta/lib/oeqa/utils/sshcontrol.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
meta/lib/oeqa/utils/sshcontrol.py +7 -2
@@ -147,8 +147,13 @@ class SSHControl(object): return self._internal_run(command, timeout, self.ignore_status) def copy_to(self, localpath, remotepath): - command = self.scp + [localpath, '%s@%s:%s' % (self.user, self.ip, remotepath)] - return self._internal_run(command, ignore_status=False) + if os.path.islink(localpath): + link = os.readlink(localpath) + dst_dir, dst_base = os.path.split(remotepath) + return self.run("cd %s; ln -s %s %s" % (dst_dir, link, dst_base)) + else: + command = self.scp + [localpath, '%s@%s:%s' % (self.user, self.ip, remotepath)] + return self._internal_run(command, ignore_status=False) def copy_from(self, remotepath, localpath): command = self.scp + ['%s@%s:%s' % (self.user, self.ip, remotepath), localpath]
Signature · 1 line
2.6.6
Quoted 10 lines · On Tue, 2016-07-26 at 07:39 +0000, [email protected] wrote:
From: Mariano Lopez <[email protected]>
Currently when using the install functionality in the DUT without package
manager, if you copy a symlink the function will fail.These series will change the behaviour and will allow to recreate the symlinks in the DUT.
[YOCTO #9932]
Would oe.path.copytree() work instead of using shutil.copytree()? copytree() was added because shutil was quite slow for our usage.
Regards,
Joshua
Quoted 7 lines · Joshua G Lock
Quoted 6 lines · On Tue, 2016-07-26 at 07:39 +0000, [email protected] wrote:
From: Mariano Lopez <[email protected]>
Currently when using the install functionality in the DUT without package
manager, if you copy a symlink the function will fail.
Quoted 7 lines
Quoted 4 lines
These series will change the behaviour and will allow to recreate the symlinks in the DUT.
[YOCTO #9932]
Would
oe.path.copytree()work instead of usingshutil.copytree()?copytree()was added because shutil was quite slow for our usage.
I didn't know the existence of that function, I tried it and it can replace shutil.copytree(). I'll send a v2.
Quoted 4 lines
Regards,
Joshua
Mariano Lopez