testing/infra: split runtime test from BRTest
PATCH v6 1/1 testing/infra: split runtime test from BRTest
From: Ricardo Martincoski <[email protected]>
Move the setup of emulator to a new class, RuntimeTestBase, that bahaves exactly like BRTest currently does.
It will avoid duplicating code when adding a common class to test the git download infra.
Change all current test cases to use the new class. Do this by first using automatic replace:
$ find support/testing/ -name '*.py' | \
xargs grep -l BRTest | \
xargs sed -i \
-e 's,import infra.basetest,\0\nimport infra.runtimetest,g' \
-e 's,infra.basetest.BRTest,infra.runtimetest.RuntimeTestBase,g'
and then manually add code to import runtimetest in test_external.py to
avoid this error:
AttributeError: 'module' object has no attribute 'LoadTestsFailure'
This explicit import was not need before because run-tests imports
BRTest and this is the only test file that do not use the defconfig
fragments from basetest.py in its code.---
WARNING: this patch changes all current test cases, so if a new test case was applied after this patch was sent, 'git am' will apply this patch cleanly, yet any testcase created in the meantime will be broken. It will probably only need the automatic replace from the commit log to be re-run.
Changes v5 -> v6:
- Fixed rebase which added extra args being passed to self.b.configure() for br2-external support
- Added additional test cases with class updates committed since v5
Changes v4 -> v5:
- no changes
Changes v3 -> v4:
- re-run the automatic replace command
Changes v2 -> v3:
- new patch
- search for "RuntimeTestBase" in
http://patchwork.ozlabs.org/patch/806161/ --- support/testing/infra/basetest.py | 10 ---------- support/testing/infra/runtimetest.py | 23 ++++++++++++++++++++++ support/testing/tests/boot/test_atf.py | 7 ++++--- .../testing/tests/core/test_file_capabilities.py | 3 ++- support/testing/tests/core/test_hardening.py | 3 ++- support/testing/tests/core/test_post_scripts.py | 3 ++- support/testing/tests/core/test_rootfs_overlay.py | 3 ++- support/testing/tests/core/test_timezone.py | 7 ++++--- support/testing/tests/fs/test_ext.py | 9 +++++---- support/testing/tests/fs/test_f2fs.py | 3 ++- support/testing/tests/fs/test_iso9660.py | 13 ++++++------ support/testing/tests/fs/test_jffs2.py | 3 ++- support/testing/tests/fs/test_squashfs.py | 3 ++- support/testing/tests/fs/test_ubi.py | 3 ++- support/testing/tests/fs/test_yaffs2.py | 3 ++- support/testing/tests/init/base.py | 3 ++- support/testing/tests/package/test_atop.py | 3 ++- support/testing/tests/package/test_dropbear.py | 3 ++- support/testing/tests/package/test_lua.py | 3 ++- support/testing/tests/package/test_perl.py | 3 ++- support/testing/tests/package/test_python.py | 3 ++- support/testing/tests/package/test_rust.py | 3 ++- support/testing/tests/package/test_syslog_ng.py | 3 ++- support/testing/tests/toolchain/test_external.py | 3 ++- 24 files changed, 79 insertions(+), 44 deletions(-) create mode 100644 support/testing/infra/runtimetest.py
support/testing/infra/basetest.py +0 -10
@@ -3,7 +3,6 @@ import os import datetime from infra.builder import Builder -from infra.emulator import Emulator BASIC_TOOLCHAIN_CONFIG = \ """
@@ -42,7 +41,6 @@ class BRTest(unittest.TestCase): super(BRTest, self).__init__(names) self.testname = self.__class__.__name__ self.builddir = self.outputdir and os.path.join(self.outputdir, self.testname) - self.emulator = None self.config += '\nBR2_DL_DIR="{}"\n'.format(self.downloaddir) self.config += "\nBR2_JLEVEL={}\n".format(self.jlevel)
@@ -58,17 +56,9 @@ class BRTest(unittest.TestCase): self.b.delete() if not self.b.is_finished(): - self.show_msg("Building") self.b.configure(make_extra_opts=["BR2_EXTERNAL={}".format(":".join(self.br2_external))]) - self.b.build() - self.show_msg("Building done") - - self.emulator = Emulator(self.builddir, self.downloaddir, - self.logtofile, self.timeout_multiplier) def tearDown(self): self.show_msg("Cleaning up") - if self.emulator: - self.emulator.stop() if self.b and not self.keepbuilds: self.b.delete()
support/testing/infra/runtimetest.py new file +23 -0
@@ -0,0 +1,23 @@ +from infra.basetest import BRTest +from infra.emulator import Emulator + + +class RuntimeTestBase(BRTest): + def __init__(self, names): + super(RuntimeTestBase, self).__init__(names) + self.emulator = None + + def setUp(self): + super(RuntimeTestBase, self).setUp() + if not self.b.is_finished(): + self.show_msg("Building") + self.b.build() + self.show_msg("Building done") + + self.emulator = Emulator(self.builddir, self.downloaddir, + self.logtofile, self.timeout_multiplier) + + def tearDown(self): + if self.emulator: + self.emulator.stop() + super(RuntimeTestBase, self).tearDown()
support/testing/tests/boot/test_atf.py +4 -3
@@ -1,7 +1,8 @@ import infra.basetest +import infra.runtimetest -class TestATFVexpress(infra.basetest.BRTest): +class TestATFVexpress(infra.runtimetest.RuntimeTestBase): config = \ """ BR2_aarch64=y
@@ -25,7 +26,7 @@ class TestATFVexpress(infra.basetest.BRTest): pass -class TestATFAllwinner(infra.basetest.BRTest): +class TestATFAllwinner(infra.runtimetest.RuntimeTestBase): config = \ """ BR2_aarch64=y
@@ -54,7 +55,7 @@ class TestATFAllwinner(infra.basetest.BRTest): pass -class TestATFMarvell(infra.basetest.BRTest): +class TestATFMarvell(infra.runtimetest.RuntimeTestBase): config = \ """ BR2_aarch64=y
support/testing/tests/core/test_file_capabilities.py +2 -1
@@ -2,9 +2,10 @@ import os import subprocess import infra.basetest +import infra.runtimetest -class TestFileCapabilities(infra.basetest.BRTest): +class TestFileCapabilities(infra.runtimetest.RuntimeTestBase): config = \ """ BR2_arm=y
support/testing/tests/core/test_hardening.py +2 -1
@@ -3,9 +3,10 @@ import subprocess import json import infra.basetest +import infra.runtimetest -class TestHardeningBase(infra.basetest.BRTest): +class TestHardeningBase(infra.runtimetest.RuntimeTestBase): config = \ """ BR2_powerpc64=y
support/testing/tests/core/test_post_scripts.py +2 -1
@@ -2,9 +2,10 @@ import os import csv import infra.basetest +import infra.runtimetest -class TestPostScripts(infra.basetest.BRTest): +class TestPostScripts(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_INIT_NONE=y
support/testing/tests/core/test_rootfs_overlay.py +2 -1
@@ -2,13 +2,14 @@ import os import subprocess import infra.basetest +import infra.runtimetest def compare_file(file1, file2): return subprocess.call(["cmp", file1, file2]) -class TestRootfsOverlay(infra.basetest.BRTest): +class TestRootfsOverlay(infra.runtimetest.RuntimeTestBase): rootfs_overlay_path = infra.filepath("tests/core/rootfs-overlay")
support/testing/tests/core/test_timezone.py +4 -3
@@ -1,6 +1,7 @@ import os import infra.basetest +import infra.runtimetest def boot_armv5_cpio(emulator, builddir):
@@ -10,7 +11,7 @@ def boot_armv5_cpio(emulator, builddir): emulator.login() -class TestNoTimezone(infra.basetest.BRTest): +class TestNoTimezone(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ # BR2_TARGET_TZ_INFO is not set
@@ -26,7 +27,7 @@ class TestNoTimezone(infra.basetest.BRTest): self.assertEqual(tz[0].strip(), "UTC") -class TestGlibcAllTimezone(infra.basetest.BRTest): +class TestGlibcAllTimezone(infra.runtimetest.RuntimeTestBase): config = \ """ BR2_arm=y
@@ -48,7 +49,7 @@ class TestGlibcAllTimezone(infra.basetest.BRTest): self.assertEqual(tz[0].strip(), "CET") -class TestGlibcNonDefaultLimitedTimezone(infra.basetest.BRTest): +class TestGlibcNonDefaultLimitedTimezone(infra.runtimetest.RuntimeTestBase): config = \ """ BR2_arm=y
support/testing/tests/fs/test_ext.py +5 -4
@@ -2,6 +2,7 @@ import os import subprocess import infra.basetest +import infra.runtimetest VOLNAME_PROP = "Filesystem volume name" REVISION_PROP = "Filesystem revision #"
@@ -41,7 +42,7 @@ def boot_img_and_check_fs_type(emulator, builddir, fs_type): return exit_code -class TestExt2(infra.basetest.BRTest): +class TestExt2(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_EXT2=y
@@ -60,7 +61,7 @@ class TestExt2(infra.basetest.BRTest): self.assertEqual(exit_code, 0) -class TestExt2r1(infra.basetest.BRTest): +class TestExt2r1(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_EXT2=y
@@ -80,7 +81,7 @@ class TestExt2r1(infra.basetest.BRTest): self.assertEqual(exit_code, 0) -class TestExt3(infra.basetest.BRTest): +class TestExt3(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_EXT2=y
@@ -99,7 +100,7 @@ class TestExt3(infra.basetest.BRTest): self.assertEqual(exit_code, 0) -class TestExt4(infra.basetest.BRTest): +class TestExt4(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_EXT2=y
support/testing/tests/fs/test_f2fs.py +2 -1
@@ -2,6 +2,7 @@ import os import subprocess import infra.basetest +import infra.runtimetest def dumpf2fs_getprop(out, prop):
@@ -11,7 +12,7 @@ def dumpf2fs_getprop(out, prop): return fields[1].strip() -class TestF2FS(infra.basetest.BRTest): +class TestF2FS(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_F2FS=y
support/testing/tests/fs/test_iso9660.py +7 -6
@@ -1,6 +1,7 @@ import os import infra.basetest +import infra.runtimetest BASIC_CONFIG = \ """
@@ -47,7 +48,7 @@ def test_touch_file(emulator): # Grub 2 -class TestIso9660Grub2External(infra.basetest.BRTest): +class TestIso9660Grub2External(infra.runtimetest.RuntimeTestBase): config = BASIC_CONFIG + \ """ BR2_TARGET_ROOTFS_ISO9660=y
@@ -67,7 +68,7 @@ class TestIso9660Grub2External(infra.basetest.BRTest): self.assertEqual(exit_code, 1) -class TestIso9660Grub2ExternalCompress(infra.basetest.BRTest): +class TestIso9660Grub2ExternalCompress(infra.runtimetest.RuntimeTestBase): config = BASIC_CONFIG + \ """ BR2_TARGET_ROOTFS_ISO9660=y
@@ -88,7 +89,7 @@ class TestIso9660Grub2ExternalCompress(infra.basetest.BRTest): self.assertEqual(exit_code, 1) -class TestIso9660Grub2Internal(infra.basetest.BRTest): +class TestIso9660Grub2Internal(infra.runtimetest.RuntimeTestBase): config = BASIC_CONFIG + \ """ BR2_TARGET_ROOTFS_ISO9660=y
@@ -111,7 +112,7 @@ class TestIso9660Grub2Internal(infra.basetest.BRTest): # Syslinux -class TestIso9660SyslinuxExternal(infra.basetest.BRTest): +class TestIso9660SyslinuxExternal(infra.runtimetest.RuntimeTestBase): config = BASIC_CONFIG + \ """ BR2_TARGET_ROOTFS_ISO9660=y
@@ -130,7 +131,7 @@ class TestIso9660SyslinuxExternal(infra.basetest.BRTest): self.assertEqual(exit_code, 1) -class TestIso9660SyslinuxExternalCompress(infra.basetest.BRTest): +class TestIso9660SyslinuxExternalCompress(infra.runtimetest.RuntimeTestBase): config = BASIC_CONFIG + \ """ BR2_TARGET_ROOTFS_ISO9660=y
@@ -150,7 +151,7 @@ class TestIso9660SyslinuxExternalCompress(infra.basetest.BRTest): self.assertEqual(exit_code, 1) -class TestIso9660SyslinuxInternal(infra.basetest.BRTest): +class TestIso9660SyslinuxInternal(infra.runtimetest.RuntimeTestBase): config = BASIC_CONFIG + \ """ BR2_TARGET_ROOTFS_ISO9660=y
support/testing/tests/fs/test_jffs2.py +2 -1
@@ -2,6 +2,7 @@ import os import subprocess import infra.basetest +import infra.runtimetest def jffs2dump_find_file(files_list, fname):
@@ -12,7 +13,7 @@ def jffs2dump_find_file(files_list, fname): return False -class TestJffs2(infra.basetest.BRTest): +class TestJffs2(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_JFFS2=y
support/testing/tests/fs/test_squashfs.py +2 -1
@@ -2,9 +2,10 @@ import os import subprocess import infra.basetest +import infra.runtimetest -class TestSquashfs(infra.basetest.BRTest): +class TestSquashfs(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_SQUASHFS=y
support/testing/tests/fs/test_ubi.py +2 -1
@@ -2,9 +2,10 @@ import subprocess import os import infra.basetest +import infra.runtimetest -class TestUbi(infra.basetest.BRTest): +class TestUbi(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_UBIFS=y
support/testing/tests/fs/test_yaffs2.py +2 -1
@@ -1,9 +1,10 @@ import os import infra.basetest +import infra.runtimetest -class TestYaffs2(infra.basetest.BRTest): +class TestYaffs2(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ infra.basetest.MINIMAL_CONFIG + \ """
support/testing/tests/init/base.py +2 -1
@@ -1,9 +1,10 @@ import os import subprocess import infra.basetest +import infra.runtimetest -class InitSystemBase(infra.basetest.BRTest): +class InitSystemBase(infra.runtimetest.RuntimeTestBase): def start_emulator(self, fs_type, kernel=None, dtb=None, init=None): img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
support/testing/tests/package/test_atop.py +2 -1
@@ -1,9 +1,10 @@ import os import infra.basetest +import infra.runtimetest -class TestAtop(infra.basetest.BRTest): +class TestAtop(infra.runtimetest.RuntimeTestBase): config = \ """ BR2_arm=y
support/testing/tests/package/test_dropbear.py +2 -1
@@ -1,9 +1,10 @@ import os import infra.basetest +import infra.runtimetest -class TestDropbear(infra.basetest.BRTest): +class TestDropbear(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_GENERIC_ROOT_PASSWD="testpwd"
support/testing/tests/package/test_lua.py +2 -1
@@ -1,9 +1,10 @@ import os import infra.basetest +import infra.runtimetest -class TestLuaBase(infra.basetest.BRTest): +class TestLuaBase(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_CPIO=y
support/testing/tests/package/test_perl.py +2 -1
@@ -1,9 +1,10 @@ import os import infra.basetest +import infra.runtimetest -class TestPerlBase(infra.basetest.BRTest): +class TestPerlBase(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_CPIO=y
support/testing/tests/package/test_python.py +2 -1
@@ -1,9 +1,10 @@ import os import infra.basetest +import infra.runtimetest -class TestPythonBase(infra.basetest.BRTest): +class TestPythonBase(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_TARGET_ROOTFS_CPIO=y
support/testing/tests/package/test_rust.py +2 -1
@@ -4,9 +4,10 @@ import subprocess import shutil import infra.basetest +import infra.runtimetest -class TestRustBase(infra.basetest.BRTest): +class TestRustBase(infra.runtimetest.RuntimeTestBase): target = 'armv7-unknown-linux-gnueabihf' crate = 'hello-world'
support/testing/tests/package/test_syslog_ng.py +2 -1
@@ -1,9 +1,10 @@ import os import infra.basetest +import infra.runtimetest -class TestSyslogNg(infra.basetest.BRTest): +class TestSyslogNg(infra.runtimetest.RuntimeTestBase): config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ """ BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
support/testing/tests/toolchain/test_external.py +2 -1
@@ -1,5 +1,6 @@ import os import infra +import infra.runtimetest BASIC_CONFIG = \ """
@@ -17,7 +18,7 @@ def has_broken_links(path): return False -class TestExternalToolchain(infra.basetest.BRTest): +class TestExternalToolchain(infra.runtimetest.RuntimeTestBase): def common_check(self): # Check for broken symlinks for d in ["lib", "usr/lib"]:
Signature · 1 line
1.9.1
PATCH v6 1/1 testing/infra: split runtime test from BRTest
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
From: Ricardo Martincoski <[email protected]> Move the setup of emulator to a new class, RuntimeTestBase, that bahaves
s/bahaves/behaves/
Quoted 3 lines
exactly like BRTest currently does.
It will avoid duplicating code when adding a common class to test the git download infra.
Sorry for the late feedback, but wouldn't it be a lot simpler/more pragmatic to keep the BRTest name for the "common" case, and instead add a new parent class with all the common stuff.
E.G.:
BaseTest <-- GitTest <- ..
|
- BRTest <-- TestATop
|
- TestDropbear
|
- ...
I agree that RuntimeTestBase is maybe a nicer name, but BRTest for something where we run a Buildroot output under qemu isn't really bad either IMHO.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v6 1/1 testing/infra: split runtime test from BRTest
Quoted 25 lines · Peter Korsgaard
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
From: Ricardo Martincoski <[email protected]> Move the setup of emulator to a new class, RuntimeTestBase, that bahaves
s/bahaves/behaves/
Quoted 3 lines
exactly like BRTest currently does.
It will avoid duplicating code when adding a common class to test the git download infra.Sorry for the late feedback, but wouldn't it be a lot simpler/more pragmatic to keep the BRTest name for the "common" case, and instead add a new parent class with all the common stuff.
E.G.:
BaseTest <-- GitTest <- ..
|
- BRTest <-- TestATop
|
- TestDropbear
|
- ...
Quoted 3 lines
I agree that RuntimeTestBase is maybe a nicer name, but BRTest for something where we run a Buildroot output under qemu isn't really bad either IMHO.
I thought of that as well, but then I thought that it would be confusing to add
a superclass. But I hadn't figured out that BRTest would indeed refer to tests doing a full Buildroot build.
BaseTest is not that great, though. BRConfigTest maybe? That more accurately
reflects what it does, I think.
Regards,
Arnout
PATCH v6 1/1 testing/infra: split runtime test from BRTest
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi,
Quoted 10 lines
Quoted 3 lines
I agree that RuntimeTestBase is maybe a nicer name, but BRTest for something where we run a Buildroot output under qemu isn't really bad either IMHO.
I thought of that as well, but then I thought that it would be confusing to add
a superclass. But I hadn't figured out that BRTest would indeed refer to tests doing a full Buildroot build.BaseTest is not that great, though. BRConfigTest maybe? That more accurately
reflects what it does, I think.
Sure, that is fine by me. I would just prefer to leave BRTest to keep the diff minimal.
Signature · 1 line
Bye, Peter Korsgaard