Add oeqa test cases

Adds the testing framework for tying tests for the MinGW SDK into the oeqa SDK test framework.

The SDK tests can be run using the standard SDK test command:

bitbake -c testsdk <image>

All testing is done using Wine in lieu of having access to an actual Windows machine. Wine is required to be installed on the host machine.

Joshua Watt (4):
Add SDK test case framework
Add .gitignore classes/toolchain-scripts-mingw32: Update variables oeqa/sdkmingw: Add test cases

 .gitignore                                  |   3 +
 classes/testsdk-mingw.bbclass               | 101 ++++++++++++++++++++
 classes/toolchain-scripts-mingw32.bbclass   |   4 +-
 conf/machine-sdk/i686-mingw32.conf          |   1 +
 conf/machine-sdk/include/mingw32-common.inc |   6 ++
 conf/machine-sdk/x86_64-mingw32.conf        |   1 +
 lib/oeqa/sdkmingw/__init__.py               |   0
 lib/oeqa/sdkmingw/case.py                   |  52 ++++++++++
 lib/oeqa/sdkmingw/cases/__init__.py         |   0
 lib/oeqa/sdkmingw/cases/binutils.py         |  57 +++++++++++
 lib/oeqa/sdkmingw/cases/gcc.py              |  49 ++++++++++
 lib/oeqa/sdkmingw/cases/gdb.py              |  20 ++++
 lib/oeqa/sdkmingw/context.py                |  66 +++++++++++++
 lib/oeqa/sdkmingw/files/testsdkmakefile     |   4 +
 14 files changed, 363 insertions(+), 1 deletion(-)
 create mode 100644 .gitignore
 create mode 100644 classes/testsdk-mingw.bbclass
 create mode 100644 lib/oeqa/sdkmingw/__init__.py
 create mode 100644 lib/oeqa/sdkmingw/case.py
 create mode 100644 lib/oeqa/sdkmingw/cases/__init__.py
 create mode 100644 lib/oeqa/sdkmingw/cases/binutils.py
 create mode 100644 lib/oeqa/sdkmingw/cases/gcc.py
 create mode 100644 lib/oeqa/sdkmingw/cases/gdb.py
 create mode 100644 lib/oeqa/sdkmingw/context.py
 create mode 100644 lib/oeqa/sdkmingw/files/testsdkmakefile

2.19.1

I see the cover letter but no patches or link to a branch...

Ross

Ya, I was a dummy and forgot to say send all in git send-email, then it timed out. I have to make other changes anyway because of feedback from Richard.

Adds the testing framework for tying tests for the MinGW SDK into the oeqa SDK test framework.

The SDK tests can be run using the standard SDK test command:

bitbake -c testsdk <image>

All testing is done using Wine in lieu of having access to an actual Windows machine. Wine is required to be installed on the host machine.

Joshua Watt (5):
Add SDK test case framework
Add .gitignore
classes/toolchain-scripts-mingw32: Update variables classes/toolchain-scripts-mingw32: Ensure exit code is success oeqa/sdkmingw: Add test cases

 .gitignore                                  |  3 +
 classes/toolchain-scripts-mingw32.bbclass   | 15 +++-
 conf/machine-sdk/i686-mingw32.conf          |  1 +
 conf/machine-sdk/include/mingw32-common.inc |  7 ++
 conf/machine-sdk/x86_64-mingw32.conf        |  1 +
 lib/oeqa/sdkmingw/__init__.py               |  0
 lib/oeqa/sdkmingw/case.py                   | 87 +++++++++++++++++++++
 lib/oeqa/sdkmingw/cases/__init__.py         |  0
 lib/oeqa/sdkmingw/cases/binutils.py         | 55 +++++++++++++
 lib/oeqa/sdkmingw/cases/gcc.py              | 47 +++++++++++
 lib/oeqa/sdkmingw/cases/gdb.py              | 20 +++++
 lib/oeqa/sdkmingw/cases/pkgconfig.py        | 18 +++++
 lib/oeqa/sdkmingw/context.py                | 69 ++++++++++++++++
 lib/oeqa/sdkmingw/files/testsdkmakefile     |  4 +
 lib/oeqa/sdkmingw/testsdk.py                | 42 ++++++++++
 15 files changed, 368 insertions(+), 1 deletion(-)
 create mode 100644 .gitignore
 create mode 100644 lib/oeqa/sdkmingw/__init__.py
 create mode 100644 lib/oeqa/sdkmingw/case.py
 create mode 100644 lib/oeqa/sdkmingw/cases/__init__.py
 create mode 100644 lib/oeqa/sdkmingw/cases/binutils.py
 create mode 100644 lib/oeqa/sdkmingw/cases/gcc.py
 create mode 100644 lib/oeqa/sdkmingw/cases/gdb.py
 create mode 100644 lib/oeqa/sdkmingw/cases/pkgconfig.py
 create mode 100644 lib/oeqa/sdkmingw/context.py
 create mode 100644 lib/oeqa/sdkmingw/files/testsdkmakefile
 create mode 100644 lib/oeqa/sdkmingw/testsdk.py

2.19.1

Adds the framework for testing SDKs that ties into the oeqa test framework. This allows commands like:

 $ bitbake -c testsdk ...

to be run for MinGW SDKs.

The test framework currently executes all tests under Wine in lieu of having access to actual Windows machines.

[YOCTO #13020]

---
 conf/machine-sdk/i686-mingw32.conf          |  1 +
 conf/machine-sdk/include/mingw32-common.inc |  7 ++
 conf/machine-sdk/x86_64-mingw32.conf        |  1 +
 lib/oeqa/sdkmingw/__init__.py               |  0
 lib/oeqa/sdkmingw/case.py                   | 87 +++++++++++++++++++++
 lib/oeqa/sdkmingw/cases/__init__.py         |  0
 lib/oeqa/sdkmingw/context.py                | 69 ++++++++++++++++
 lib/oeqa/sdkmingw/testsdk.py                | 42 ++++++++++
 8 files changed, 207 insertions(+)
 create mode 100644 lib/oeqa/sdkmingw/__init__.py
 create mode 100644 lib/oeqa/sdkmingw/case.py
 create mode 100644 lib/oeqa/sdkmingw/cases/__init__.py
 create mode 100644 lib/oeqa/sdkmingw/context.py
 create mode 100644 lib/oeqa/sdkmingw/testsdk.py

index 5090168..fef48b5 100644
@@ -1,3 +1,4 @@
 SDK_ARCH = "i686"
+TESTSDK_WINEARCH = "win32"
 
 require conf/machine-sdk/include/mingw32-common.inc

index 733d092..71e8d45 100644
@@ -26,6 +26,9 @@ SDKPKGSUFFIX = "nativesdk-mingw32"
 
 MACHINEOVERRIDES .= ":sdkmingw32"
 
+TESTSDK_CLASS_NAME = "oeqa.sdkmingw.testsdk.TestSDKMinGW"
+TESTSDKEXT_CLASS_NAME = ""
+
 WINDRES_mingw32 = "${HOST_PREFIX}windres --include-dir=${STAGING_INCDIR}"
 RC_mingw32 = "${WINDRES}"
 

@@ -39,3 +42,7 @@ DISABLE_STATIC_mingw32 = ""
 
 # disable security flags
 GCCPIE_mingw32 = ""
+
+# wine and wineserver are required to test MinGW SDKs
+HOSTTOOLS += "${@'wine wineserver' if (bb.utils.contains_any('IMAGE_CLASSES', 'testsdk-mingw', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testsdk-mingw.bbclass"])) else ''}"
+

index fc53822..188debc 100644
@@ -1,3 +1,4 @@
 SDK_ARCH = "x86_64"
+TESTSDK_WINEARCH = "win64"
 
 require conf/machine-sdk/include/mingw32-common.inc

new file mode 100644
index 0000000..e69de29

new file mode 100644
index 0000000..e69de29

2.19.1

Adds a .gitignore which ignores python artifact files, as these are now generated by the SDK tests

[YOCTO #13020]

---
 .gitignore | 3 +++
 1 file changed, 3 insertions(+)
 create mode 100644 .gitignore

new file mode 100644
index 0000000..5da7ef5
@@ -0,0 +1,3 @@
+*.pyc
+*.pyo
+

2.19.1

Updates the variables configured by the toolchain script to match the oe-core class. This combines the changes from:

0b5f1b0430 ("toolchain-scripts: put TARGET_ARCH and TARGET_OS into environment") 47160a8c51 ("cmake: fix linking in multilib SDKs")
668f56bb24 ("toolchain-scripts: drop PYTHONHOME")
81f126f4a8 ("toolchain-scripts.bbclass: Use PYTHONPATH instead of PYTHONHOME")

[YOCTO #13020]

---
 classes/toolchain-scripts-mingw32.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index b996a3e..a64f5f5 100644
@@ -29,7 +29,9 @@ toolchain_create_sdk_env_script_sdkmingw32 () {
 	echo "set OECORE_NATIVE_SYSROOT=$sdkpathnative" >> $script
 	echo 'set OECORE_TARGET_SYSROOT=%SDKTARGETSYSROOT%' >> $script
 	echo "set OECORE_ACLOCAL_OPTS=-I $sdkpathnative/usr/share/aclocal" >> $script
-	echo "set PYTHONHOME=$sdkpathnative$prefix" >> $script
+	echo 'set OECORE_BASELIB=${baselib}' >> $script
+	echo 'set OECORE_TARGET_ARCH=${TARGET_ARCH}' >> $script
+	echo 'set OECORE_TARGET_OS=${TARGET_OS}' >> $script
 
 	toolchain_shared_env_script
 

2.19.1

The "set" command in Windows has some unfortunate behaviors that can frequently make it look like the toolchain script had an error. This makes automated testing of the SDK more difficult, so add a line to the environment setup script that ensures the script exits with success.

[YOCTO #13020]

---
 classes/toolchain-scripts-mingw32.bbclass | 11 +++++++++++
 1 file changed, 11 insertions(+)

index a64f5f5..32493d7 100644
@@ -81,6 +81,17 @@ toolchain_shared_env_script_sdkmingw32 () {
    FOR %%x IN (%OECORE_NATIVE_SYSROOT%\\environment-setup.d\\*.bat) DO call "%%x"
 )
 EOF
+
+	# set has some annoying properties:
+	# 1) If it is successful %ERRORLEVEL% is unchanged (as opposed to being set
+	#	 to 0 to indicate success)
+	# 2) Making an assignment like "set A=" is considered an error and sets
+	#	 %ERRORLEVEL% to 1.
+	#
+	# Practically, this means that if any of the set calls make an empty
+	# assignment that error will be propagated. To prevent this, a command is
+	# run to ensure that the "exit code" of this script is 0
+	echo "@%COMSPEC% /C exit 0 > NUL" >> $script
 }
 
 toolchain_create_sdk_siteconfig_append_sdkmingw32 () {

2.19.1

Adds test cases to verify that gcc, binutils, gdb, and pkg-config are functioning in the MinGW SDK

[YOCTO #13020]

---
 lib/oeqa/sdkmingw/cases/binutils.py     | 55 +++++++++++++++++++++++++
 lib/oeqa/sdkmingw/cases/gcc.py          | 47 +++++++++++++++++++++
 lib/oeqa/sdkmingw/cases/gdb.py          | 20 +++++++++
 lib/oeqa/sdkmingw/cases/pkgconfig.py    | 18 ++++++++
 lib/oeqa/sdkmingw/files/testsdkmakefile |  4 ++
 5 files changed, 144 insertions(+)
 create mode 100644 lib/oeqa/sdkmingw/cases/binutils.py
 create mode 100644 lib/oeqa/sdkmingw/cases/gcc.py
 create mode 100644 lib/oeqa/sdkmingw/cases/gdb.py
 create mode 100644 lib/oeqa/sdkmingw/cases/pkgconfig.py
 create mode 100644 lib/oeqa/sdkmingw/files/testsdkmakefile

new file mode 100644
index 0000000..fdd5a47
@@ -0,0 +1,55 @@
+# Copyright 2018 by Garmin Ltd. or its subsidiaries
+# Released under the MIT license (see COPYING.MIT)
+
+import os
+import unittest
+
+from oeqa.sdkmingw.case import OESDKMinGWTestCase
+
+class BinutilsTest(OESDKMinGWTestCase):
+    td_vars = ['MACHINE']
+
+    def setUp(self):
+        super().setUp()
+
+        self.copyTestFile(os.path.join(self.tc.files_dir, 'test.c'))
+
+        machine = self.td.get("MACHINE")
+        if not (self.tc.hasHostPackage("packagegroup-cross-canadian-%s" % machine) or
+                self.tc.hasHostPackage("^gcc-", regex=True)):
+            raise unittest.SkipTest(self.__class__.__name__ + " class: SDK doesn't contain a cross-canadian toolchain")
+        if not (self.tc.hasHostPackage("packagegroup-cross-canadian-%s" % machine) or
+                self.tc.hasHostPackage('binutils-cross-canadian-%s' % machine)):
+            raise unittest.SkipTest(self.__class__.__name__ + " class: SDK doesn't contain a binutils")
+
+        self._run('%CC% -c -g test.c -o test.o')
+        self._run('%CC% -o test test.o -lm')
+
+    def test_strip(self):
+        self._run('%STRIP% -s test')
+        self.assertIsTargetElf(os.path.join(self.test_dir, 'test'))
+
+    def test_ar(self):
+        self._run('%AR% -rcs lib.a test.o')
+        self._run('%CC% -o test lib.a -lm')
+        self.assertIsTargetElf(os.path.join(self.test_dir, 'test'))
+
+    def test_ranlib(self):
+        self._run('%AR% -rc lib.a test.o')
+        self._run('%RANLIB% lib.a')
+        self._run('%CC% -o test lib.a -lm')
+        self.assertIsTargetElf(os.path.join(self.test_dir, 'test'))
+
+    def test_objcopy(self):
+        self._run('%OBJCOPY% -g test.o test_no_debug.o')
+        self.assertIsTargetElf(os.path.join(self.test_dir, 'test_no_debug.o'))
+        self._run('%CC% -o test test_no_debug.o -lm')
+        self.assertIsTargetElf(os.path.join(self.test_dir, 'test'))
+
+    def test_objdump(self):
+        self._run('%OBJDUMP% -S test.o')
+
+    def test_nm(self):
+        self._run('%NM% test.o')
+
+

new file mode 100644
index 0000000..f1bde7a
@@ -0,0 +1,20 @@
+# Copyright 2018 by Garmin Ltd. or its subsidiaries
+# Released under the MIT license (see COPYING.MIT)
+
+import unittest
+
+from oeqa.sdkmingw.case import OESDKMinGWTestCase
+
+class GdbTest(OESDKMinGWTestCase):
+    td_vars = ['MACHINE']
+
+    def setUp(self):
+        super().setUp()
+
+        machine = self.td.get("MACHINE")
+        if not (self.tc.hasHostPackage("packagegroup-cross-canadian-%s" % machine) or
+                self.tc.hasHostPackage("^gdb-", regex=True)):
+            raise unittest.SkipTest(self.__class__.__name__ + " class: SDK doesn't contain a cross-canadian toolchain")
+
+    def test_gdb(self):
+        self._run('%GDB% -ex quit')

new file mode 100644
index 0000000..24185a0
@@ -0,0 +1,18 @@
+# Copyright 2018 by Garmin Ltd. or its subsidiaries
+# Released under the MIT license (see COPYING.MIT)
+
+import os
+import unittest
+
+from oeqa.sdkmingw.case import OESDKMinGWTestCase
+
+class PkgConfigTest(OESDKMinGWTestCase):
+    def setUp(self):
+        super().setUp()
+
+        if not self.tc.hasHostPackage("nativesdk-pkgconfig"):
+            raise unittest.SkipTest(self.__class__.__name__ + " class: SDK doesn't contain nativesdk-pkgconfig")
+
+    def test_pkg_config(self):
+        self._run('pkg-config --list-all')
+

new file mode 100644
index 0000000..b5cac86
@@ -0,0 +1,4 @@
+test: test.o
+	$(CC) -o test test.o -lm
+test.o: test.c
+	$(CC) -c test.c

2.19.1