0/8] Toaster improve test suite
RFC PATCH 1/1 0/8] Toaster improve test suite
This set of changes refactor/improve the current test suite now it can be parameterized from command line.
The main goal for this set of patches are for integrate the toaster test suite into AB. The AB part is already on master. [1] If this patchset is merged the way for run Toaster test suite will change for use it read README in this patchset.
Futher work are related to improve/fix the current test suite also i added a TODO list.
Aníbal Limón (8):
toaster/tests: Reorder/move current test code. toaster/tests: Add __init__.py and base.py. toaster/tests: Improve ui.py module. toaster/tests: Adds setup test case. toaster/tests: Add helpers.py module.
toaster-test: Refactor and imporvements. toaster/tests: Add README, TODO and requeriments. toaster/tests: Adds Copyright information.
bin/toaster-test | 238 ++ .../contrib/tts/toasteruitest/run_toastertests.py | 155 -- .../tts/toasteruitest/toaster_automation_test.py | 2376 ------------------- .../contrib/tts/toasteruitest/toaster_test.cfg | 25 - lib/toaster/tests/README | 64 + lib/toaster/tests/TODO | 13 + lib/toaster/tests/__init__.py | 0 lib/toaster/tests/base.py | 28 + lib/toaster/tests/helpers.py | 192 ++ lib/toaster/tests/setup.py | 94 + lib/toaster/tests/toaster_test.cfg | 25 + lib/toaster/tests/ui.py | 2379 ++++++++++++++++++++ toaster-tests-requirements.txt | 5 + 13 files changed, 3038 insertions(+), 2556 deletions(-) create mode 100755 bin/toaster-test delete mode 100755 lib/toaster/contrib/tts/toasteruitest/run_toastertests.py delete mode 100755 lib/toaster/contrib/tts/toasteruitest/toaster_automation_test.py delete mode 100644 lib/toaster/contrib/tts/toasteruitest/toaster_test.cfg create mode 100644 lib/toaster/tests/README create mode 100644 lib/toaster/tests/TODO create mode 100644 lib/toaster/tests/__init__.py create mode 100644 lib/toaster/tests/base.py create mode 100755 lib/toaster/tests/helpers.py create mode 100644 lib/toaster/tests/setup.py create mode 100644 lib/toaster/tests/toaster_test.cfg create mode 100644 lib/toaster/tests/ui.py create mode 100644 toaster-tests-requirements.txt
Signature · 1 line
2.1.4
RFC PATCH 1/1 1/8] toaster/tests: Reorder/move current test code.
In order to provide a better structure of the toaster test suite:
bin/toaster-test: Main script for run toaster tests. lib/toaster/tests: Contains ui test code and configuration.
--- bin/toaster-test | 155 ++ .../contrib/tts/toasteruitest/run_toastertests.py | 155 -- .../tts/toasteruitest/toaster_automation_test.py | 2376 -------------------- .../contrib/tts/toasteruitest/toaster_test.cfg | 25 - lib/toaster/tests/toaster_test.cfg | 25 + lib/toaster/tests/ui.py | 2376 ++++++++++++++++++++ 6 files changed, 2556 insertions(+), 2556 deletions(-) create mode 100755 bin/toaster-test delete mode 100755 lib/toaster/contrib/tts/toasteruitest/run_toastertests.py delete mode 100755 lib/toaster/contrib/tts/toasteruitest/toaster_automation_test.py delete mode 100644 lib/toaster/contrib/tts/toasteruitest/toaster_test.cfg create mode 100644 lib/toaster/tests/toaster_test.cfg create mode 100644 lib/toaster/tests/ui.py
bin/toaster-test new file +155 -0
@@ -0,0 +1,155 @@ +#!/usr/bin/python + +# Copyright + +# DESCRIPTION +# This is script for running all selected toaster cases on +# selected web browsers manifested in toaster_test.cfg. + +# 1. How to start toaster in yocto: +# $ source poky/oe-init-build-env +# $ source toaster start +# $ bitbake core-image-minimal + +# 2. How to install selenium on Ubuntu: +# $ sudo apt-get install scrot python-pip +# $ sudo pip install selenium + +# 3. How to install selenium addon in firefox: +# Download the lastest firefox addon from http://release.seleniumhq.org/selenium-ide/ +# Then install it. You can also install firebug and firepath addon + +# 4. How to start writing a new case: +# All you need to do is to implement the function test_xxx() and pile it on. + +# 5. How to test with Chrome browser +# Download/install chrome on host +# Download chromedriver from https://code.google.com/p/chromedriver/downloads/list according to your host type +# put chromedriver in PATH, (e.g. /usr/bin/, bear in mind to chmod) +# For windows host, you may put chromedriver.exe in the same directory as chrome.exe + +import unittest, sys, os, platform +import ConfigParser +import argparse +from toaster_automation_test import toaster_cases + + +def get_args_parser(): + description = "Script that runs toaster auto tests." + parser = argparse.ArgumentParser(description=description) + parser.add_argument('--run-all-tests', required=False, action="store_true", dest="run_all_tests", default=False, + help='Run all tests.') + parser.add_argument('--run-suite', required=False, dest='run_suite', default=False, + help='run suite (defined in cfg file)') + + return parser + + +def get_tests(): + testslist = [] + + prefix = 'toaster_automation_test.toaster_cases' + + for t in dir(toaster_cases): + if t.startswith('test_'): + testslist.append('.'.join((prefix, t))) + + return testslist + + +def get_tests_from_cfg(suite=None): + + testslist = [] + config = ConfigParser.SafeConfigParser() + config.read('toaster_test.cfg') + + if suite is not None: + target_suite = suite.lower() + + # TODO: if suite is valid suite + + else: + target_suite = platform.system().lower() + + try: + tests_from_cfg = eval(config.get('toaster_test_' + target_suite, 'test_cases')) + except: + print 'Failed to get test cases from cfg file. Make sure the format is correct.' + return None + + prefix = 'toaster_automation_test.toaster_cases.test_' + for t in tests_from_cfg: + testslist.append(prefix + str(t)) + + return testslist + +def main(): + + # In case this script is called from other directory + os.chdir(os.path.abspath(sys.path[0])) + + parser = get_args_parser() + args = parser.parse_args() + + if args.run_all_tests: + testslist = get_tests() + elif args.run_suite: + testslist = get_tests_from_cfg(args.run_suite) + os.environ['TOASTER_SUITE'] = args.run_suite + else: + testslist = get_tests_from_cfg() + + if not testslist: + print 'Failed to get test cases.' + exit(1) + + suite = unittest.TestSuite() + loader = unittest.TestLoader() + loader.sortTestMethodsUsing = None + runner = unittest.TextTestRunner(verbosity=2, resultclass=buildResultClass(args)) + + for test in testslist: + try: + suite.addTests(loader.loadTestsFromName(test)) + except: + return 1 + + result = runner.run(suite) + + if result.wasSuccessful(): + return 0 + else: + return 1 + + +def buildResultClass(args): + """Build a Result Class to use in the testcase execution""" + + class StampedResult(unittest.TextTestResult): + """ + Custom TestResult that prints the time when a test starts. As toaster-auto + can take a long time (ie a few hours) to run, timestamps help us understand + what tests are taking a long time to execute. + """ + def startTest(self, test): + import time + self.stream.write(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) + " - ") + super(StampedResult, self).startTest(test) + + return StampedResult + + +if __name__ == "__main__": + + try: + ret = main() + except: + ret = 1 + import traceback + traceback.print_exc(5) + finally: + if os.getenv('TOASTER_SUITE'): + del os.environ['TOASTER_SUITE'] + sys.exit(ret) + +
lib/toaster/contrib/tts/toasteruitest/run_toastertests.py deleted +0 -155
155 lines not on this page · raw message
lib/toaster/contrib/tts/toasteruitest/toaster_automation_test.py deleted +0 -2376
2,376 lines not on this page · raw message
lib/toaster/contrib/tts/toasteruitest/toaster_test.cfg deleted +0 -25
@@ -1,25 +0,0 @@ -# Configuration file for toaster_test -# Sorted by different host type - -# test browser could be: firefox; chrome; ie(still under development) -# logging_level could be: CRITICAL; ERROR; WARNING; INFO; DEBUG; NOTSET - - -[toaster_test_linux] -toaster_url = 'http://127.0.0.1:8000' -test_browser = 'firefox' -test_cases = [946] -logging_level = 'INFO' - - -[toaster_test_windows] -toaster_url = 'http://127.0.0.1:8000' -test_browser = ['ie', 'firefox', 'chrome'] -test_cases = [901, 902, 903] -logging_level = 'DEBUG' - -[toaster_test_darwin] -toaster_url = 'http://127.0.0.1:8000' -test_browser = 'firefox' -test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956] -logging_level = 'INFO'
lib/toaster/tests/toaster_test.cfg new file +25 -0
@@ -0,0 +1,25 @@ +# Configuration file for toaster_test +# Sorted by different host type + +# test browser could be: firefox; chrome; ie(still under development) +# logging_level could be: CRITICAL; ERROR; WARNING; INFO; DEBUG; NOTSET + + +[toaster_test_linux] +toaster_url = 'http://localhost:8000' +test_browser = 'firefox' +test_cases = [946] +logging_level = 'INFO' + + +[toaster_test_windows] +toaster_url = 'http://localhost:8000' +test_browser = ['ie', 'firefox', 'chrome'] +test_cases = [901, 902, 903] +logging_level = 'DEBUG' + +[toaster_test_darwin] +toaster_url = 'http://localhost:8000' +test_browser = 'firefox' +test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956] +logging_level = 'INFO'
lib/toaster/tests/ui.py new file +2376 -0
264 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.1.4
Part of the diff is not shown here. View the raw message.
RFC PATCH 1/1 2/8] toaster/tests: Add __init__.py and base.py.
In reply to Aníbal LimónAdd empty __init__.py for set folder as module.
Add base.py module that contains a base class for toaster test cases now have an object with options and a logger.
--- lib/toaster/tests/__init__.py | 0 lib/toaster/tests/base.py | 11 +++++++++++ 2 files changed, 11 insertions(+) create mode 100644 lib/toaster/tests/__init__.py create mode 100644 lib/toaster/tests/base.py
lib/toaster/tests/__init__.py new file +0 -0
lib/toaster/tests/base.py new file +11 -0
@@ -0,0 +1,11 @@ +import unittest + +class ToasterOptions(object): + pass + +class ToasterTestCase(unittest.TestCase): + def __init__(self, testname, opts, logger): + super(ToasterTestCase, self).__init__(testname) + + self.opts = opts + self.logger = logger
Signature · 1 line
2.1.4
RFC PATCH 1/1 3/8] toaster/tests: Improve ui.py module.
In reply to Aníbal LimónRemove shebang because this module can't be executed.
Minor fix use os.path.lexists instead of os.path.exists when test if symlink exists.
The toaster_cases_base class now inherit ToasterTestCase from base.py it enables to use self.opts instead of hardcod'ed sqlite db path, browser and toaster url.
--- lib/toaster/tests/ui.py | 158 ++++++++++++++++++++++-------------------------- 1 file changed, 73 insertions(+), 85 deletions(-)
lib/toaster/tests/ui.py +73 -85
@@ -1,4 +1,3 @@ -#!/usr/bin/python # Copyright # DESCRIPTION
@@ -32,6 +31,8 @@ from selenium.webdriver.common.keys import Keys from selenium.webdriver.support.ui import Select import sqlite3 as sqlite +from toaster.tests.base import ToasterTestCase + ########################################### # #
@@ -287,7 +288,7 @@ def LogResults(original_class): local_log.results("Testcase "+str(test_case)+": PASSED") # Create symlink to the current log - if os.path.exists(linkfile): + if os.path.lexists(linkfile): os.remove(linkfile) os.symlink(logfile, linkfile)
@@ -303,7 +304,7 @@ def LogResults(original_class): ########################################### @LogResults -class toaster_cases_base(unittest.TestCase): +class toaster_cases_base(ToasterTestCase): @classmethod def setUpClass(cls):
@@ -313,18 +314,6 @@ class toaster_cases_base(unittest.TestCase): self.screenshot_sequence = 1 self.verificationErrors = [] self.accept_next_alert = True - self.host_os = platform.system().lower() - if os.getenv('TOASTER_SUITE'): - self.target_suite = os.getenv('TOASTER_SUITE') - else: - self.target_suite = self.host_os - - self.parser = ConfigParser.SafeConfigParser() - self.parser.read('toaster_test.cfg') - self.base_url = eval(self.parser.get('toaster_test_' + self.target_suite, 'toaster_url')) - - # create log dir . Currently , we put log files in log/tmp. After all - # test cases are done, move them to log/$datetime dir self.log_tmp_dir = os.path.abspath(sys.path[0]) + os.sep + 'log' + os.sep + 'tmp' try: mkdir_p(self.log_tmp_dir)
@@ -360,13 +349,12 @@ class toaster_cases_base(unittest.TestCase): def setup_browser(self, *browser_path): - self.browser = eval(self.parser.get('toaster_test_' + self.target_suite, 'test_browser')) - print self.browser - if self.browser == "firefox": + print self.opts.test_browser + if self.opts.test_browser == "firefox": driver = webdriver.Firefox() - elif self.browser == "chrome": + elif self.opts.test_browser == "chrome": driver = webdriver.Chrome() - elif self.browser == "ie": + elif self.opts.test_browser == "ie": driver = webdriver.Ie() else: driver = None
@@ -401,12 +389,12 @@ class toaster_cases_base(unittest.TestCase): for item in types: log_dir = self.log_tmp_dir + os.sep + sub_dir mkdir_p(log_dir) - log_path = log_dir + os.sep + self.browser + '-' +\ + log_path = log_dir + os.sep + self.opts.test_browser + '-' +\ item + '-' + add_name + '-' + str(self.screenshot_sequence) + '.png' if item == 'native': - if self.host_os == "linux": + if self.opts.host_os == "linux": os.system("scrot " + log_path) - elif self.host_os=="darwin": + elif self.opts.host_os=="darwin": os.system("screencapture -x " + log_path) elif item == 'selenium': self.driver.get_screenshot_as_file(log_path)
@@ -419,7 +407,7 @@ class toaster_cases_base(unittest.TestCase): But for firefox, mostly we don't need this. To be discussed """ - if self.browser == "chrome": + if self.opts.test_browser == "chrome": time.sleep(1) return
@@ -691,7 +679,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) # open all columns self.driver.find_element_by_id("edit-columns-button").click() # adding explicitly wait for chromedriver..-_-
@@ -731,7 +719,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) # Could add more test patterns here in the future. Also, could search some items other than target column in future.. patterns = ["minimal", "sato"] for pattern in patterns:
@@ -757,7 +745,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) # when opening a new page, "started_on" is not displayed by default self.driver.find_element_by_id("edit-columns-button").click() # currently all the delay are for chrome driver -_-
@@ -806,7 +794,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_partial_link_text("core-image").click() self.driver.find_element_by_link_text("Tasks").click() self.table_name = 'otable'
@@ -947,7 +935,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.find_element_by_link_text_in_table('nav', 'Packages').click() # find "bash" in first column (Packages)
@@ -984,7 +972,7 @@ class toaster_cases(toaster_cases_base): test_package1="busybox" test_package2="lib" self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text(image_type).click() self.driver.find_element_by_link_text("Recipes").click() self.save_screenshot(screenshot_type='selenium', append_name='step3')
@@ -1123,7 +1111,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.find_element_by_link_text_in_table('nav', 'Recipes').click() # step 3-5
@@ -1150,7 +1138,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.find_element_by_link_text_in_table('nav', 'Recipes').click() # step 3
@@ -1190,7 +1178,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.find_element_by_link_text_in_table('nav', 'Recipes').click() # step 3
@@ -1217,7 +1205,7 @@ class toaster_cases(toaster_cases_base): test_package3="gettext-native" driver = self.driver driver.maximize_window() - driver.get(self.base_url) + driver.get(self.opts.toaster_url) driver.find_element_by_link_text(image_type).click() driver.find_element_by_link_text("Recipes").click() driver.find_element_by_link_text(test_package1).click()
@@ -1398,7 +1386,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() # step 3 self.find_element_by_link_text_in_table('nav', 'Configuration').click()
@@ -1440,7 +1428,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() # step 2-3 self.find_element_by_link_text_in_table('nav', 'Configuration').click()
@@ -1468,7 +1456,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) # Step 2 # default sequence in "Completed on" column is inverted c_list = self.get_table_column_text('class', 'completed_on')
@@ -1490,7 +1478,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) # Please refer to case 924 requirement # default sequence in "Completed on" column is inverted c_list = self.get_table_column_text('class', 'completed_on')
@@ -1516,7 +1504,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() # Step 2-3 self.find_element_by_link_text_in_table('nav', 'Packages').click()
@@ -1545,7 +1533,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() # Step 2-3 self.find_element_by_link_text_in_table('nav', 'Packages').click()
@@ -1562,7 +1550,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.driver.find_element_by_link_text("Packages").click() #get initial table header
@@ -1582,7 +1570,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.driver.find_element_by_link_text("Packages").click() #search for the "bash" package -> this should definitely be present
@@ -1600,7 +1588,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() # step 1: test Recipes page stuff self.driver.find_element_by_link_text("Recipes").click()
@@ -1667,7 +1655,7 @@ class toaster_cases(toaster_cases_base): self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() for item in ["Packages", "Recipes", "Tasks"]: - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.driver.find_element_by_link_text(items).click()
@@ -1702,7 +1690,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.driver.find_element_by_link_text("Configuration").click() # step 3-4
@@ -1765,7 +1753,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.find_element_by_link_text_in_table('nav', 'Configuration').click() # step 2
@@ -1801,7 +1789,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.find_element_by_link_text_in_table('nav', 'Configuration').click() self.driver.find_element_by_link_text("BitBake variables").click()
@@ -1823,7 +1811,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_link_text("core-image-minimal").click() self.find_element_by_link_text_in_table('nav', 'core-image-minimal').click() # step 3
@@ -1873,7 +1861,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) # step3&4: so far we're not sure if there's "successful build" or "failed # build".If either of them doesn't exist, we can still go on other steps check_list = ['Configuration', 'Tasks', 'Recipes', 'Packages', 'Time', 'CPU usage', 'Disk I/O']
@@ -1930,7 +1918,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) # currently test case itself isn't responsible for creating "1 successful and # 1 failed build" has_successful_build = 1
@@ -1957,7 +1945,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.log.info(" You should manually create all images before test starts!") # So far the case itself is not responsable for creating all sorts of images. # So assuming they are already there
@@ -1973,7 +1961,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) # step 2-3 need to run manually self.log.info("step 2-3: checking the help message when you hover on help icon of target,\ tasks, recipes, packages need to run manually")
@@ -1992,7 +1980,7 @@ class toaster_cases(toaster_cases_base): def test_1066(self): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select count(name) from orm_project a, auth_user b where a.user_id = b.id and b.username='_anonuser';" cursor.execute(query)
@@ -2006,7 +1994,7 @@ class toaster_cases(toaster_cases_base): def test_1071(self): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select name from orm_release;" cursor.execute(query)
@@ -2015,7 +2003,7 @@ class toaster_cases(toaster_cases_base): data[i] = data[i][0] data.sort() print data - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_data = [] for i in range (0,4): json_data.append(json_parse['releases'][i]['name'])
@@ -2029,7 +2017,7 @@ class toaster_cases(toaster_cases_base): def test_1072(self): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select value from orm_toastersetting where name like 'DEFCONF%';" cursor.execute(query)
@@ -2037,7 +2025,7 @@ class toaster_cases(toaster_cases_base): for i in range(0,6): data[i] = data[i][0] print data - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_data=json_parse['config'] json_data = json_data.values() print json_data
@@ -2050,7 +2038,7 @@ class toaster_cases(toaster_cases_base): def test_1074(self): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select name from orm_layersource;" cursor.execute(query)
@@ -2058,7 +2046,7 @@ class toaster_cases(toaster_cases_base): for i in range(0,3): data[i] = data[i][0] print data - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_data = [] for i in range(0,3): json_data.append(json_parse['layersources'][i]['name'])
@@ -2071,14 +2059,14 @@ class toaster_cases(toaster_cases_base): def test_1075(self): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select value from orm_toastersetting where name like 'DEFAULT_RELEASE';" cursor.execute(query) data = cursor.fetchall() data = data[0][0] print data - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_data = json_parse['defaultrelease'] print json_data self.failUnless(set(data) == set(json_data))
@@ -2091,7 +2079,7 @@ class toaster_cases(toaster_cases_base): self.log.info(' CASE %s log: ' % str(self.case_no)) print 'Checking branches for "Local Yocto Project"' - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select name from orm_branch where layer_source_id=1;" cursor.execute(query)
@@ -2103,7 +2091,7 @@ class toaster_cases(toaster_cases_base): except: pass print data - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_location = json_parse['layersources'][0]['name'] print json_location json_data = json_parse['layersources'][0]['branches']
@@ -2111,7 +2099,7 @@ class toaster_cases(toaster_cases_base): self.failUnless(set(data) == set(json_data)) print 'Checking branches for "OpenEmbedded"' - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select name from orm_branch where layer_source_id=3;" cursor.execute(query)
@@ -2120,7 +2108,7 @@ class toaster_cases(toaster_cases_base): for i in range(0,lenght): data[i] = data[i][0] print data - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_location = json_parse['layersources'][1]['name'] print json_location json_data = json_parse['layersources'][1]['branches']
@@ -2128,7 +2116,7 @@ class toaster_cases(toaster_cases_base): self.failUnless(set(data) == set(json_data)) print 'Checking branches for "Imported layers"' - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select name from orm_branch where layer_source_id=2;" cursor.execute(query)
@@ -2137,7 +2125,7 @@ class toaster_cases(toaster_cases_base): for i in range(0,lenght): data[i] = data[i][0] print data - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_location = json_parse['layersources'][2]['name'] print json_location json_data = json_parse['layersources'][2]['branches']
@@ -2151,7 +2139,7 @@ class toaster_cases(toaster_cases_base): def test_1077(self): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select name from orm_bitbakeversion;" cursor.execute(query)
@@ -2159,7 +2147,7 @@ class toaster_cases(toaster_cases_base): for i in range(0,4): data[i] = data[i][0] print data - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_data = [] for i in range(0,4): json_data.append(json_parse['bitbake'][i]['name'])
@@ -2173,11 +2161,11 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_id("new-project-button").click() self.driver.find_element_by_id("new-project-name").send_keys("new-test-project") self.driver.find_element_by_id("create-project-button").click() - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select count(name) from orm_project where name = 'new-test-project';" cursor.execute(query)
@@ -2192,16 +2180,16 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_id("new-project-button").click() self.driver.find_element_by_id("new-project-name").send_keys("new-default-project") self.driver.find_element_by_id("create-project-button").click() - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select a.name from orm_release a, orm_project b where a.id = b.release_id and b.name = 'new-default-project' limit 1;" cursor.execute(query) db_data = str(cursor.fetchone()[0]) - json_parse = json.loads(open('toasterconf.json').read()) + json_parse = json.loads(open(self.opts.toaster_config).read()) json_data = str(json_parse['defaultrelease']) self.failUnless(db_data == json_data)
@@ -2212,14 +2200,14 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_css_selector("a[href='/toastergui/projects/']").click() self.driver.find_element_by_link_text('new-default-project').click() self.driver.find_element_by_id('project-change-form-toggle').click() self.driver.find_element_by_id('project-name-change-input').clear() self.driver.find_element_by_id('project-name-change-input').send_keys('new-name') self.driver.find_element_by_id('project-name-change-btn').click() - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select count(name) from orm_project where name = 'new-name';" cursor.execute(query)
@@ -2239,7 +2227,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_css_selector("a[href='/toastergui/projects/']").click() self.driver.find_element_by_link_text('new-default-project').click() self.driver.find_element_by_id('change-machine-toggle').click()
@@ -2247,7 +2235,7 @@ class toaster_cases(toaster_cases_base): self.driver.find_element_by_id('machine-change-input').send_keys('qemuarm64') # self.driver.find_element_by_id('machine-change-input').send_keys(Keys.RETURN) self.driver.find_element_by_id('machine-change-btn').click() - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select count(id) from orm_projectvariable where name like 'machine' and value like 'qemuarm64';" cursor.execute(query)
@@ -2266,7 +2254,7 @@ class toaster_cases(toaster_cases_base): def test_1090(self): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select username from auth_user where is_superuser = 1;" cursor.execute(query)
@@ -2285,7 +2273,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - self.driver.get(self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_css_selector("a[href='/toastergui/projects/']").click() self.driver.find_element_by_link_text('new-default-project').click() self.driver.find_element_by_id('release-change-toggle').click()
@@ -2296,7 +2284,7 @@ class toaster_cases(toaster_cases_base): self.driver.find_element_by_id('change-release-btn').click() #wait for the changes to register in the DB time.sleep(1) - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select count(*) from orm_layer_version a, orm_projectlayer b, orm_project c where a.\"commit\"=\"HEAD\" and a.id = b.layercommit_id and b.project_id=c.id and c.name='new-default-project';" cursor.execute(query)
@@ -2319,7 +2307,7 @@ class toaster_cases(toaster_cases_base): self.case_no = self.get_case_number() self.log.info(' CASE %s log: ' % str(self.case_no)) self.driver.maximize_window() - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select a.name, a.value from orm_projectvariable a, orm_project b where a.project_id = b.id and b.name = 'new-default-project';" cursor.execute(query)
@@ -2336,7 +2324,7 @@ class toaster_cases(toaster_cases_base): self.log.info(' CASE %s log: ' % str(self.case_no)) #get initial values - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select layercommit_id from orm_projectlayer a, orm_project b where a.project_id=b.id and b.name='new-default-project';" cursor.execute(query)
@@ -2344,7 +2332,7 @@ class toaster_cases(toaster_cases_base): print data_initial self.driver.maximize_window() - self.driver.get('localhost:8000')#self.base_url) + self.driver.get(self.opts.toaster_url) self.driver.find_element_by_css_selector("a[href='/toastergui/projects/']").click() self.driver.find_element_by_link_text('new-default-project').click() self.driver.find_element_by_id('release-change-toggle').click()
@@ -2357,7 +2345,7 @@ class toaster_cases(toaster_cases_base): time.sleep(1) #get changed values - con=sqlite.connect('toaster.sqlite') + con=sqlite.connect(self.opts.toaster_db) cursor = con.cursor() query = "select layercommit_id from orm_projectlayer a, orm_project b where a.project_id=b.id and b.name='new-default-project';" cursor.execute(query)
Signature · 1 line
2.1.4
RFC PATCH 1/1 4/8] toaster/tests: Adds setup test case.
In reply to Aníbal LimónThe setup test case is responsible to leave toaster in certain state for execute ui tests now it creates selenium-project and builds core-image-{minimal, sato} images.
--- lib/toaster/tests/setup.py | 77 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 lib/toaster/tests/setup.py
lib/toaster/tests/setup.py new file +77 -0
@@ -0,0 +1,77 @@ +import time + +from selenium import webdriver +from selenium.common.exceptions import NoSuchElementException +from selenium import selenium +from selenium.webdriver.common.by import By +from selenium.webdriver.common.keys import Keys +from selenium.webdriver.support.ui import Select + +from toaster.tests.base import ToasterTestCase + +class ToasterSetupTestCase(ToasterTestCase): + def setUp(self): + self.driver = webdriver.Firefox() + + def tearDown(self): + self.driver.close() + + def is_text_present (self, patterns): + for pattern in patterns: + if str(pattern) not in self.driver.page_source: + self.logger.debug(pattern) + return False + return True + + def test_setupToaster(self): + self.driver.maximize_window() + self.driver.get(self.opts.toaster_url) + try: + self.driver.find_element_by_css_selector("a[href='/toastergui/projects/']").click() + except: + self.driver.find_element_by_id("new-project-button").click() + self.driver.find_element_by_id("new-project-name").send_keys("selenium-project") + self.driver.find_element_by_id("create-project-button").click() + + try: + self.driver.find_element_by_link_text("selenium-project").click() + except: + self.driver.find_element_by_id("new-project-button").click() + self.driver.find_element_by_id("new-project-name").send_keys("selenium-project") + self.driver.find_element_by_id("create-project-button").click() + time.sleep(5) + + #queue up a core-image-minimal + self.driver.find_element_by_id("build-input").send_keys("core-image-minimal") + self.driver.find_element_by_id("build-button").click() + time.sleep(5) + #queue up a core-image-sato + self.driver.find_element_by_id("build-input").send_keys("core-image-sato") + self.driver.find_element_by_id("build-button").click() + time.sleep(5) + + #go back to the main project page + self.driver.find_element_by_css_selector("a[href='/toastergui/projects/']").click() + time.sleep(5) + self.driver.find_element_by_link_text("selenium-project").click() + time.sleep(5) + + #move to all builds page + self.driver.find_element_by_css_selector("a[href='/toastergui/builds/']").click() + self.driver.refresh() + time.sleep(5) + + #check progress bar is displayed to signal a build has started + try: + while (self.driver.find_element_by_xpath("//div[@class='progress']").is_displayed()): + self.logger.info('Builds are running ...') + self.driver.refresh() + time.sleep(15) + except: + pass + + self.logger.info("Looking for successful build...") + if (self.driver.find_element_by_xpath("//div[@class='alert build-result alert-success project-name']").is_displayed()): + self.logger.info("Builds complete!") + else: + self.fail(msg="Builds did not complete successfully.")
Signature · 1 line
2.1.4
RFC PATCH 1/1 5/8] toaster/tests: Add helpers.py module.
In reply to Aníbal LimónThe helpers.py module is a utility for clone/setup_venv/start/stop a toaster instance.
It can be executed in a standalone way or use as utility class.
--- lib/toaster/tests/helpers.py | 178 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 178 insertions(+) create mode 100755 lib/toaster/tests/helpers.py
lib/toaster/tests/helpers.py new file +178 -0
@@ -0,0 +1,178 @@ +#!/usr/bin/env python + +# Copyright (C) 2016 Intel Corporation +# + +import subprocess +import os +import shutil +import signal +import tempfile + +from proc.core import find_processes + +TOASTER_TEST_BRANCH = 'toaster_tests' +VENV_NAME = 'venv' +SHELL_CMD = os.environ['SHELL'] if 'SHELL' in os.environ else "/bin/bash" +TOASTER_BUILD_DIR_NAME = 'build' + +def _check_output1(*popenargs, **kwargs): + """ + Almost the same as subprocess.check_output but change the stdout from + PIPE to tempfile to avoid deadlocks when trying to read the PIPE using + communicate(). This scenario can be seen calling toaster_start on failure + scenarios. + + This causes a little overhead by the tempfile. + """ + + f = tempfile.TemporaryFile(mode='rw+') + if 'stdout' in kwargs: + raise ValueError('stdout argument not allowed, it will be overridden.') + process = subprocess.Popen(stdout=f, *popenargs, **kwargs) + retcode = process.wait() + + f.flush() + os.fsync(f.fileno()) + f.seek(0, 0) + output = f.read() + f.close() + + if retcode: + cmd = kwargs.get("args") + if cmd is None: + cmd = popenargs[0] + raise subprocess.CalledProcessError(retcode, cmd, output=output) + return output + +class ToasterHelper(object): + def __init__(self, directory, repo, repo_ref='master', port='8000', + build_dir=TOASTER_BUILD_DIR_NAME): + self.directory = directory + self.repo = repo + self.repo_ref = repo_ref + self.port = port + self.build_dir = build_dir + + def _execute_command(self, cmd): + return _check_output1([SHELL_CMD, "-c", "cd %s; %s" % \ + (self.directory, cmd)], stderr=subprocess.STDOUT) + + def _execute_command_oe(self, cmd): + return self._execute_command("source oe-init-build-env %s; %s"\ + % (self.build_dir, cmd)) + + def _execute_command_oe_venv(self, cmd): + return self._execute_command_oe("source %s/bin/activate; %s"\ + % (VENV_NAME, cmd)) + + def clone(self, rm=False): + if os.path.exists(self.directory): + if rm: + shutil.rmtree(self.directory) + else: + raise IOError + + subprocess.check_output([SHELL_CMD, "-c", "git clone %s %s" %\ + (self.repo, self.directory)], stderr=subprocess.STDOUT) + self._execute_command("git checkout %s; git branch %s; git checkout %s;"\ + % (self.repo_ref, TOASTER_TEST_BRANCH, TOASTER_TEST_BRANCH)) + + def setup(self): + self._execute_command_oe("virtualenv %s" % VENV_NAME) + self._execute_command_oe_venv("pip install -r" \ + " ../bitbake/toaster-requirements.txt") + self._execute_command_oe_venv("pip install -r" \ + " ../bitbake/toaster-tests-requirements.txt") + + def start(self): + return self._execute_command_oe_venv("source %s/bitbake/bin/toaster"\ + " start webport=%s" % (self.directory, self.port)) + + def _stop_force(self): + """ + The _stop_force method iterates over the /proc and search for toaster path + in the process cmdline then send SIGKILL for every matched process. + """ + + pids = [] + for p in find_processes(): + if len(p.cmdline) > 1 and \ + os.path.basename(p.cmdline[0]) == 'python' and \ + p.cmdline[1].startswith(self.directory): + pids.append(p.pid) + + for pid in pids: + try: + os.kill(pid, signal.SIGKILL) + except: + pass + + return '' + + def stop(self, force=False): + """ + The stop method have force mode because toaster without production + setup have known issues when is on load, the server response 503 + service unavailable. + """ + if force: + return self._stop_force() + else: + return self._execute_command_oe_venv("source %s/bitbake/bin/toaster"\ + " stop webport=%s" % (self.directory, self.port)) + + def get_sqlite_path(self): + return os.path.join(self.directory, self.build_dir, 'toaster.sqlite') + + def get_conf_path(self): + output = self._execute_command_oe_venv("source %s/bitbake/bin/toaster"\ + " start webport=%s; echo $TOASTER_CONF;" % \ + (self.directory, self.port)) + + return output.split()[-1] + +DEFAULT_WORK_DIRECTORY = '/tmp/toaster' +DEFAULT_POKY_URL = 'http://git.yoctoproject.org/git/poky.git' +DEFAULT_POKY_BRANCH = 'master' +DEFAULT_WEB_PORT = '8000' + +if __name__ == "__main__": + import argparse + import sys + + parser = argparse.ArgumentParser(description='Toaster helper', + formatter_class=argparse.RawTextHelpFormatter) + parser.add_argument("-a", "--action", choices=['clone', 'setup', 'start', 'stop'], + required=True, + help="Action to execute can be clone/setup/start/stop") + parser.add_argument("-d", "--work-directory", default=DEFAULT_WORK_DIRECTORY, + help="Directory for setup toaster, default: %s" % DEFAULT_WORK_DIRECTORY) + parser.add_argument("-u", "--url-repository", default=DEFAULT_POKY_URL, + help="GIT repository for setup toaster, default %s" % DEFAULT_POKY_URL) + parser.add_argument("-r", "--revision", default=DEFAULT_POKY_BRANCH, + help="GIT repository revision (branch, tag, hash) for setup toaster" \ + ", default: %s" % DEFAULT_POKY_BRANCH) + parser.add_argument("-p", "--web-port", default=DEFAULT_WEB_PORT, + help="Web port for start toaster server, default: %s" % DEFAULT_WEB_PORT) + parser.add_argument("-b", "--build-dir", default=TOASTER_BUILD_DIR_NAME, + help="Toaster build directory name, default: %s" % TOASTER_BUILD_DIR_NAME) + + args = parser.parse_args() + + try: + th = ToasterHelper(args.work_directory, args.url_repository, + args.revision, args.web_port, args.build_dir) + if args.action == 'stop': + th.stop(force=True) + else: + getattr(th, args.action)() + except subprocess.CalledProcessError as e: + print e.output + raise + except IOError as e: + if args.action == 'clone': + print "ERROR: Directory exists: %s" % args.work_directory + raise + + sys.exit(0)
Signature · 1 line
2.1.4
RFC PATCH 1/1 6/8] toaster-test: Refactor and imporvements.
In reply to Aníbal LimónAdd command line options for specify config, toaster_config, database, toaster url instance, web browser and verbose/debug. Some options (toaster url, test cases) are read from conf first and then command line option could override it.
Options are passed to the unit tests that now uses ToasterTestCase base class. Also the logger is initialized and passed to the test cases.
The logic for discover tests was improved now it looks at a fixed dictionary test_suites and could be extended easily.
The setup test case is executed first before another test cases like the UI one. If setup test case fails another tests aren't executed.
--- bin/toaster-test | 278 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 172 insertions(+), 106 deletions(-)
bin/toaster-test +172 -106
@@ -1,155 +1,221 @@ -#!/usr/bin/python +#!/usr/bin/env python -# Copyright +import sys +import os +import argparse +import unittest +import logging -# DESCRIPTION -# This is script for running all selected toaster cases on -# selected web browsers manifested in toaster_test.cfg. +import importlib -# 1. How to start toaster in yocto: -# $ source poky/oe-init-build-env -# $ source toaster start -# $ bitbake core-image-minimal +bitbake_libdir = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib') +sys.path.insert(0, bitbake_libdir) -# 2. How to install selenium on Ubuntu: -# $ sudo apt-get install scrot python-pip -# $ sudo pip install selenium +from toaster.tests.base import ToasterOptions +from toaster.tests.setup import ToasterSetupTestCase -# 3. How to install selenium addon in firefox: -# Download the lastest firefox addon from http://release.seleniumhq.org/selenium-ide/ -# Then install it. You can also install firebug and firepath addon +DEFAULT_TOASTER_TESTS_CFG = os.path.join(bitbake_libdir, 'toaster', 'tests', + 'toaster_test.cfg') +DEFAULT_TOASTER_CFG = os.path.join(os.environ['BUILDDIR'], '..', 'meta-yocto', + 'conf', 'toasterconf.json') +DEFAULT_TOASTER_DB = os.path.join(os.environ['BUILDDIR'], 'toaster.sqlite') -# 4. How to start writing a new case: -# All you need to do is to implement the function test_xxx() and pile it on. +test_suites = {} +test_suites['toaster.tests.ui'] = ['toaster_cases'] -# 5. How to test with Chrome browser -# Download/install chrome on host -# Download chromedriver from https://code.google.com/p/chromedriver/downloads/list according to your host type -# put chromedriver in PATH, (e.g. /usr/bin/, bear in mind to chmod) -# For windows host, you may put chromedriver.exe in the same directory as chrome.exe +logger = None -import unittest, sys, os, platform -import ConfigParser -import argparse -from toaster_automation_test import toaster_cases +def load_test(test, args, logger): + """ + Returns an instance of class passing arguments + to the class. + """ + module_name = '.'.join(test.split('.')[:-2]) + class_name = test.split('.')[-2] + test_name = test.split('.')[-1] -def get_args_parser(): - description = "Script that runs toaster auto tests." - parser = argparse.ArgumentParser(description=description) - parser.add_argument('--run-all-tests', required=False, action="store_true", dest="run_all_tests", default=False, - help='Run all tests.') - parser.add_argument('--run-suite', required=False, dest='run_suite', default=False, - help='run suite (defined in cfg file)') + module = importlib.import_module(module_name) + clss = getattr(module, class_name) - return parser + return clss(test_name, args, logger) +def get_tests(tests_filter=None): + """ + Iterate over available tests suites and serach + for tests that starts with test_. + + If tests_filter is specified only add tests that + are in the test_filter list. + """ -def get_tests(): testslist = [] + tests_filter_dic = None + + if tests_filter: + tests_filter_dic = dict((str(tf), False) for tf in tests_filter) + + def _filter_tests(test, tests_filter_dic): + if not tests_filter_dic: + return True + if test in tests_filter_dic.keys(): + tests_filter_dic[test] = True + return True + return False + + for ts in test_suites.keys(): + m = importlib.import_module(ts) + for clss in test_suites[ts]: + for o in dir(getattr(m, clss)): + full_clss = '.'.join((ts, clss)) + if o.startswith('test_') and _filter_tests(o[5:], tests_filter_dic): + testslist.append('.'.join((full_clss, o))) - prefix = 'toaster_automation_test.toaster_cases' + return testslist - for t in dir(toaster_cases): - if t.startswith('test_'): - testslist.append('.'.join((prefix, t))) +def get_all_tests(): + return get_tests() - return testslist +def get_tests_from_cfg(config, suite): + try: + tests_from_cfg = eval(config.get('toaster_test_' + suite, 'test_cases')) + except: + logger.error('Failed to get test cases from config file.' \ + 'Make sure the format is correct.') + raise + testslist = get_tests(tests_from_cfg) -def get_tests_from_cfg(suite=None): + return testslist - testslist = [] - config = ConfigParser.SafeConfigParser() - config.read('toaster_test.cfg') +def buildResultClass(args): + """Build a Result Class to use in the testcase execution""" - if suite is not None: - target_suite = suite.lower() + class StampedResult(unittest.TextTestResult): + """ + Custom TestResult that prints the time when a test starts. As toaster-auto + can take a long time (ie a few hours) to run, timestamps help us understand + what tests are taking a long time to execute. + """ + def startTest(self, test): + import time + self.stream.write(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) + " - ") + super(StampedResult, self).startTest(test) - # TODO: if suite is valid suite + return StampedResult +def get_args(): + parser = argparse.ArgumentParser(description="Script for run toaster tests.") + + parser.add_argument('--run-all-tests', required=False, action="store_true", + dest="run_all_tests", default=False, help='Run all tests.') + parser.add_argument('--run-suite', required=False, dest='run_suite', + choices=['linux', 'windows', 'darwin'], + help='Run suite (defined in config file)') + parser.add_argument('--config', dest="config", + default=DEFAULT_TOASTER_TESTS_CFG, + help='Configuration for toaster tests, default: %s' %\ + DEFAULT_TOASTER_TESTS_CFG) + parser.add_argument('--toaster-config', dest="toaster_config", + default=DEFAULT_TOASTER_CFG, + help='Configuration of toaster, default: %s' %\ + DEFAULT_TOASTER_CFG) + parser.add_argument('--toaster-db', dest="toaster_db", + default=DEFAULT_TOASTER_DB, + help='Database of toaster, default: %s' %\ + DEFAULT_TOASTER_DB) + parser.add_argument('--toaster-url', dest="toaster_url", + help='URL of toaster instance') + parser.add_argument('--test-browser', dest='test_browser', + choices=['firefox', 'chrome', 'ie'], + help='Web browser for use to run tests.') + parser.add_argument('--verbose', required=False, action="store_true", + dest="verbose", default=False, help='Enable verbose mode.') + parser.add_argument('--debug', required=False, action="store_true", + dest="debug", default=False, help='Enable debug mode.') + + return parser.parse_args() + +def get_options(args): + opts = ToasterOptions() + + opts.host_os = args.host_os + opts.config = args.config + opts.toaster_config = args.toaster_config + opts.toaster_db = args.toaster_db + opts.toaster_url = args.toaster_url + opts.test_browser = args.test_browser + + opts.test_list = args.test_list + + return opts + +def main(args): + import platform + import ConfigParser + + if args.verbose or args.debug: + logging.basicConfig(stream=sys.stdout) + root = logging.getLogger() + root.setLevel(logging.DEBUG if args.debug else logging.INFO) else: - target_suite = platform.system().lower() + logging.basicConfig(stream=sys.stderr) + logger = logging.getLogger() - try: - tests_from_cfg = eval(config.get('toaster_test_' + target_suite, 'test_cases')) - except: - print 'Failed to get test cases from cfg file. Make sure the format is correct.' - return None + args.host_os = platform.system().lower() + if not args.run_suite: + args.run_suite = args.host_os - prefix = 'toaster_automation_test.toaster_cases.test_' - for t in tests_from_cfg: - testslist.append(prefix + str(t)) + config = ConfigParser.SafeConfigParser() + config.read(args.config) - return testslist + if args.run_all_tests: + args.test_list = get_all_tests() + else: + args.test_list = get_tests_from_cfg(config, args.run_suite) -def main(): + if not args.toaster_url: + args.toaster_url = eval(config.get('toaster_test_' + args.run_suite, + 'toaster_url')) + if not args.test_browser: + args.test_browser = eval(config.get('toaster_test_' + args.run_suite, + 'test_browser')) - # In case this script is called from other directory - os.chdir(os.path.abspath(sys.path[0])) + if not args.test_list: + logger.error('Failed to get test cases.') + sys.exit(1) - parser = get_args_parser() - args = parser.parse_args() + opts = get_options(args) - if args.run_all_tests: - testslist = get_tests() - elif args.run_suite: - testslist = get_tests_from_cfg(args.run_suite) - os.environ['TOASTER_SUITE'] = args.run_suite - else: - testslist = get_tests_from_cfg() + runner = unittest.TextTestRunner(verbosity=2, resultclass=buildResultClass(args)) - if not testslist: - print 'Failed to get test cases.' - exit(1) + setup_suite = unittest.TestSuite() + setup_suite.addTest(ToasterSetupTestCase('test_setupToaster', opts, logger)) suite = unittest.TestSuite() - loader = unittest.TestLoader() - loader.sortTestMethodsUsing = None - runner = unittest.TextTestRunner(verbosity=2, resultclass=buildResultClass(args)) - - for test in testslist: + for test in args.test_list: try: - suite.addTests(loader.loadTestsFromName(test)) + suite.addTest(load_test(test, opts, logger)) except: - return 1 + logger.error("Can't load test %s" % str(test)) + raise - result = runner.run(suite) + result = runner.run(setup_suite) + if result.wasSuccessful(): + result = runner.run(suite) if result.wasSuccessful(): return 0 else: return 1 - -def buildResultClass(args): - """Build a Result Class to use in the testcase execution""" - - class StampedResult(unittest.TextTestResult): - """ - Custom TestResult that prints the time when a test starts. As toaster-auto - can take a long time (ie a few hours) to run, timestamps help us understand - what tests are taking a long time to execute. - """ - def startTest(self, test): - import time - self.stream.write(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) + " - ") - super(StampedResult, self).startTest(test) - - return StampedResult - - if __name__ == "__main__": + args = get_args() try: - ret = main() + ret = main(args) except: - ret = 1 import traceback - traceback.print_exc(5) - finally: - if os.getenv('TOASTER_SUITE'): - del os.environ['TOASTER_SUITE'] + ret = 1 + traceback.print_exc() sys.exit(ret) - -
Signature · 1 line
2.1.4
RFC PATCH 1/1 7/8] toaster/tests: Add README, TODO and requeriments.
In reply to Aníbal Limón--- lib/toaster/tests/README | 64 ++++++++++++++++++++++++++++++++++++++++++ lib/toaster/tests/TODO | 13 +++++++++ toaster-tests-requirements.txt | 5 ++++ 3 files changed, 82 insertions(+) create mode 100644 lib/toaster/tests/README create mode 100644 lib/toaster/tests/TODO create mode 100644 toaster-tests-requirements.txt
lib/toaster/tests/README new file +64 -0
@@ -0,0 +1,64 @@ += Toaster tests = + +== Host requirements == + +All the test components REQUIRE bash as your default shell, you +need to review if /bin/sh points to bash because some problems are +detected when use dash. + +=== Toaster === + +- virtualenv for python +- pip for python + +=== Toaster UI === + +- firefox web browser for use selenium +- firefox selenium addon, for install download latest from [1]. +- vncserver (optional) + +[1] http://release.seleniumhq.org/selenium-ide/ + +=== Debian INSTALL === + +# apt-get install iceweasel python-virtualenv python-pip + +Optional: + +# apt-get install vnc4server + +== Setup == + +For run any of the tests scripts you need to setup virtualenv and install +requirements with pip. + +$ cd /home/user/pokydir +$ source oe-init-build-env build +$ virtualenv venv +$ source venv/bin/activate +$ pip install -r toaster-requirements.txt +$ pip install -r toaster-tests-requirements.txt + +== Toaster test == + +For run toaster you need to start a toaster instance and then use toaster-test +program, it can take sometime because it setup a toaster instance and run +testsuite over it. + +For start/stop a toaster instance a helper exists and need to be called with +the path to the poky directory. + +$ cd /home/user/pokydir +$ source oe-init-build-env build +$ source venv/bin/activate +$ ../bitbake/lib/toaster/tests/helpers.py -a start -d /home/user/pokydir -b build +$ toaster-test --run-all-tests +$ ../bitbake/lib/toaster/tests/helpers.py -a stop -d /home/user/pokydir -b build + +Optional: Toaster test suite uses selenium and spawn Firefox web browser for +run the tests, if you are in a development machine may be is a good idea to +start vncserver and set DISPLAY before run to don't interfer with your tasks, +example: + +$ vncserver :10 +$ export DISPLAY=:10
lib/toaster/tests/TODO new file +13 -0
@@ -0,0 +1,13 @@ +setup.py: + + - Improve logic of timeout now have a fixed time, + will be better to ask toaster instance about the + state of image building. + +ui.py: + - Class toaster_cases_base review and move common code + to ToasterTestCase. + - Class toaster_cases split into ToasterUITestCases (< 1000) + and ToasterBackendTestCases (> 1000). + - Class toaster_cases fix current tests to match Toaster in + master (css id's update)
toaster-tests-requirements.txt new file +5 -0
@@ -0,0 +1,5 @@ +# helpers, for stop force +proc + +# ui, general +selenium
Signature · 1 line
2.1.4
RFC PATCH 1/1 8/8] toaster/tests: Adds Copyright information.
In reply to Aníbal Limón--- bin/toaster-test | 17 +++++++++++++++++ lib/toaster/tests/base.py | 17 +++++++++++++++++ lib/toaster/tests/helpers.py | 14 ++++++++++++++ lib/toaster/tests/setup.py | 17 +++++++++++++++++ lib/toaster/tests/ui.py | 17 ++++++++++++++++- 5 files changed, 81 insertions(+), 1 deletion(-)
bin/toaster-test +17 -0
@@ -1,5 +1,22 @@ #!/usr/bin/env python +# Toaster test runner +# +# Copyright (C) 2015-2016 Intel Corporation +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + import sys import os import argparse
lib/toaster/tests/base.py +17 -0
@@ -1,3 +1,20 @@ +# Toaster tests base class +# +# Copyright (C) 2016 Intel Corporation +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + import unittest class ToasterOptions(object):
lib/toaster/tests/helpers.py +14 -0
@@ -1,7 +1,21 @@ #!/usr/bin/env python +# Toaster helper for clone/setup/start/stop +# # Copyright (C) 2016 Intel Corporation # +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import subprocess import os
lib/toaster/tests/setup.py +17 -0
@@ -1,3 +1,20 @@ +# Toaster setup test case +# +# Copyright (C) 2015-2016 Intel Corporation +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + import time from selenium import webdriver
lib/toaster/tests/ui.py +16 -1
@@ -1,4 +1,19 @@ -# Copyright +# Toaster UI/Backend test cases +# +# Copyright (C) 2015-2016 Intel Corporation +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. # DESCRIPTION # This is toaster automation base class and test cases file
Signature · 1 line
2.1.4
I forget to comment that the changes can be also found in:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash
Cheers,
alimon
Quoted 51 lines · Aníbal Limón
This set of changes refactor/improve the current test suite now it can be parameterized from command line.
The main goal for this set of patches are for integrate the toaster test suite into AB. The AB part is already on master. [1] If this patchset is merged the way for run Toaster test suite will change for use it read README in this patchset.
Futher work are related to improve/fix the current test suite also i added a TODO list.
Aníbal Limón (8):
toaster/tests: Reorder/move current test code. toaster/tests: Add__init__.pyandbase.py. toaster/tests: Improveui.pymodule. toaster/tests: Adds setup test case. toaster/tests: Addhelpers.pymodule.
toaster-test: Refactor and imporvements. toaster/tests: Add README, TODO and requeriments. toaster/tests: Adds Copyright information.bin/toaster-test | 238 ++ .../contrib/tts/toasteruitest/run_toastertests.py | 155 -- .../tts/toasteruitest/toaster_automation_test.py | 2376 ------------------- .../contrib/tts/toasteruitest/toaster_test.cfg | 25 - lib/toaster/tests/README | 64 + lib/toaster/tests/TODO | 13 + lib/toaster/tests/__init__.py | 0 lib/toaster/tests/base.py | 28 + lib/toaster/tests/helpers.py | 192 ++ lib/toaster/tests/setup.py | 94 + lib/toaster/tests/toaster_test.cfg | 25 + lib/toaster/tests/ui.py | 2379 ++++++++++++++++++++ toaster-tests-requirements.txt | 5 + 13 files changed, 3038 insertions(+), 2556 deletions(-) create mode 100755 bin/toaster-test delete mode 100755 lib/toaster/contrib/tts/toasteruitest/run_toastertests.py delete mode 100755 lib/toaster/contrib/tts/toasteruitest/toaster_automation_test.py delete mode 100644 lib/toaster/contrib/tts/toasteruitest/toaster_test.cfg create mode 100644 lib/toaster/tests/README create mode 100644 lib/toaster/tests/TODO create mode 100644 lib/toaster/tests/__init__.py create mode 100644 lib/toaster/tests/base.py create mode 100755 lib/toaster/tests/helpers.py create mode 100644 lib/toaster/tests/setup.py create mode 100644 lib/toaster/tests/toaster_test.cfg create mode 100644 lib/toaster/tests/ui.py create mode 100644 toaster-tests-requirements.txt
+ Bill
Quoted 61 lines · Aníbal Limón
I forget to comment that the changes can be also found in:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/toaster_tests_squash
Cheers,
alimonQuoted 51 lines · Aníbal Limón
This set of changes refactor/improve the current test suite now it can be parameterized from command line.
The main goal for this set of patches are for integrate the toaster test suite into AB. The AB part is already on master. [1] If this patchset is merged the way for run Toaster test suite will change for use it read README in this patchset.
Futher work are related to improve/fix the current test suite also i added a TODO list.
Aníbal Limón (8):
toaster/tests: Reorder/move current test code. toaster/tests: Add__init__.pyandbase.py. toaster/tests: Improveui.pymodule. toaster/tests: Adds setup test case. toaster/tests: Addhelpers.pymodule.
toaster-test: Refactor and imporvements. toaster/tests: Add README, TODO and requeriments. toaster/tests: Adds Copyright information.bin/toaster-test | 238 ++ .../contrib/tts/toasteruitest/run_toastertests.py | 155 -- .../tts/toasteruitest/toaster_automation_test.py | 2376 ------------------- .../contrib/tts/toasteruitest/toaster_test.cfg | 25 - lib/toaster/tests/README | 64 + lib/toaster/tests/TODO | 13 + lib/toaster/tests/__init__.py | 0 lib/toaster/tests/base.py | 28 + lib/toaster/tests/helpers.py | 192 ++ lib/toaster/tests/setup.py | 94 + lib/toaster/tests/toaster_test.cfg | 25 + lib/toaster/tests/ui.py | 2379 ++++++++++++++++++++ toaster-tests-requirements.txt | 5 + 13 files changed, 3038 insertions(+), 2556 deletions(-) create mode 100755 bin/toaster-test delete mode 100755 lib/toaster/contrib/tts/toasteruitest/run_toastertests.py delete mode 100755 lib/toaster/contrib/tts/toasteruitest/toaster_automation_test.py delete mode 100644 lib/toaster/contrib/tts/toasteruitest/toaster_test.cfg create mode 100644 lib/toaster/tests/README create mode 100644 lib/toaster/tests/TODO create mode 100644 lib/toaster/tests/__init__.py create mode 100644 lib/toaster/tests/base.py create mode 100755 lib/toaster/tests/helpers.py create mode 100644 lib/toaster/tests/setup.py create mode 100644 lib/toaster/tests/toaster_test.cfg create mode 100644 lib/toaster/tests/ui.py create mode 100644 toaster-tests-requirements.txt