Add monitor tests to XTF
cover Add monitor tests to XTF
Extend the framework to support (simple) monitor related tests.
Petre Pircalabu (4):
xtf-runner: split into logical components
xtf: Add executable test class
xtf: Add monitor test class
xtf: Add emul-unimpl test
Makefile | 6 +- build/common.mk | 22 ++- build/files.mk | 3 + build/gen.mk | 25 ++- build/mkinfo.py | 84 +++++++-- docs/all-tests.dox | 5 + include/monitor/monitor.h | 117 ++++++++++++ monitor/Makefile | 20 ++ monitor/monitor.c | 409 +++++++++++++++++++++++++++++++++++++++++ tests/emul-unimpl/Makefile | 15 ++ tests/emul-unimpl/extra.cfg.in | 3 + tests/emul-unimpl/main.c | 59 ++++++ tests/emul-unimpl/monitor.c | 310 +++++++++++++++++++++++++++++++ xtf-runner | 334 ++++----------------------------- xtf/__init__.py | 12 ++ xtf/domu_test.py | 179 ++++++++++++++++++ xtf/exceptions.py | 6 + xtf/executable_test.py | 83 +++++++++ xtf/logger.py | 23 +++ xtf/monitor_test.py | 132 +++++++++++++ xtf/suite.py | 100 ++++++++++ xtf/test.py | 139 ++++++++++++++ xtf/utils.py | 17 ++ xtf/xl_domu.py | 121 ++++++++++++ 24 files changed, 1900 insertions(+), 324 deletions(-) create mode 100644 include/monitor/monitor.h create mode 100644 monitor/Makefile create mode 100644 monitor/monitor.c create mode 100644 tests/emul-unimpl/Makefile create mode 100644 tests/emul-unimpl/extra.cfg.in create mode 100644 tests/emul-unimpl/main.c create mode 100644 tests/emul-unimpl/monitor.c create mode 100644 xtf/__init__.py create mode 100644 xtf/domu_test.py create mode 100644 xtf/exceptions.py create mode 100644 xtf/executable_test.py create mode 100644 xtf/logger.py create mode 100644 xtf/monitor_test.py create mode 100644 xtf/suite.py create mode 100644 xtf/test.py create mode 100644 xtf/utils.py create mode 100644 xtf/xl_domu.py
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 1/4 xtf-runner: split into logical components
Split the xtf-runner script file into multiple modules in order to support multiple test types.
Features:
- 2 abstract types (TestInfo and TestInstance) to represent the test information (info.json) and, respectively to implement the test execution.
TestInfo has to implement the "all_instances" method to create the list of TestInstance objects.
TestInstance has to implement "set_up", "run", and "clean-up" methods.
- TestResult - represents an XTF test result (SUCCESS, SKIP, ERROR, FAILURE, CRASH). The values should be kept in sync with the C code from report.h
- Dynamic test class loading. Each info.json shoudl contain a "class_name" field which specifies the test info class describing the test. This value defaults to "xtf.domu_test.DomuTestInfo"
- custom test info parameters. info.json can have the "extra" field, implemented as a dictionary, which contains parameters specific for a certain test info class.
e.g. TEST-EXTRA-INFO := arg1='--address=0x80000000 --id=4' arg2=42
- logger class (print depending on the quiet field)
- DomuTestInfo/DomuTest instance. Simple test which loads a XEN DomU and checks the output for a specific pattern.
- toolstack abstraction using a wrapper class (e.g. (xtf.xl_domu.XLDomU)
--- build/gen.mk | 13 ++- build/mkinfo.py | 84 +++++++++++--- xtf-runner | 334 +++++------------------------------------------------- xtf/__init__.py | 12 ++ xtf/domu_test.py | 179 +++++++++++++++++++++++++++++ xtf/exceptions.py | 6 + xtf/logger.py | 23 ++++ xtf/suite.py | 97 ++++++++++++++++ xtf/test.py | 139 +++++++++++++++++++++++ xtf/xl_domu.py | 121 ++++++++++++++++++++ 10 files changed, 687 insertions(+), 321 deletions(-) create mode 100644 xtf/__init__.py create mode 100644 xtf/domu_test.py create mode 100644 xtf/exceptions.py create mode 100644 xtf/logger.py create mode 100644 xtf/suite.py create mode 100644 xtf/test.py create mode 100644 xtf/xl_domu.py
build/gen.mk +12 -1
@@ -27,12 +27,23 @@ else TEST-CFGS := $(foreach env,$(TEST-ENVS),test-$(env)-$(NAME).cfg) endif +CLASS ?= "xtf.domu_test.DomuTestInfo" + .PHONY: build build: $(foreach env,$(TEST-ENVS),test-$(env)-$(NAME)) $(TEST-CFGS) build: info.json +MKINFO-OPTS := -n "$(NAME)" +MKINFO-OPTS += -c "$(CLASS)" +MKINFO-OPTS += -t "$(CATEGORY)" +MKINFO-OPTS += -e "$(TEST-ENVS)" +MKINFO-OPTS += -v "$(VARY-CFG)" +ifneq (x$(TEST-EXTRA-INFO), x) +MKINFO-OPTS += -x "$(TEST-EXTRA-INFO)" +endif + info.json: $(ROOT)/build/mkinfo.py FORCE - @$(PYTHON) $< [email protected] "$(NAME)" "$(CATEGORY)" "$(TEST-ENVS)" "$(VARY-CFG)" + @$(PYTHON) $< $(MKINFO-OPTS) [email protected] @$(call move-if-changed,[email protected],$@) .PHONY: install install-each-env
build/mkinfo.py +67 -17
91 lines not on this page · raw message
xtf-runner +31 -303
394 lines not on this page · raw message
xtf/__init__.py new file +12 -0
@@ -0,0 +1,12 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# All test categories +default_categories = set(("functional", "xsa")) +non_default_categories = set(("special", "utility", "in-development")) +all_categories = default_categories | non_default_categories + +# All test environments +pv_environments = set(("pv64", "pv32pae")) +hvm_environments = set(("hvm64", "hvm32pae", "hvm32pse", "hvm32")) +all_environments = pv_environments | hvm_environments
xtf/domu_test.py new file +179 -0
179 lines not on this page · raw message
xtf/exceptions.py new file +6 -0
@@ -0,0 +1,6 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +class RunnerError(Exception): + """ Errors relating to xtf-runner itself """ +
xtf/logger.py new file +23 -0
@@ -0,0 +1,23 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +class Singleton(type): + """Singleton meta class""" + _instances = {} + def __call__(cls, *args, **kwargs): + if cls not in cls._instances: + cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs) + return cls._instances[cls] + +class Logger(object): + """Logger class for XTF.""" + __metaclass__ = Singleton + + def initialize(self, opts): + """Initialize logger""" + self.quiet = opts.quiet + + def log(self, message): + """Display the message""" + if not self.quiet: + print message
xtf/suite.py new file +97 -0
97 lines not on this page · raw message
xtf/test.py new file +139 -0
139 lines not on this page · raw message
xtf/xl_domu.py new file +121 -0
121 lines not on this page · raw message
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 2/4 xtf: Add executable test class
In reply to Petre PircalabuThe Executable test class runs on host (dom0). The class spawns a process and searches the program output(stdio) for a specific pattern.
--- xtf/__init__.py | 2 +- xtf/executable_test.py | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++ xtf/suite.py | 5 ++- 3 files changed, 88 insertions(+), 2 deletions(-) create mode 100644 xtf/executable_test.py
xtf/__init__.py +1 -1
@@ -3,7 +3,7 @@ # All test categories default_categories = set(("functional", "xsa")) -non_default_categories = set(("special", "utility", "in-development")) +non_default_categories = set(("special", "utility", "in-development", "host")) all_categories = default_categories | non_default_categories # All test environments
xtf/executable_test.py new file +83 -0
83 lines not on this page · raw message
xtf/suite.py +4 -1
@@ -75,7 +75,10 @@ def gather_all_test_info(): try: info_file = open(path.join("tests", test, "info.json")) except IOError: - continue + try: + info_file = open(path.join("tests", test, "host.json")) + except IOError: + continue # Ignore tests which have bad JSON try:
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 3/4 xtf: Add monitor test class
In reply to Petre PircalabuThis class starts alongside the domain a monitor application which opens an event channel corresponding to that domain and handles the received requests.
Use the "monitor_args" key to pass test specific arguments to the monitor application.
The arguments will be added in the test's Makefile using the TEST-EXTRA-INFO variable.
--- Makefile | 6 +- build/common.mk | 22 ++- build/files.mk | 3 + build/gen.mk | 12 ++ docs/all-tests.dox | 5 + include/monitor/monitor.h | 117 +++++++++++++ monitor/Makefile | 20 +++ monitor/monitor.c | 409 ++++++++++++++++++++++++++++++++++++++++++++++ xtf/__init__.py | 2 +- xtf/monitor_test.py | 132 +++++++++++++++ xtf/utils.py | 17 ++ 11 files changed, 741 insertions(+), 4 deletions(-) create mode 100644 include/monitor/monitor.h create mode 100644 monitor/Makefile create mode 100644 monitor/monitor.c create mode 100644 xtf/monitor_test.py create mode 100644 xtf/utils.py
Makefile +4 -2
@@ -32,7 +32,9 @@ INSTALL_PROGRAM := $(INSTALL) -p OBJCOPY := $(CROSS_COMPILE)objcopy PYTHON := python -export CC CPP INSTALL INSTALL_DATA INSTALL_DIR INSTALL_PROGRAM OBJCOPY PYTHON +HOSTCC := gcc + +export CC CPP INSTALL INSTALL_DATA INSTALL_DIR INSTALL_PROGRAM OBJCOPY PYTHON HOSTCC .PHONY: all all:
@@ -51,7 +53,7 @@ install: done define all_sources - find include/ arch/ common/ tests/ -name "*.[hcsS]" + find include/ arch/ common/ tests/ monitor/ -name "*.[hcsS]" endef .PHONY: cscope
build/common.mk +21 -1
@@ -1,4 +1,4 @@ -ALL_CATEGORIES := special functional xsa utility in-development +ALL_CATEGORIES := special functional xsa utility in-development monitor ALL_ENVIRONMENTS := pv64 pv32pae hvm64 hvm32pae hvm32pse hvm32
@@ -35,11 +35,20 @@ COMMON_AFLAGS-x86_64 := -m64 COMMON_CFLAGS-x86_32 := -m32 COMMON_CFLAGS-x86_64 := -m64 +#HOSTCFLAGS := -Wall -Werror +HOSTCFLAGS := +HOSTLDFLAGS := +HOSTLDLIBS := +HOSTCFLAGS += -D__XEN_TOOLS__ -g -O3 -I$(ROOT)/include/monitor +HOSTCFLAGS += -DXC_WANT_COMPAT_DEVICEMODEL_API -DXC_WANT_COMPAT_MAP_FOREIGN_API +HOSTLDLIBS += -lxenctrl -lxenstore -lxenevtchn + defcfg-pv := $(ROOT)/config/default-pv.cfg.in defcfg-hvm := $(ROOT)/config/default-hvm.cfg.in obj-perarch := obj-perenv := +obj-monitor := include $(ROOT)/build/files.mk
@@ -90,8 +99,19 @@ DEPS-$(1) = $$(head-$(1)) \ endef +# Setup monitor rules +define MONITOR_setup +DEPS-MONITOR = \ + $$(obj-monitor:%.o=%-monitor.o) + +%-monitor.o: %.c + $$(HOSTCC) $$(HOSTCFLAGS) -c $$< -o $$@ +endef + $(foreach env,$(ALL_ENVIRONMENTS),$(eval $(call PERENV_setup,$(env)))) +$(eval $(call MONITOR_setup)) + define move-if-changed if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi endef
build/files.mk +3 -0
@@ -54,3 +54,6 @@ $(foreach env,$(32BIT_ENVIRONMENTS),$(eval obj-$(env) += $(obj-32))) # 64bit specific objects obj-64 += $(ROOT)/arch/x86/entry_64.o $(foreach env,$(64BIT_ENVIRONMENTS),$(eval obj-$(env) += $(obj-64))) + +# Monitor common objects +obj-monitor += $(ROOT)/monitor/monitor.o
build/gen.mk +12 -0
@@ -32,6 +32,9 @@ CLASS ?= "xtf.domu_test.DomuTestInfo" .PHONY: build build: $(foreach env,$(TEST-ENVS),test-$(env)-$(NAME)) $(TEST-CFGS) build: info.json +ifeq (x$(CATEGORY),xmonitor) +build: test-monitor-$(NAME) +endif MKINFO-OPTS := -n "$(NAME)" MKINFO-OPTS += -c "$(CLASS)"
@@ -100,6 +103,15 @@ install-each-env: install-$(1) install-$(1).cfg endef $(foreach env,$(TEST-ENVS),$(eval $(call PERENV_build,$(env)))) +define MONITOR_build +test-monitor-$(NAME): $(DEPS-MONITOR) + @echo $(obj-monitor) + @echo $(DEPS-MONITOR) + $(HOSTCC) $(HOSTLDFLAGS) $(DEPS-MONITOR) $(HOSTLDLIBS) -o $$@ +endef + +$(eval $(call MONITOR_build)) + .PHONY: clean clean: find $(ROOT) \( -name "*.o" -o -name "*.d" \) -delete
docs/all-tests.dox +5 -0
@@ -145,4 +145,9 @@ enable BTS. @subpage test-nested-svm - Nested SVM tests. @subpage test-nested-vmx - Nested VT-x tests. + + +@section index-monitor Monitor + +@subpage test-emul_unimplemented - @Test EMUL_UNIMPLEMENTED event generation */
include/monitor/monitor.h new file +117 -0
117 lines not on this page · raw message
monitor/Makefile new file +20 -0
@@ -0,0 +1,20 @@ +.PHONY: all + +all: monitor + +HOSTCC ?= gcc + +OBJS = monitor.o + +#HOSTCFLAGS += -Wall -Werror +HOSTCFLAGS += -D__XEN_TOOLS__ -g -O0 +HOSTCFLAGS += -DXC_WANT_COMPAT_DEVICEMODEL_API -DXC_WANT_COMPAT_MAP_FOREIGN_API + +%.o : %.c + $(HOSTCC) -c $(HOSTCFLAGS) $(HOSTCPPFLAGS) $< -o $@ + +monitor: $(OBJS) + $(HOSTCC) -o $@ $^ -lxenctrl -lxenstore -lxenevtchn + +clean: + $(RM) $(OBJS) monitor
monitor/monitor.c new file +409 -0
409 lines not on this page · raw message
xtf/__init__.py +1 -1
@@ -3,7 +3,7 @@ # All test categories default_categories = set(("functional", "xsa")) -non_default_categories = set(("special", "utility", "in-development", "host")) +non_default_categories = set(("special", "utility", "in-development", "host", "monitor")) all_categories = default_categories | non_default_categories # All test environments
xtf/monitor_test.py new file +132 -0
132 lines not on this page · raw message
xtf/utils.py new file +17 -0
@@ -0,0 +1,17 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +""" XTF utils """ + +import threading + +class XTFAsyncCall(threading.Thread): + def __init__(self, group=None, target=None, name=None, args=(), kwargs={}): + super(XTFAsyncCall, self).__init__(group, target, name, args, kwargs) + self._return = None + def run(self): + if self._Thread__target is not None: + self._return = self._Thread__target(*self._Thread__args, + **self._Thread__kwargs) + def join(self): + threading.Thread.join(self) + return self._return
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 4/4 xtf: Add emul-unimpl test
In reply to Petre PircalabuAdd a new test to verify if XEN can correctly handle the X86EMUL_UNIMPLEMENTED event.
The XTF DomU test image just executes a instruction not implemented by the XEN X86 emulator (fstenv) and checks if the execution was successfull. This instruction will be the first one in a custom .text section.
In order to instruct XEN to try to emulate that instruction the monitor application changes the attributes of that specific page in order to inhibit execution. This will trigger a MEM_ACCESS request that will be responded by toggling the EMULATE flag.
The emulation will fail, which will trigger an EMUL_UNIMPLEMENTED request which will be handled by enabling execution on that specific page (altp2m) and singlestepping that instruction.
The test will be successfull if the instruction can be executed correctly.
--- docs/all-tests.dox | 2 +- tests/emul-unimpl/Makefile | 15 ++ tests/emul-unimpl/extra.cfg.in | 3 + tests/emul-unimpl/main.c | 59 ++++++++ tests/emul-unimpl/monitor.c | 310 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 388 insertions(+), 1 deletion(-) create mode 100644 tests/emul-unimpl/Makefile create mode 100644 tests/emul-unimpl/extra.cfg.in create mode 100644 tests/emul-unimpl/main.c create mode 100644 tests/emul-unimpl/monitor.c
docs/all-tests.dox +1 -1
@@ -149,5 +149,5 @@ enable BTS. @section index-monitor Monitor -@subpage test-emul_unimplemented - @Test EMUL_UNIMPLEMENTED event generation +@subpage test-emul-unimpl - @Test EMUL_UNIMPLEMENTED event generation */
tests/emul-unimpl/Makefile new file +15 -0
@@ -0,0 +1,15 @@ +include $(ROOT)/build/common.mk + +NAME := emul-unimpl +CATEGORY := monitor +TEST-ENVS := hvm64 +CLASS := xtf.monitor_test.MonitorTestInfo +TEST-EXTRA-INFO := monitor_args='--address=0x\$$(nm --defined-only @@VM_PATH@@ | grep test_fn | cut -d \ -f 1)' + +TEST-EXTRA-CFG := extra.cfg.in + +obj-perenv += main.o + +obj-monitor += monitor.o + +include $(ROOT)/build/gen.mk
tests/emul-unimpl/extra.cfg.in new file +3 -0
@@ -0,0 +1,3 @@ +# Enable altp2m +altp2m = "mixed" +altp2mhvm = 1
tests/emul-unimpl/main.c new file +59 -0
59 lines not on this page · raw message
tests/emul-unimpl/monitor.c new file +310 -0
310 lines not on this page · raw message
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH XTF 4/4] xtf: Add emul-unimpl test
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 14.12.18 at 14:34, <[email protected]> wrote:
Add a new test to verify if XEN can correctly handle the
X86EMUL_UNIMPLEMENTEDevent.The XTF DomU test image just executes a instruction not implemented by the XEN X86 emulator (fstenv) and checks if the execution was successfull. This instruction will be the first one in a custom .text section.
May I suggest that you use an insn which is liable to remain unimplemented? FSTENV, together with {F,FX,X}{SAVE,RSTOR} are at the top of my (emulator) list of items to work on. If you pick any instruction which we _can_ reasonably implement, chances would then be that eventually your test will fail, preventing an osstest push. This would better be avoided from the beginning.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel