Adding a userspace application crash handling system to autotest
PATCH 1/1 Adding a userspace application crash handling system to autotest
This patch adds a system to watch user space segmentation faults, writing core dumps and some degree of core dump analysis report. We believe that such a system will be beneficial for autotest as a whole, since the ability to get core dumps and dump analysis for each app crashing during an autotest execution can help test engineers with richer debugging information.
The system is comprised by 2 parts:
* Modifications on test code that enable core dumps
generation, register a core handler script in the kernel and check by generated core files at the end of each test.
* A core handler script that is going to write the
core on each test debug dir in a convenient way, with a report that currently is comprised by the process that died and a gdb stacktrace of the process. As the system gets in shape, we could add more scripts that can do fancier stuff (such as handlers that use frysk to get more info such as memory maps, provided that we have frysk installed in the machine).
This is the proof of concept of the system. I am sending it to the mailing list on this early stage so I can get feedback on the feature. The system passes my basic tests:
* Run a simple long test, such as the kvm test, and
then crash an application while the test is running. I get reports generated on test.debugdir
* Run a slightly more complex control file, with 3 parallel
bonnie instances at once and crash an application while the test is running. I get reports generated on all test.debugdirs.
4th try:
* Implemented most of the suggestions made by Michael.
* Moved the crash_report method call outside the finally clause that calls job.logging.restore, so if it something goes wrong logging.restore still gets executed, as John pointed out.
--- client/common_lib/test.py | 66 ++++++++++++++++- client/tools/crash_handler.py | 165 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 229 insertions(+), 2 deletions(-) create mode 100755 client/tools/crash_handler.py
client/common_lib/test.py +64 -2
@@ -17,7 +17,7 @@ # tmpdir eg. tmp/<tempname>_<testname.tag> import fcntl, os, re, sys, shutil, tarfile, tempfile, time, traceback -import warnings, logging +import warnings, logging, glob, resource from autotest_lib.client.common_lib import error from autotest_lib.client.bin import utils
@@ -31,7 +31,6 @@ class base_test: self.job = job self.pkgmgr = job.pkgmgr self.autodir = job.autodir - self.outputdir = outputdir self.tagged_testname = os.path.basename(self.outputdir) self.resultsdir = os.path.join(self.outputdir, 'results')
@@ -40,6 +39,7 @@ class base_test: os.mkdir(self.profdir) self.debugdir = os.path.join(self.outputdir, 'debug') os.mkdir(self.debugdir) + self.configure_crash_handler() self.bindir = bindir if hasattr(job, 'libdir'): self.libdir = job.libdir
@@ -54,6 +54,66 @@ class base_test: self.after_iteration_hooks = [] + def configure_crash_handler(self): + """ + Configure the crash handler by: + * Setting up core size to unlimited + * Putting an appropriate crash handler on /proc/sys/kernel/core_pattern + * Create files that the crash handler will use to figure which tests + are active at a given moment + + The crash handler will pick up the core file and write it to + self.debugdir, and perform analysis on it to generate a report. The + program also outputs some results to syslog. + + If multiple tests are running, an attempt to verify if we still have + the old PID on the system process table to determine whether it is a + parent of the current test execution. If we can't determine it, the + core file and the report file will be copied to all test debug dirs. + """ + self.pattern_file = '/proc/sys/kernel/core_pattern' + try: + # Enable core dumps + resource.setrlimit(resource.RLIMIT_CORE, (-1, -1)) + # Trying to backup core pattern and register our script + self.core_pattern_backup = open(self.pattern_file, 'r').read() + pattern_file = open(self.pattern_file, 'w') + tools_dir = os.path.join(self.autodir, 'tools') + crash_handler_path = os.path.join(tools_dir, 'crash_handler.py') + pattern_file.write('|' + crash_handler_path + ' %p %t %u %s %h %e') + # Writing the files that the crash handler is going to use + self.debugdir_tmp_file = ('/tmp/autotest_results_dir.%s' % + os.getpid()) + utils.open_write_close(self.debugdir_tmp_file, self.debugdir + "\n") + except Exception, e: + self.crash_handling_enabled = False + logging.error('Crash handling system disabled: %s' % e) + else: + self.crash_handling_enabled = True + logging.debug('Crash handling system enabled.') + + + def crash_handler_report(self): + """ + If core dumps are found on the debugdir after the execution of the + test, let the user know. + """ + if self.crash_handling_enabled: + core_dirs = glob.glob('%s/crash.*' % self.debugdir) + if core_dirs: + logging.warning('Programs crashed during test execution:') + for dir in core_dirs: + logging.warning('Please verify %s for more info', dir) + # Remove the debugdir info file + os.unlink(self.debugdir_tmp_file) + # Restore the core pattern backup + try: + utils.open_write_close(self.pattern_file, + self.core_pattern_backup) + except EnvironmentError: + pass + + def assert_(self, expr, msg='Assertion failed.'): if not expr: raise error.TestError(msg)
@@ -376,6 +436,7 @@ class base_test: print 'Ignoring exception during cleanup() phase:' traceback.print_exc() print 'Now raising the earlier %s error' % exc_info[0] + self.crash_handler_report() finally: self.job.logging.restore() try:
@@ -388,6 +449,7 @@ class base_test: try: if run_cleanup: _cherry_pick_call(self.cleanup, *args, **dargs) + self.crash_handler_report() finally: self.job.logging.restore() except error.AutotestError:
client/tools/crash_handler.py new file +165 -0
165 lines not on this page · raw message
Signature · 1 line
1.6.2.5