Error reporting tool
cover Error reporting tool
Hello,
This patch together with the web app from git://git.yoctoproject.org/error-report-web
form an error reporting system similar to kernel-oops service.
This tool proposes a way of keeping track of the errors appeared during builds by dumping the debug information of the build into a database which can be analyzed.
The data collected consists of the log file of the tasks that failed and common configuration variables
(such as machine, distro, host distro, etc as well as the version of the build system - branch/commit id, which
is commonly missing from bug reports :) ).
The project will benefit from the multitude of build results published by quickly identifying the issues and the areas of interest for the contributors. The objectives of the application were achieved by creating a client-server system. The client (basically this bitbake class) collects all the information needed for solving the errors and the server receives that data and saves in a database (see the Django web app at git://git.yoctoproject.org/error-report-web).
All the debug information collected during a build which failed is stored on the user's machine, in a JSON format and its path is displayed in the output of the build as well as the command that the contributor can run a script to send the error report file to the server. The script will give the user a link corresponding to his entry in the database. The scripts accepts a parameter which is the hostname to the web app instance (by default it's localhost).
Contributors can search by certain configurations (e.g. machine, component, error, task) and see statistics through the web interface.
Thanks,
Andreea
The following changes since commit abf22bed20870e6283853674a51a39f44af93936:
Revert "nss: avoid to use the hardcode kernel version" (2014-02-13 12:06:01 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib andreeap/report-poky
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=andreeap/report-poky
Andreea Proca (1):
report-error.bbclass: new class to save build information when errors
occur
meta/classes/report-error.bbclass | 67 +++++++++++++++++++++++++++++++ scripts/send-error-report | 78 +++++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) create mode 100644 meta/classes/report-error.bbclass create mode 100755 scripts/send-error-report
Signature · 1 line
1.7.9.5
PATCH 1/1 report-error.bbclass: new class to save build information when errors occur
Class is used to save data about errors after every task that failed.
Debug data is saved in json files in ERROR_REPORT_DIR (defaults to tmp/log/error-report). To use this class one has to add INHERIT += "report-error" to local.conf.
scripts/send-error-report is a simple script that sends the json file to a HTTP server that collects data (git://git.yoctoproject.org/error-report-web
is a Django web interface that can be used to receive and visualize the error reports). The script will give you an URL where you can find your report.
--- meta/classes/report-error.bbclass | 67 +++++++++++++++++++++++++++++++ scripts/send-error-report | 78 +++++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) create mode 100644 meta/classes/report-error.bbclass create mode 100755 scripts/send-error-report
meta/classes/report-error.bbclass new file +67 -0
@@ -0,0 +1,67 @@ +# +# Collects debug information in order to create error report files. +# +# Copyright (C) 2013 Intel Corporation +# Author: Andreea Brandusa Proca <[email protected]> +# +# Licensed under the MIT license, see COPYING.MIT for details + +ERR_REPORT_DIR ?= "${LOG_DIR}/error-report" + +def getdata(e): + logpath = e.data.getVar('ERR_REPORT_DIR', True) + datafile = os.path.join(logpath, "error-report.txt") + f = open(datafile, "r") + data = f.read() + f.close() + return data + +def savedata(e, newdata, file): + import json + logpath = e.data.getVar('ERR_REPORT_DIR', True) + bb.utils.mkdirhier(logpath) + datafile = os.path.join(logpath, file) + f = open(datafile, "w") + json.dump(newdata, f, indent=4, sort_keys=True) + f.close() + return datafile + +python report () { + import os, re, sys, json + + if isinstance(e, bb.event.BuildStarted): + data = {} + machine = e.data.getVar("MACHINE") + data['machine'] = machine + data['build_sys'] = e.data.getVar("BUILD_SYS", True) + data['nativelsb'] = e.data.getVar("NATIVELSBSTRING") + data['distro'] = e.data.getVar("DISTRO") + data['target_sys'] = e.data.getVar("TARGET_SYS", True) + data['failures'] = [] + data['component'] = e.getPkgs()[0] + data['branch_commit'] = base_detect_branch(e.data) + ": " + base_detect_revision(e.data) + savedata(e, data, "error-report.txt") + + elif isinstance(e, bb.build.TaskFailed): + task = e.task + taskdata={} + log = e.data.getVar('BB_LOGFILE', True) + logFile = open(log, 'r') + taskdata['package'] = e.data.expand("${PF}") + taskdata['task'] = task + taskdata['log'] = logFile.read() + logFile.close() + jsondata = json.loads(getdata(e)) + jsondata['failures'].append(taskdata) + savedata(e, jsondata, "error-report.txt") + + elif isinstance(e, bb.event.BuildCompleted): + jsondata = json.loads(getdata(e)) + failures = jsondata['failures'] + if(len(failures) > 0): + filename = "error_report_" + e.data.getVar("BUILDNAME")+".txt" + datafile = savedata(e, jsondata, filename) + bb.note("The errors of this build are stored in: %s. You can send the errors to an upstream server by running: send-error-report %s [server]" % (datafile, datafile)) +} + +addhandler report
scripts/send-error-report new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
1.7.9.5
Re: [PATCH 1/1] report-error.bbclass: new class to save build information when errors occur
Quoted 41 lines · Andreea Proca
Class is used to save data about errors after every task that failed.
Debug data is saved in json files inERROR_REPORT_DIR(defaults to tmp/log/error-report). To use this class one has to add INHERIT += "report-error" to local.conf.
scripts/send-error-reportis a simple script that sends the json file to a HTTP server that collects data (git://git.yoctoproject.org/error-report-web
is a Django web interface that can be used to receive and visualize the error reports). The script will give you an URL where you can find your report.Signed-off-by Andreea Proca <[email protected]>--- meta/classes/report-error.bbclass | 67 +++++++++++++++++++++++++++++++ scripts/send-error-report | 78 +++++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) create mode 100644 meta/classes/report-error.bbclass create mode 100755 scripts/send-error-reportmeta/classes/report-error.bbclass new file +17
-0@@ -0,0 +1,67 @@ +# +# Collects debug information in order to create error report files. +# +# Copyright (C) 2013 Intel Corporation +# Author: Andreea Brandusa Proca <[email protected]> +# +# Licensed under the MIT license, see COPYING.MIT for details + +ERR_REPORT_DIR ?= "${LOG_DIR}/error-report" + +def getdata(e): + logpath = e.data.getVar('ERR_REPORT_DIR', True) + datafile = os.path.join(logpath, "error-report.txt") + f = open(datafile, "r") + data = f.read() + f.close() + return data
meta/classes/report-error.bbclass line 27
Quoted 10 lines
meta/classes/report-error.bbclass new file +10
-0+ +def savedata(e, newdata, file): + import json + logpath = e.data.getVar('ERR_REPORT_DIR', True) + bb.utils.mkdirhier(logpath) + datafile = os.path.join(logpath, file) + f = open(datafile, "w") + json.dump(newdata, f, indent=4, sort_keys=True) + f.close() + return datafile
Since these functions will end up in the main namespace, could we prefix them so something like errorreport_getdata and errorreport_savedata? This makes it clearer when you look at the names out of context of this class.
Quoted 29 lines
meta/classes/report-error.bbclass new file +29
-0+python report () { + import os, re, sys, json + + if isinstance(e, bb.event.BuildStarted): + data = {} + machine = e.data.getVar("MACHINE") + data['machine'] = machine + data['build_sys'] = e.data.getVar("BUILD_SYS", True) + data['nativelsb'] = e.data.getVar("NATIVELSBSTRING") + data['distro'] = e.data.getVar("DISTRO") + data['target_sys'] = e.data.getVar("TARGET_SYS", True) + data['failures'] = [] + data['component'] = e.getPkgs()[0] + data['branch_commit'] = base_detect_branch(e.data) + ": " + base_detect_revision(e.data) + savedata(e, data, "error-report.txt") + + elif isinstance(e, bb.build.TaskFailed): + task = e.task + taskdata={} + log = e.data.getVar('BB_LOGFILE', True) + logFile = open(log, 'r') + taskdata['package'] = e.data.expand("${PF}") + taskdata['task'] = task + taskdata['log'] = logFile.read() + logFile.close() + jsondata = json.loads(getdata(e)) + jsondata['failures'].append(taskdata) + savedata(e, jsondata, "error-report.txt") +
meta/classes/report-error.bbclass line 67
Quoted 10 lines
meta/classes/report-error.bbclass new file +10
-0+ elif isinstance(e, bb.event.BuildCompleted): + jsondata = json.loads(getdata(e)) + failures = jsondata['failures'] + if(len(failures) > 0): + filename = "error_report_" + e.data.getVar("BUILDNAME")+".txt" + datafile = savedata(e, jsondata, filename) + bb.note("The errors of this build are stored in: %s. You can send the errors to an upstream server by running: send-error-report %s [server]" % (datafile, datafile)) +} + +addhandler report
Could you name this something like errorreport_handler for the same reason as the functions and add an eventmask:
errorreport_handler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted bb.build.TaskFailed"
which means that handler only gets the events it is interested in. I suspect this syntax was added after you started work on this code but it would be nice to add.
I'm really looking forward to seeing this in action!
Cheers,
Richard
Quoted 84 lines
scripts/send-error-report new file +78
-0@@ -0,0 +1,78 @@ +#!/usr/bin/env python + +# Sends an error report (if the report-error class was enabled) to a remote server. +# +# Copyright (C) 2013 Intel Corporation +# Author: Andreea Proca <[email protected]> + + + +import httplib, urllib, os, sys, json + + +def sendData(json_file, server): + + if os.path.isfile(json_file): + + home = os.path.expanduser("~") + userfile = os.path.join(home, ".oe-send-error") + if os.path.isfile(userfile): + with open(userfile) as g: + username = g.readline() + email = g.readline() + else: + print("Please enter your name and your email (optionally), they'll be saved in the file you send.") + username = raw_input("Name: ") + email = raw_input("E-mail (not required): ") + if len(username) > 0 and len(username) < 50: + with open(userfile, "w") as g: + g.write(username + "\n") + g.write(email + "\n") + else: + print("Invalid inputs, try again.") + return + + with open(json_file) as f: + data = f.read() + + try: + jsondata = json.loads(data) + jsondata['username'] = username.strip() + jsondata['email'] = email.strip() + data = json.dumps(jsondata, indent=4, sort_keys=True) + except: + print("Invalid json data") + return + + try: + params = urllib.urlencode({'data': data}) + headers = {"Content-type": "application/json"} + conn = httplib.HTTPConnection(server) + conn.request("POST", "/ClientPost/", params, headers) + response = conn.getresponse() + print response.status, response.reason + res = response.read() + if response.status == 200: + print res + else: + print("There was a problem submiting your data") + conn.close() + except: + print("Server connection failed") + + else: + print("No data file found.") + + +if __name__ == '__main__': + print ("\nSends an error report (if the report-error class was enabled) to a remote server.") + if len(sys.argv) < 2: + print("\nThis scripts sends the contents of a file to an upstream server.") + print("\nUsage: send-error-report <error_fileName> [server]") + print("\nIf this is the first when sending a report you'll be asked for your name and optionally your email address.") + print("They will be associated with your report.\n") + + elif len(sys.argv) == 3: + sendData(sys.argv[1], sys.argv[2]) + else: + sendData(sys.argv[1], "localhost:8000")
Quoted 22 lines · Andreea Proca
Hello,
This patch together with the web app from
git://git.yoctoproject.org/error-report-web
form an error reporting system similar to kernel-oops service.
This tool proposes a way of keeping track of the errors appeared during builds by dumping the debug information of the build into a database which can be analyzed.The data collected consists of the log file of the tasks that failed and common configuration variables
(such as machine, distro, host distro, etc as well as the version of the build system - branch/commit id, which
is commonly missing from bug reports :) ).
The project will benefit from the multitude of build results published by quickly identifying the issues and the areas of interest for the contributors. The objectives of the application were achieved by creating a client-server system. The client (basically this bitbake class) collects all the information needed for solving the errors and the server receives that data and saves in a database (see the Django web app atgit://git.yoctoproject.org/error-report-web).All the debug information collected during a build which failed is stored on the user's machine, in a JSON format and its path is displayed in the output of the build as well as the command that the contributor can run a script to send the error report file to the server. The script will give the user a link corresponding to his entry in the database. The scripts accepts a parameter which is the hostname to the web app instance (by default it's localhost).
Contributors can search by certain configurations (e.g. machine, component, error, task) and see statistics through the web interface.
Quoted 3 lines
Thanks,
Andreea
+1
Good initiative !
kerneloops-daemon/www.kerneloops.org for bitbake as far as I understand the explanation above.
Br,
David